Author: ddas
Date: Wed May 7 05:23:52 2008
New Revision: 654080
URL: http://svn.apache.org/viewvc?rev=654080&view=rev
Log:
HADOOP-2019. Adds support for .tar, .tgz and .tar.gz files in DistributedCache.
Contributed by Amareshwari Sriramadasu.
Added:
hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar (with props)
hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz (with
props)
hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz (with props)
Modified:
hadoop/core/trunk/CHANGES.txt
hadoop/core/trunk/build.xml
hadoop/core/trunk/docs/changes.html
hadoop/core/trunk/docs/mapred_tutorial.html
hadoop/core/trunk/docs/mapred_tutorial.pdf
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java
Modified: hadoop/core/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Wed May 7 05:23:52 2008
@@ -59,6 +59,9 @@
"hdfs:" URIs now defaults to 8020, so that one may simply use URIs
of the form "hdfs://example.com/dir/file".
+ HADOOP-2019. Adds support for .tar, .tgz and .tar.gz files in
+ DistributedCache (Amareshwari Sriramadasu via ddas)
+
IMPROVEMENTS
HADOOP-2928. Remove deprecated FileSystem.getContentLength().
Modified: hadoop/core/trunk/build.xml
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/build.xml?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/build.xml (original)
+++ hadoop/core/trunk/build.xml Wed May 7 05:23:52 2008
@@ -528,6 +528,9 @@
<copy file="${test.src.dir}/org/apache/hadoop/mapred/test.txt"
todir="${test.cache.data}"/>
<copy file="${test.src.dir}/org/apache/hadoop/mapred/test.jar"
todir="${test.cache.data}"/>
<copy file="${test.src.dir}/org/apache/hadoop/mapred/test.zip"
todir="${test.cache.data}"/>
+ <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tar"
todir="${test.cache.data}"/>
+ <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tgz"
todir="${test.cache.data}"/>
+ <copy file="${test.src.dir}/org/apache/hadoop/mapred/test.tar.gz"
todir="${test.cache.data}"/>
<copy file="${test.src.dir}/org/apache/hadoop/dfs/hadoop-14-dfs-dir.tgz"
todir="${test.cache.data}"/>
<copy file="${test.src.dir}/org/apache/hadoop/dfs/hadoop-dfs-dir.txt"
todir="${test.cache.data}"/>
</target>
Modified: hadoop/core/trunk/docs/changes.html
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/docs/changes.html?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/changes.html (original)
+++ hadoop/core/trunk/docs/changes.html Wed May 7 05:23:52 2008
@@ -91,7 +91,7 @@
</ol>
</li>
<li><a
href="javascript:toggleList('trunk_(unreleased_changes)_._new_features_')">
NEW FEATURES
-</a> (4)
+</a> (6)
<ol id="trunk_(unreleased_changes)_._new_features_">
<li><a
href="http://issues.apache.org/jira/browse/HADOOP-3074";>HADOOP-3074</a>.
Provides a UrlStreamHandler for DFS and other FS,
relying on FileSystem<br />(taton)</li>
@@ -101,10 +101,16 @@
Bialecki via omalley)</li>
<li><a
href="http://issues.apache.org/jira/browse/HADOOP-2857";>HADOOP-2857</a>. Allow
libhdfs to set jvm options.<br />(Craig Macdonald
via omalley)</li>
+ <li><a
href="http://issues.apache.org/jira/browse/HADOOP-3317";>HADOOP-3317</a>. Add
default port for HDFS namenode. The port in
+"hdfs:" URIs now defaults to 8020, so that one may simply use URIs
+of the form "hdfs://example.com/dir/file".
+</li>
+ <li><a
href="http://issues.apache.org/jira/browse/HADOOP-2019";>HADOOP-2019</a>. Adds
support for .tar, .tgz and .tar.gz files in
+DistributedCache<br />(Amareshwari Sriramadasu via ddas)</li>
</ol>
</li>
<li><a
href="javascript:toggleList('trunk_(unreleased_changes)_._improvements_')">
IMPROVEMENTS
-</a> (10)
+</a> (11)
<ol id="trunk_(unreleased_changes)_._improvements_">
<li><a
href="http://issues.apache.org/jira/browse/HADOOP-2928";>HADOOP-2928</a>. Remove
deprecated FileSystem.getContentLength().<br />(Lohit Vjayarenu via
rangadi)</li>
<li><a
href="http://issues.apache.org/jira/browse/HADOOP-3130";>HADOOP-3130</a>. Make
the connect timeout smaller for getFile.<br />(Amar Ramesh Kamat via ddas)</li>
@@ -129,6 +135,9 @@
<li><a
href="http://issues.apache.org/jira/browse/HADOOP-3144";>HADOOP-3144</a>.
Improve robustness of LineRecordReader by defining a maximum
line length (mapred.linerecordreader.maxlength), thereby avoiding reading
too far into the following split.<br />(Zheng Shao via cdouglas)</li>
+ <li><a
href="http://issues.apache.org/jira/browse/HADOOP-3334";>HADOOP-3334</a>. Move
lease handling from FSNamesystem into a seperate class.
+(Tsz Wo (Nicholas), SZE via rangadi)
+</li>
</ol>
</li>
<li><a
href="javascript:toggleList('trunk_(unreleased_changes)_._optimizations_')">
OPTIMIZATIONS
Modified: hadoop/core/trunk/docs/mapred_tutorial.html
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/docs/mapred_tutorial.html?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/mapred_tutorial.html (original)
+++ hadoop/core/trunk/docs/mapred_tutorial.html Wed May 7 05:23:52 2008
@@ -1872,7 +1872,8 @@
<p>
<span class="codefrag">DistributedCache</span> can be used to distribute
simple,
read-only data/text files and more complex types such as archives and
- jars. Archives (zip files) are <em>un-archived</em> at the slave
nodes.
+ jars. Archives (zip, tar, tgz and tar.gz files) are
+ <em>un-archived</em> at the slave nodes.
Optionally users can also direct the <span
class="codefrag">DistributedCache</span> to
<em>symlink</em> the cached file(s) into the <span
class="codefrag">current working
directory</span> of the task via the
Modified: hadoop/core/trunk/docs/mapred_tutorial.pdf
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/docs/mapred_tutorial.pdf?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/docs/mapred_tutorial.pdf (original)
+++ hadoop/core/trunk/docs/mapred_tutorial.pdf Wed May 7 05:23:52 2008
@@ -1519,10 +1519,10 @@
>>
endobj
175 0 obj
-<< /Length 2600 /Filter [ /ASCII85Decode /FlateDecode ]
+<< /Length 2590 /Filter [ /ASCII85Decode /FlateDecode ]
>>
stream
-Gatm=D3N__&cR6oJfcbso1XhtqhqAsN8sQ#Au99WlW]Sa&-YYp;'ZO4B4%/-pFZ`/>`\rfefBh?6l^EU]6JMn=m5bMh4M0#s1%Zj'E!?(Qat%Wi])[EMAIL
PROTECTED]'UU*(U^(cm7/qn'W#IK0<`n:MF(9s6cDArc&lde7qKp/[EMAIL
PROTECTED],K%oq':,NB7"WI^&NXBXl0)ZAV[`O5I4G:DpJ&(:_8TnY/7oB&mq!fbq%GZ\)EoR+-/8A"`IbFQ!./5r4ifkar^MhQ2=l0r^uM$HZnUn[!!/U_ELuZ$-?/qk?M.R^<I&:%$'PC#8miO[dWWgp::,;3?[lKIqB-nqFCe`YXJ1DB`G*f]pbNb`iPc[\h85&>@*R(0q.;lOKU"F6#WNhCR4Re/UW3BVD.Dj/7ciCB9(9@'*V:[EMAIL
PROTECTED],ijKP"8TA-f/5N2t<Eb=*WWGI-b0_1$l`Oaeg<Vs7*H!Go0jE`)CMih`&mHUPucu)8,s#gao1i8]LU':KhgY7`YSL(b:`$4dKd85Bndm>a7d#sDHXDBUd9ot8gO/,iaerS!q',&?Z0(K9#I<]I0G$,[EMAIL
PROTECTED]@g5>X3u+.MH+aQ4i^JD`?.HBj-s=iaL5M=e1.;jn$Mo]=<<n8"l<%bU:[EMAIL
PROTECTED]<PYB&SI:g#XX+%Khr(2N<DZ$>"-F)[EMAIL
PROTECTED]<EhF[TZWH)jk?rg3-ah]a!5k/7,\.jLSR+3>u<cAU_mW/[EMAIL
PROTECTED]'<[CiQC(2hQYRGin3.H\&[EMAIL
PROTECTED]"jNYXM`I-h!;/nU*:sSnqUX(fNPqil7E)pIoS8J-ppOR0'_=^M*3ku-V8D$I`uptSY!?3W'ggo/iSI9k"LVucirf"j6qIdQ$K.Xd(53Y\0lp77bfeHO;%TVh=&:hM3iAf76sYG_K<sPkDMTL_06P)BAWf2iUZ_9#[e)kK/UKe!BmQc&"Y`WEW;u(Gh"NT[oVL+Md:rEU'q]b
,[EMAIL PROTECTED]><^)nEV-/iiL&uD;iN)h[7LD\#PM;[EMAIL
PROTECTED]:H#aWdGZ6J#K,*GNIYX/9mW'`3`hHXCHeK0*&-5!ni8Mi)eJsT96JWF67R@(8?)58-`;rs9Hg"eSJ<F5/)[EMAIL
PROTECTED],[EMAIL PROTECTED],\][:EH;[EMAIL
PROTECTED]'L3K;E<uKpTHl6<GTGqN4clGM^)UeC5B@@:+U=:>c5.&)qT*E*SPqM=\jbTOuhB<1KI]7"]/f$Zih>*9p<-FL49]h7]='po+pT)4,@39uA2qi.lq*81gr(8r?-P=IlA.14_!tP('4S'FlE1[h9f8(Z$u`[Tf/[EMAIL
PROTECTED]<fpKfKbR3Z,nQ8n"F*,1WWF9+G-_`jJ,qmJhCWuG:K$Rt@;2R$9-]7gO,1C[#G)/*+2chV$/t90irK2_f^HCl$PGoTGg"2XI)''M%-l6POEO:tIDlArX6WBb2I^c_/lAR[1(^)l/",>k[.#;'<#'-]d'#'[EMAIL
PROTECTED]:im!,LI&@MP?BT>jG1&hDKY\BQJ\FQJ&\9\([EMAIL
PROTECTED]<u4jE0\<J#@)Em%&naK'on7gE)!'5_ScC]1K'?E]r\:54^p%\P*Q?m3256TRoWf#/Y3DLqF[#P.GV6Bu<.bqHk+jIihEGR0/$G!iQ`<ON_'+G`UhdP$5>&]NeQc%S-=Q];oIn%05!-))o6rU[M,$8C43g5rGkYZ-b0<;7Un7j(i5'k;]OU<*6Ccpc++YJ1rZKJ/g)Y8TuGUf$<e4o,_M3%c\+f<fdqVW-*2TRiT9!s;>EZb-_B\r*91e^n.Z5Ai0?c;!0Qc!B#A+*64HVfg+(^,*=\_p>SCD14a=C/I8qCg[269T.1?S0Scd2oc+o(KF4ghb+<m"B,,#eoJ)(W/TacDEbK:'elQY,;Kq$&o4E<TmpY#U;p"JinKcr.f.5bjKLOJhZq1%Psa
"(N;E/3s*4tK?k)`naIZLV\7&fsI,$#aam=?T8I$?]2JYH5EJGP^.Eco9^j>AA/8&!+`#dRF6RCL:Du_([EMAIL
PROTECTED]>t=Htld`\BPq(XctR5'#&Dkm1DH(T&E[VQd3N`bKOiOBDh:h^oK=<JPjA$%#c_/SIO->f\9H,0aA+7CP117;b_LRE095$k7uVA0d,;4Eg\/)>*d!-IG<[EMAIL
PROTECTED]/S1c.))dR/n"u`#i=]=oWO+<1+?"?SZCu_jK$!mL*[:W:E2je!8;"D;9569sR5Ga?5VW6m,jGYiAcL]JV5d&BL:#+&LD[e^GAd0I#p-J;-pr;a;CAcIFHgJX^E>DpQtLobPt_CkQ-QNV2_/WH%!SGnZU;!MJWE*ni'[&)[:X54QP*+4PKP_j8$#P&R[9!Ys;X1UZ>oih6fn8aOc]jLDki+NBoe3WhIC`Z`+98YVp/\548XV`A(1KkIb;$V?cqB)FtkqJKVo]^X^Mdu`'[EMAIL
PROTECTED])'Z%a\2m;O">n<W+-:l.jn07"1&[EMAIL
PROTECTED]'Ck[IRFB_">YjA.V[H)6&[EMAIL PROTECTED]>
+Gatm=D3N__&cR6oJfcbso1Xf^_08\E)9;!PbtQU9f!d%L+;0jqj'+h?lq>h-hBbIE$OWQW]&U&D`,QU*Mqr$<"i&sOrHWbacTBH5_V]Q]WCpJDMB_gS#`#$2Tc]IT#TUsWrS9^+^An3@:d9O2k1:[r;obZ;[EMAIL
PROTECTED]"ejiM8+([FgaMdK(,Mm/54lQ:[EMAIL
PROTECTED]/&I?1pVQG3R?lR4o0"Sm!F3V)f%A#H/<1Im3#QR5Lo<2ajfq.RBlb][O?A^3MLlgV^+7V^:Wk/kO)R8JBUdJRC?ko,Epct$JF=2rTe4mC4<[EMAIL
PROTECTED]<[EMAIL PROTECTED]<p/n4h2t_R/KaO<Gj@"ZjD)(.cF4rfeMYf+/[EMAIL
PROTECTED]@8lFI1KM'F.CKlL$dV,oYlKH?h$Uq?S8]mj6fLQtgI?&qBH0f![Y?/8H`g'ro9^K1H%Ao6:h>0p'H]Bsjm=UOId>X_s`)CCS]<_1MVXlt#QWt+<l.HSFr,K\W*p#GVuQpafEM`cSa+(!Z]QRcM9iq-F&=92jsc2b16XeWZZ;%tFXlc)s@(!h-Z1APrhV.cT?_4Ip/3p4#M%^gHc't/Y#:M>Lc/fSYcO7dsAW.qg8nI/b%1XCele^)g:ItPS92ZDDRPPF)Z1;U""=Zr9#[T_7Y/X]bmflgZEH7a""Wd)pO;!;e.k%Fn6>knhq[TI"[ekBQM7>k>`.-Jm+o'Rd/PJs60g.VS!j&p!GV&*7m.]!&F^m-8&8X;NI([EMAIL
PROTECTED]<YmI#JD;]h`=dr8?>!W<%M7sGk`Zfh\Ma(@TfXa,BhNR.a0LD'#[`)(MM!l7[^d$Epk=rJm,RmnH"!pZ=3u\"nJDG([EMAIL
PROTECTED](?U<L)l&8X)&G7Ub&$CffZ6>+<.%>f5au&a`NF('ole'*?[d8NpbI"@q!#WZ5%"MJDm/%Y<qU0$pd;/ESJ^:W
h[G;)1LQ_>[EMAIL
PROTECTED]):m[1pga5:^s_^*(5*(,6Zdu=/7O-/Q%,<DTkL,8PC$d-pDcpb'%D<Q+%)Ll[[EmR`+ncb4f/[&:(CAB*"[bTP2%6Yef+PlJc,3;h;Je[0M]b4YtV8W*Q?(u`r$W0'h75s^,RP)OmLk`P=<i=,Bgqt$F$=dEM4A+?^l_RQK06.d9!D*>TR28idl[f%;I2Q!jWOAQ!^hkWuFnT2_>EsBprB!MDY(7[R0N:(s"WBT.I9J'>QR$#+Hu_EL3Hc%#7qZhEIQ4fkAQ"Tu0W5(G3Zd/GI[DG"Z'Q]"%ri[c_dfrIK?4V!Q8J_TrhBs*Vm/psg#T($37_aM-^_sA"Gf]"g/cIB/(76],&[EMAIL
PROTECTED];[>+qbP2cU`1ai+(2Q(]4tfJ0sIV"YZModI%f%"m76V7$8bbDTi4/K7f6Cm=5?/G3O75!C#jo?F_?q/kYW!ZW$XL4[5OfUN`Eu\-*.siNI0g>';fEF94]\<*^A4d#hC7MJQVo$d7kI_;)IM6,Nn+@/eno[fP(:mmeia90\E?"GN-)'[EMAIL
PROTECTED])^&[EMAIL
PROTECTED];bR<#0rX2M^.P]VD!m_npqq6jfqY&Rr5^o<h-*%ECTgf87jg^ESG%N*pg>8X25YJWJ?:Q>XPAueB-#(:G:KXmkA>\QFtsIf<uE&EiMbRom"if-+GP>8&ljB0!(@`1<c/+NJ%0aA8bp*(^g85^C,\>+CI<[EMAIL
PROTECTED]/#H.'@&[EN,"W-o1f)FHPk`bHJo$5m)U7N?g?0VfU]LqbS=?$n>E;G'Krto%9;U^s,r4^8JWS^(C*$`6jt8lS$,9ld*qgai=4g1V<ElL0O#W5&d(]9J03;pOZ).+-<?hf$G,CQTmX>U:1EKnXSU.6]7au3b-.t'n]lqF+$2Z\QK(Yj5gt6+1j$<0WU1C17+FBCT"#U^J\amQVJLX)jduc`lP+<YVocZ_(j$RWTca0dcDj,AI2
;>#'qCk:S^&,ZuD<+\4bJ3R0)DD-6LGiCHXOuV]BD!bP)!E4ncLR'/*2%k'Z&2a>Pmq8<[EMAIL
PROTECTED]((",Ag&`Q!t87-V^S;arCoOW'/@/$Y_F>[EMAIL
PROTECTED]'B]fb0W;(CN;&55J*o#]H:W4jd"5jQ_rMP&,[EMAIL
PROTECTED]'7%rYZ1p:.Z"fH0kB]1[([EMAIL PROTECTED]'o%6$"WLB>0A>H$5g1+,nc&[EMAIL
PROTECTED])3$LG>o6g_h_^;[EMAIL
PROTECTED]:5D?(Za9^km(3\I_aBQia(]q70qu\GF&Nu+^%79&L!b`E.3X+U'k#tK-ik(&Erq9Og?11D^MiH*T]aN\F1H1=F?T+&B)%uT6h*X+kpu.]l%8S2JpOi=60l9LQKEP@<`dk97:">PNMha3X`XuL.jCNVk6];1!F9ZnfNFVaH'8KMeE<./9:3/Dh$I<BU)NZ59bkRSHp"d7pH/Z;4V]cA+K+N6d0I1?H3^?bT85ZtTf$0*ea4#j&naIk%IRcbDI\NtacVG2X;@d]7o>Rq[r_O]J:XaI!>'[2r@/?35A.hh5NRYkUGDi%QE9q5Qf/[mO!U3.0kl~>
endstream
endobj
176 0 obj
@@ -2855,53 +2855,53 @@
xref
0 319
0000000000 65535 f
-0000114936 00000 n
-0000115217 00000 n
-0000115310 00000 n
+0000114926 00000 n
+0000115207 00000 n
+0000115300 00000 n
0000000015 00000 n
0000000071 00000 n
0000001299 00000 n
0000001419 00000 n
0000001577 00000 n
-0000115462 00000 n
+0000115452 00000 n
0000001712 00000 n
-0000115525 00000 n
+0000115515 00000 n
0000001849 00000 n
-0000115591 00000 n
+0000115581 00000 n
0000001986 00000 n
-0000115657 00000 n
+0000115647 00000 n
0000002123 00000 n
-0000115721 00000 n
+0000115711 00000 n
0000002259 00000 n
-0000115787 00000 n
+0000115777 00000 n
0000002396 00000 n
-0000115853 00000 n
+0000115843 00000 n
0000002533 00000 n
-0000115917 00000 n
+0000115907 00000 n
0000002669 00000 n
-0000115981 00000 n
+0000115971 00000 n
0000002806 00000 n
-0000116045 00000 n
+0000116035 00000 n
0000002943 00000 n
-0000116109 00000 n
+0000116099 00000 n
0000003078 00000 n
-0000116176 00000 n
+0000116166 00000 n
0000003215 00000 n
-0000116241 00000 n
+0000116231 00000 n
0000003352 00000 n
-0000116306 00000 n
+0000116296 00000 n
0000003487 00000 n
-0000116371 00000 n
+0000116361 00000 n
0000003624 00000 n
-0000116438 00000 n
+0000116428 00000 n
0000003761 00000 n
-0000116505 00000 n
+0000116495 00000 n
0000003897 00000 n
-0000116570 00000 n
+0000116560 00000 n
0000004034 00000 n
-0000116637 00000 n
+0000116627 00000 n
0000004171 00000 n
-0000116702 00000 n
+0000116692 00000 n
0000004308 00000 n
0000006939 00000 n
0000007062 00000 n
@@ -2994,7 +2994,7 @@
0000051557 00000 n
0000051602 00000 n
0000051811 00000 n
-0000116767 00000 n
+0000116757 00000 n
0000051947 00000 n
0000052238 00000 n
0000054964 00000 n
@@ -3030,149 +3030,149 @@
0000067897 00000 n
0000068125 00000 n
0000068330 00000 n
-0000071024 00000 n
-0000071150 00000 n
-0000071211 00000 n
-0000071464 00000 n
-0000071654 00000 n
-0000071859 00000 n
-0000072106 00000 n
-0000072307 00000 n
-0000074778 00000 n
-0000074904 00000 n
-0000074965 00000 n
-0000075240 00000 n
-0000075515 00000 n
-0000075773 00000 n
-0000076002 00000 n
-0000076237 00000 n
-0000078926 00000 n
-0000079052 00000 n
-0000079201 00000 n
-0000079412 00000 n
-0000079620 00000 n
-0000079795 00000 n
-0000079990 00000 n
-0000080164 00000 n
-0000080340 00000 n
-0000080563 00000 n
-0000080804 00000 n
-0000081000 00000 n
-0000081213 00000 n
-0000081430 00000 n
-0000081649 00000 n
-0000081921 00000 n
-0000082186 00000 n
-0000082466 00000 n
-0000082679 00000 n
-0000085278 00000 n
-0000085404 00000 n
-0000085457 00000 n
-0000085778 00000 n
-0000086096 00000 n
-0000086282 00000 n
-0000086480 00000 n
-0000088762 00000 n
-0000088872 00000 n
-0000091219 00000 n
-0000091329 00000 n
-0000093653 00000 n
-0000093763 00000 n
-0000096102 00000 n
-0000096212 00000 n
-0000098426 00000 n
-0000098536 00000 n
-0000100530 00000 n
-0000100640 00000 n
-0000101895 00000 n
-0000102005 00000 n
-0000103719 00000 n
-0000116827 00000 n
-0000103829 00000 n
-0000103965 00000 n
-0000104158 00000 n
-0000104316 00000 n
-0000104532 00000 n
-0000104816 00000 n
-0000104986 00000 n
-0000105136 00000 n
-0000105312 00000 n
-0000105628 00000 n
-0000116881 00000 n
-0000105818 00000 n
-0000116948 00000 n
-0000106012 00000 n
-0000117013 00000 n
-0000106204 00000 n
-0000117080 00000 n
-0000106419 00000 n
-0000117147 00000 n
-0000106587 00000 n
-0000117214 00000 n
-0000106794 00000 n
-0000117279 00000 n
-0000106998 00000 n
-0000117345 00000 n
-0000107175 00000 n
-0000117412 00000 n
-0000107415 00000 n
-0000117479 00000 n
-0000107612 00000 n
-0000117545 00000 n
-0000107809 00000 n
-0000117613 00000 n
-0000107988 00000 n
-0000108194 00000 n
-0000108415 00000 n
-0000108699 00000 n
-0000117681 00000 n
-0000109032 00000 n
-0000109198 00000 n
-0000117749 00000 n
-0000109413 00000 n
-0000117815 00000 n
-0000109589 00000 n
-0000109777 00000 n
-0000117883 00000 n
-0000109998 00000 n
-0000117951 00000 n
-0000110243 00000 n
-0000110431 00000 n
-0000118017 00000 n
-0000110703 00000 n
-0000118085 00000 n
-0000110867 00000 n
-0000118153 00000 n
-0000111094 00000 n
-0000118219 00000 n
-0000111249 00000 n
-0000118287 00000 n
-0000111470 00000 n
-0000118353 00000 n
-0000111697 00000 n
-0000118421 00000 n
-0000111998 00000 n
-0000118489 00000 n
-0000112261 00000 n
-0000118555 00000 n
-0000112487 00000 n
-0000118623 00000 n
-0000112678 00000 n
-0000118691 00000 n
-0000112931 00000 n
-0000118759 00000 n
-0000113176 00000 n
-0000113367 00000 n
-0000113636 00000 n
-0000113806 00000 n
-0000113991 00000 n
-0000114156 00000 n
-0000114270 00000 n
-0000114381 00000 n
-0000114493 00000 n
-0000114602 00000 n
-0000114709 00000 n
-0000114826 00000 n
+0000071014 00000 n
+0000071140 00000 n
+0000071201 00000 n
+0000071454 00000 n
+0000071644 00000 n
+0000071849 00000 n
+0000072096 00000 n
+0000072297 00000 n
+0000074768 00000 n
+0000074894 00000 n
+0000074955 00000 n
+0000075230 00000 n
+0000075505 00000 n
+0000075763 00000 n
+0000075992 00000 n
+0000076227 00000 n
+0000078916 00000 n
+0000079042 00000 n
+0000079191 00000 n
+0000079402 00000 n
+0000079610 00000 n
+0000079785 00000 n
+0000079980 00000 n
+0000080154 00000 n
+0000080330 00000 n
+0000080553 00000 n
+0000080794 00000 n
+0000080990 00000 n
+0000081203 00000 n
+0000081420 00000 n
+0000081639 00000 n
+0000081911 00000 n
+0000082176 00000 n
+0000082456 00000 n
+0000082669 00000 n
+0000085268 00000 n
+0000085394 00000 n
+0000085447 00000 n
+0000085768 00000 n
+0000086086 00000 n
+0000086272 00000 n
+0000086470 00000 n
+0000088752 00000 n
+0000088862 00000 n
+0000091209 00000 n
+0000091319 00000 n
+0000093643 00000 n
+0000093753 00000 n
+0000096092 00000 n
+0000096202 00000 n
+0000098416 00000 n
+0000098526 00000 n
+0000100520 00000 n
+0000100630 00000 n
+0000101885 00000 n
+0000101995 00000 n
+0000103709 00000 n
+0000116817 00000 n
+0000103819 00000 n
+0000103955 00000 n
+0000104148 00000 n
+0000104306 00000 n
+0000104522 00000 n
+0000104806 00000 n
+0000104976 00000 n
+0000105126 00000 n
+0000105302 00000 n
+0000105618 00000 n
+0000116871 00000 n
+0000105808 00000 n
+0000116938 00000 n
+0000106002 00000 n
+0000117003 00000 n
+0000106194 00000 n
+0000117070 00000 n
+0000106409 00000 n
+0000117137 00000 n
+0000106577 00000 n
+0000117204 00000 n
+0000106784 00000 n
+0000117269 00000 n
+0000106988 00000 n
+0000117335 00000 n
+0000107165 00000 n
+0000117402 00000 n
+0000107405 00000 n
+0000117469 00000 n
+0000107602 00000 n
+0000117535 00000 n
+0000107799 00000 n
+0000117603 00000 n
+0000107978 00000 n
+0000108184 00000 n
+0000108405 00000 n
+0000108689 00000 n
+0000117671 00000 n
+0000109022 00000 n
+0000109188 00000 n
+0000117739 00000 n
+0000109403 00000 n
+0000117805 00000 n
+0000109579 00000 n
+0000109767 00000 n
+0000117873 00000 n
+0000109988 00000 n
+0000117941 00000 n
+0000110233 00000 n
+0000110421 00000 n
+0000118007 00000 n
+0000110693 00000 n
+0000118075 00000 n
+0000110857 00000 n
+0000118143 00000 n
+0000111084 00000 n
+0000118209 00000 n
+0000111239 00000 n
+0000118277 00000 n
+0000111460 00000 n
+0000118343 00000 n
+0000111687 00000 n
+0000118411 00000 n
+0000111988 00000 n
+0000118479 00000 n
+0000112251 00000 n
+0000118545 00000 n
+0000112477 00000 n
+0000118613 00000 n
+0000112668 00000 n
+0000118681 00000 n
+0000112921 00000 n
+0000118749 00000 n
+0000113166 00000 n
+0000113357 00000 n
+0000113626 00000 n
+0000113796 00000 n
+0000113981 00000 n
+0000114146 00000 n
+0000114260 00000 n
+0000114371 00000 n
+0000114483 00000 n
+0000114592 00000 n
+0000114699 00000 n
+0000114816 00000 n
trailer
<<
/Size 319
@@ -3180,5 +3180,5 @@
/Info 4 0 R
>>
startxref
-118827
+118817
%%EOF
Modified:
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
---
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
(original)
+++
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/mapred_tutorial.xml
Wed May 7 05:23:52 2008
@@ -1416,7 +1416,8 @@
<p><code>DistributedCache</code> can be used to distribute simple,
read-only data/text files and more complex types such as archives and
- jars. Archives (zip files) are <em>un-archived</em> at the slave
nodes.
+ jars. Archives (zip, tar, tgz and tar.gz files) are
+ <em>un-archived</em> at the slave nodes.
Optionally users can also direct the <code>DistributedCache</code>
to
<em>symlink</em> the cached file(s) into the <code>current working
directory</code> of the task via the
Modified:
hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
---
hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
(original)
+++
hadoop/core/trunk/src/java/org/apache/hadoop/filecache/DistributedCache.java
Wed May 7 05:23:52 2008
@@ -50,11 +50,11 @@
*
* <p><code>DistributedCache</code> can be used to distribute simple, read-only
* data/text files and/or more complex types such as archives, jars etc.
- * Archives (zip files) are un-archived at the slave nodes. Jars maybe be
- * optionally added to the classpath of the tasks, a rudimentary software
- * distribution mechanism. Files have execution permissions. Optionally users
- * can also direct it to symlink the distributed cache file(s) into
- * the working directory of the task.</p>
+ * Archives (zip, tar and tgz/tar.gz files) are un-archived at the slave
nodes.
+ * Jars may be optionally added to the classpath of the tasks, a rudimentary
+ * software distribution mechanism. Files have execution permissions.
+ * Optionally users can also direct it to symlink the distributed cache file(s)
+ * into the working directory of the task.</p>
*
* <p><code>DistributedCache</code> tracks modification timestamps of the
cache
* files. Clearly the cache files should not be modified by the application
@@ -70,6 +70,9 @@
* $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat
* $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip
* $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar
+ * $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar
+ * $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz
+ * $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz
*
* 2. Setup the application's <code>JobConf</code>:
*
@@ -78,7 +81,10 @@
* job);
* DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);
* DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);
- *
+ * DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);
+ * DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);
+ * DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);
+ *
* 3. Use the cached files in the [EMAIL PROTECTED] Mapper} or [EMAIL
PROTECTED] Reducer}:
*
* public static class MapClass extends MapReduceBase
@@ -129,9 +135,11 @@
* @param conf The Confguration file which contains the filesystem
* @param baseDir The base cache Dir where you wnat to localize the
files/archives
* @param fileStatus The file status on the dfs.
- * @param isArchive if the cache is an archive or a file. In case it is an
archive
- * with a .zip or .jar extension it will be unzipped/unjarred automatically
- * and the directory where the archive is unjarred is returned as the Path.
+ * @param isArchive if the cache is an archive or a file. In case it is an
+ * archive with a .zip or .jar or .tar or .tgz or .tar.gz extension it will
+ * be unzipped/unjarred/untarred automatically
+ * and the directory where the archive is unzipped/unjarred/untarred is
+ * returned as the Path.
* In case of a file, the path to the file is returned
* @param confFileStamp this is the hdfs file modification timestamp to
verify that the
* file to be cached hasn't changed since the job started
@@ -185,9 +193,11 @@
* being used in the Configuration
* @param conf The Confguration file which contains the filesystem
* @param baseDir The base cache Dir where you wnat to localize the
files/archives
- * @param isArchive if the cache is an archive or a file. In case it is an
archive
- * with a .zip or .jar extension it will be unzipped/unjarred automatically
- * and the directory where the archive is unjarred is returned as the Path.
+ * @param isArchive if the cache is an archive or a file. In case it is an
+ * archive with a .zip or .jar or .tar or .tgz or .tar.gz extension it will
+ * be unzipped/unjarred/untarred automatically
+ * and the directory where the archive is unzipped/unjarred/untarred
+ * is returned as the Path.
* In case of a file, the path to the file is returned
* @param confFileStamp this is the hdfs file modification timestamp to
verify that the
* file to be cached hasn't changed since the job started
@@ -331,13 +341,14 @@
fs.copyToLocalFile(new Path(cacheId), parchive);
if (isArchive) {
String tmpArchive = parchive.toString().toLowerCase();
+ File srcFile = new File(parchive.toString());
+ File destDir = new File(parchive.getParent().toString());
if (tmpArchive.endsWith(".jar")) {
- RunJar.unJar(new File(parchive.toString()), new File(parchive
-
.getParent().toString()));
+ RunJar.unJar(srcFile, destDir);
} else if (tmpArchive.endsWith(".zip")) {
- FileUtil.unZip(new File(parchive.toString()), new File(parchive
-
.getParent().toString()));
-
+ FileUtil.unZip(srcFile, destDir);
+ } else if (isTarFile(tmpArchive)) {
+ FileUtil.unTar(srcFile, destDir);
}
// else will not do anyhting
// and copy the file into the dir as it is
@@ -373,6 +384,11 @@
}
}
+ private static boolean isTarFile(String filename) {
+ return (filename.endsWith(".tgz") || filename.endsWith(".tar.gz") ||
+ filename.endsWith(".tar"));
+ }
+
// Checks if the cache has already been localized and is fresh
private static boolean ifExistsAndFresh(Configuration conf, FileSystem fs,
URI cache, long confFileStamp,
Modified: hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java (original)
+++ hadoop/core/trunk/src/java/org/apache/hadoop/fs/FileUtil.java Wed May 7
05:23:52 2008
@@ -26,6 +26,7 @@
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.StringUtils;
import org.apache.hadoop.util.Shell;
+import org.apache.hadoop.util.Shell.ShellCommandExecutor;
/**
* A collection of file-processing util methods
@@ -453,7 +454,51 @@
zipFile.close();
}
}
-
+
+ /**
+ * Given a Tar File as input it will untar the file in a the untar directory
+ * passed as the second parameter
+ *
+ * This utility will untar ".tar" files and ".tar.gz","tgz" files.
+ *
+ * @param inFile The tar file as input.
+ * @param untarDir The untar directory where to untar the tar file.
+ * @throws IOException
+ */
+ public static void unTar(File inFile, File untarDir) throws IOException {
+ if (!untarDir.mkdirs()) {
+ if (!untarDir.isDirectory()) {
+ throw new IOException("Mkdirs failed to create " + untarDir);
+ }
+ }
+
+ StringBuffer untarCommand = new StringBuffer();
+ boolean gzipped = inFile.toString().endsWith("gz");
+ if (gzipped) {
+ untarCommand.append(" gzip -dc '");
+ untarCommand.append(FileUtil.makeShellPath(inFile));
+ untarCommand.append("' | (");
+ }
+ untarCommand.append("cd '");
+ untarCommand.append(FileUtil.makeShellPath(untarDir));
+ untarCommand.append("' ; ");
+ untarCommand.append("tar -xf ");
+
+ if (gzipped) {
+ untarCommand.append(" -)");
+ } else {
+ untarCommand.append(FileUtil.makeShellPath(inFile));
+ }
+ String[] shellCmd = { "bash", "-c", untarCommand.toString() };
+ ShellCommandExecutor shexec = new ShellCommandExecutor(shellCmd);
+ shexec.execute();
+ int exitcode = shexec.getExitCode();
+ if (exitcode != 0) {
+ throw new IOException("Error untarring file " + inFile +
+ ". Tar process exited with exit code " + exitcode);
+ }
+ }
+
/**
* Class for creating hardlinks.
* Supports Unix, Cygwin, WindXP.
Modified:
hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
---
hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
(original)
+++
hadoop/core/trunk/src/test/org/apache/hadoop/dfs/TestDFSUpgradeFromImage.java
Wed May 7 05:23:52 2008
@@ -27,7 +27,6 @@
import java.util.zip.CRC32;
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.util.Shell;
import org.apache.hadoop.fs.FSInputStream;
import org.apache.hadoop.fs.FileUtil;
import org.apache.hadoop.dfs.FSConstants.StartupOption;
@@ -72,12 +71,7 @@
if ( dfsDir.exists() && !FileUtil.fullyDelete(dfsDir) ) {
throw new IOException("Could not delete dfs directory '" + dfsDir + "'");
}
- String cmd = "gzip -dc '" + FileUtil.makeShellPath(tarFile) + "' | (cd '" +
- FileUtil.makeShellPath(dataDir) + "' ; tar -xf -)";
- LOG.info("Unpacking the tar file. Cmd : " + cmd);
- String[] shellCmd = { "bash", "-c", cmd };
- Shell.execCommand(shellCmd);
-
+ FileUtil.unTar(new File(tarFile), new File(dataDir));
//Now read the reference info
BufferedReader reader = new BufferedReader(
Modified: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java?rev=654080&r1=654079&r2=654080&view=diff
==============================================================================
--- hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java
(original)
+++ hadoop/core/trunk/src/test/org/apache/hadoop/mapred/MRCaching.java Wed May
7 05:23:52 2008
@@ -175,6 +175,9 @@
Path txtPath = new Path(localPath, new Path("test.txt"));
Path jarPath = new Path(localPath, new Path("test.jar"));
Path zipPath = new Path(localPath, new Path("test.zip"));
+ Path tarPath = new Path(localPath, new Path("test.tgz"));
+ Path tarPath1 = new Path(localPath, new Path("test.tar.gz"));
+ Path tarPath2 = new Path(localPath, new Path("test.tar"));
Path cachePath = new Path(cacheDir);
fs.delete(cachePath, true);
if (!fs.mkdirs(cachePath)) {
@@ -183,13 +186,23 @@
fs.copyFromLocalFile(txtPath, cachePath);
fs.copyFromLocalFile(jarPath, cachePath);
fs.copyFromLocalFile(zipPath, cachePath);
+ fs.copyFromLocalFile(tarPath, cachePath);
+ fs.copyFromLocalFile(tarPath1, cachePath);
+ fs.copyFromLocalFile(tarPath2, cachePath);
// setting the cached archives to zip, jar and simple text files
URI uri1 = fs.getUri().resolve(cachePath + "/test.jar");
URI uri2 = fs.getUri().resolve(cachePath + "/test.zip");
URI uri3 = fs.getUri().resolve(cachePath + "/test.txt");
+ URI uri4 = fs.getUri().resolve(cachePath + "/test.tgz");
+ URI uri5 = fs.getUri().resolve(cachePath + "/test.tar.gz");
+ URI uri6 = fs.getUri().resolve(cachePath + "/test.tar");
+
DistributedCache.addCacheArchive(uri1, conf);
DistributedCache.addCacheArchive(uri2, conf);
DistributedCache.addCacheFile(uri3, conf);
+ DistributedCache.addCacheArchive(uri4, conf);
+ DistributedCache.addCacheArchive(uri5, conf);
+ DistributedCache.addCacheArchive(uri6, conf);
RunningJob job = JobClient.runJob(conf);
int count = 0;
// after the job ran check to see if the the input from the localized cache
@@ -208,7 +221,7 @@
}
file.close();
}
- if (count != 3)
+ if (count != 6)
return new TestResult(job, false);
return new TestResult(job, true);
Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar?rev=654080&view=auto
==============================================================================
Binary file - no diff available.
Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz?rev=654080&view=auto
==============================================================================
Binary file - no diff available.
Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tar.gz
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Added: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz?rev=654080&view=auto
==============================================================================
Binary file - no diff available.
Propchange: hadoop/core/trunk/src/test/org/apache/hadoop/mapred/test.tgz
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
