> On March 10, 2015, 6:42 p.m., Andrew Onischuk wrote: > > Alejandro, this will make our deploy longer much, since those fs commands > > take lot of time on loaded cluster. > > > > Can we like set x retries with y delay for CopyFromLocal, in that case we > > won't increase time of successful deployment. > > Andrew Onischuk wrote: > Meaning this will increase time of every CopyFromLocal x2 or so, and we > have a lot of that things called, and they seems to take a lot of time even > now. > > Alejandro Fernandez wrote: > What triggers this is YARN History Server START, Hive Sever 2 START, Pig > Service Check, and Spark History Server START. This also includes starting > after a Rolling Upgrade. > In all of these cases, we first check if the file is already present in > HDFS, so the only performance impact comes from the actual copy the first > time. > I used this approach because it guarantees 100% success. I already wrote > a retry-decorator, so I'll see what it takes to make it do a random sleep, > but this will require more logic. > > Andrew Onischuk wrote: > What I mean is that any hadoop call takes some time because it initiates > jvm, connections, etc. everytime. On loaded machine, 1 hadoop call can > somethimes take 15-20sec doesn't matter if it's -copyFromLocal -mv or -ls. We > might want to rethink this a little if this is possible. Random sleep is the > same thing which takes time. I think we can add to Execute below tries=x, > tries_sleep=y, so if it fails due to race condition, it will retry in some > time and success?
These are the possiblities. A = cp and ls (if fails, then retry). B = cp and then mv (guarantees succees) 1. No Race Condition: both strategies will have to execute two hadoop calls. 2. Race Condition. A. If strategy A fails the first time, it has to keep trying x times, and each time performs it 2 operations, on top of the sleeps. B. Stategy B only performs 2 operations and the mv is atomic since it is handled by the NameNode I believe the current implementation is better suited to our needs right now. - Alejandro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31878/#review75927 ----------------------------------------------------------- On March 10, 2015, 1:41 a.m., Alejandro Fernandez wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/31878/ > ----------------------------------------------------------- > > (Updated March 10, 2015, 1:41 a.m.) > > > Review request for Ambari, Andrew Onischuk, Jonathan Hurley, Nate Cole, and > Sid Wagle. > > > Bugs: AMBARI-9990 > https://issues.apache.org/jira/browse/AMBARI-9990 > > > Repository: ambari > > > Description > ------- > > Pig Service Check and Hive Server 2 START ran on 2 different machines during > the stack installation and failed to copy the tez tarball to HDFS. > > I was able to reproduce this locally by calling CopyFromLocal from two > clients simultaneously. See the HDFS audit log, datanode logs on c6408 & > c6410, and namenode log on c6410. > > The copyFromLocal command's behavior is: > * Try to create a temporary file <filename>._COPYING_ and write the real data > there > * If hit any exception, delete the file with the name <filename>._COPYING_ > > Thus we have the following race condition in this test: > Process P1 created file "tez.tar.gz._COPYING_" and wrote data to it > Process P2 fired the same copyFromLocal command and hit exception because it > could not get the lease > P2 then deleted the file "tez.tar.gz._COPYING_" > P1 could not close the file "tez.tar.gz._COPYING_" since it had been deleted > by P2. The exception would say "could not find lease for file..." > In general we do not have the correct synchronization guarantee for the > "copyFromLocal" command. > > One solution is for the destination file name to be unique. Because the mv > command is synchronized by the namenode, at least one of them will succeed in > naming the file. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/dynamic_variable_interpretation.py > 00b8d70 > > Diff: https://reviews.apache.org/r/31878/diff/ > > > Testing > ------- > > Unit tests on builds.apache.org passed, > https://builds.apache.org/job/Ambari-trunk-test-patch/1977/ > > I also deployed a cluster and verified that it was able to copy the tarballs > to HDFS when installing YARN, Hive, Pig. > > [root@c6408 ~]# su - hdfs -c 'hadoop fs -ls -R /hdp/apps/2.2.2.0-2538/' > dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:55 > /hdp/apps/2.2.2.0-2538/hive > -r--r--r-- 3 hdfs hadoop 82982575 2015-03-10 00:55 > /hdp/apps/2.2.2.0-2538/hive/hive.tar.gz > dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:57 > /hdp/apps/2.2.2.0-2538/mapreduce > -r--r--r-- 3 hdfs hadoop 105000 2015-03-10 00:57 > /hdp/apps/2.2.2.0-2538/mapreduce/hadoop-streaming.jar > -r--r--r-- 3 hdfs hadoop 192699956 2015-03-09 18:15 > /hdp/apps/2.2.2.0-2538/mapreduce/mapreduce.tar.gz > dr-xr-xr-x - hdfs hdfs 0 2015-03-10 00:56 > /hdp/apps/2.2.2.0-2538/pig > -r--r--r-- 3 hdfs hadoop 97542246 2015-03-10 00:56 > /hdp/apps/2.2.2.0-2538/pig/pig.tar.gz > dr-xr-xr-x - hdfs hdfs 0 2015-03-09 18:15 > /hdp/apps/2.2.2.0-2538/tez > -r--r--r-- 3 hdfs hadoop 40656789 2015-03-09 18:15 > /hdp/apps/2.2.2.0-2538/tez/tez.tar.gz > > > Thanks, > > Alejandro Fernandez > >
