On Sat, Jun 15, 2013 at 8:19 AM, Alejandro Abdelnur <t...@cloudera.com> wrote:
> If the intention is to get the release out in time for the Hadoop Summit we
> have a very tight schedule.
>
> Because the release vote runs for 7 days, we should have an RC latest
> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> if we need to cut a new RC we can do it on Tuesday. Another thing to
> consider is that if the changes on an RC are corrections that do not affect
> code, we could agree on not reseting the voting period clock if we need to
> cut a new RC (ie doc, build, notes changes).
I didn't realize the schedule was tight. Hence I tried my best to run the
integration tests on Hadoop 2.1.0 as quickly as possible. Here are the
results:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/
The good news is that quite a few tests passed.
But there's a bit of bad news too (or at least the news that need to
be triaged). At
this point I don't know whether the Hadoop code is to blame or the
tests/components
themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
1. HDFS append integration tests failed:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
It seems that there's some incompatibility between the client code
that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
and the current Hadoop 2.1.0.
2. Quite a few Sqoop tests ended up failing because of what seems
like AM not
realizing that one of the tasks exited and waiting for it to
timeout. In the end
the task is getting killed like this:
AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
after 600 secsContainer killed by the ApplicationMaster.
but it takes a VERY long time (on the task side the log is
attached bellow).
3. There's a couple of Hive tests (out of more than a dozen) that
failed in a pretty odd way
(scroll to the very bottom of every page to see the excpetion):
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
What's peculiar here is that nothing has failed *before* or
*after* these particular
tests. Hence I don't think that the state of the cluster
deployment is to blame.
4. All of the Mahout tests failed with the following:
http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
This odd, and as I said -- all I know at this point is that
the very same
tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
Given the tight schedule any help in triaging these results will be appreciated.
In the meantime I'll run the same set of tests against a fully secure cluster.
Stay tuned!
Thanks,
Roman.
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Running extractor class
org.apache.sqoop.job.etl.HdfsExportExtractor
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Working on partition:
{hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
(offset=0, end=8, length=8)}
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting file
hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: from offset 0
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: to offset 8
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: of length 8
2013-06-16 03:16:14,373 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting text file
2013-06-16 03:16:14,542 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Start position: 0
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting ended on
position: 77
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Extractor has finished
2013-06-16 03:16:14,552 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
2013-06-16 03:16:14,571 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2013-06-16 03:16:15,300 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while
loading data out of MR job.
java.lang.NumberFormatException: For input string: ""9999-12-31""
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:410)
at java.lang.Long.parseLong(Long.java:468)
at org.apache.sqoop.job.io.Data.parseField(Data.java:449)
at org.apache.sqoop.job.io.Data.parse(Data.java:374)
at org.apache.sqoop.job.io.Data.getContent(Data.java:88)
at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readContent(SqoopOutputFormatLoadExecutor.java:169)
at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readArrayRecord(SqoopOutputFormatLoadExecutor.java:145)
at
org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:48)
at
org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:25)
at
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:216)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2013-06-16 03:16:15,303 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
