Build failed in Jenkins: Hadoop-Common-0.23-Build #1042
See https://builds.apache.org/job/Hadoop-Common-0.23-Build/1042/ -- [...truncated 8263 lines...] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Running org.apache.hadoop.io.TestVersionedWritable Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec Running org.apache.hadoop.io.TestMapFile Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.792 sec Running org.apache.hadoop.io.TestText Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.913 sec Running org.apache.hadoop.io.TestBloomMapFile Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.034 sec Running org.apache.hadoop.io.serializer.TestSerializationFactory Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.322 sec Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.662 sec Running org.apache.hadoop.io.serializer.TestWritableSerialization Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.39 sec Running org.apache.hadoop.io.TestDataByteBuffers Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.507 sec Running org.apache.hadoop.io.TestArrayFile Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.628 sec Running org.apache.hadoop.io.TestWritableName Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.171 sec Running org.apache.hadoop.io.TestIOUtils Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.372 sec Running org.apache.hadoop.io.TestSetFile Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.004 sec Running org.apache.hadoop.io.TestSequenceFile Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.275 sec Running org.apache.hadoop.io.TestObjectWritableProtos Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.377 sec Running org.apache.hadoop.io.TestMD5Hash Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.183 sec Running org.apache.hadoop.io.TestArrayWritable Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec Running org.apache.hadoop.io.retry.TestFailoverProxy Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.233 sec Running org.apache.hadoop.io.retry.TestRetryProxy Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.235 sec Running org.apache.hadoop.io.TestEnumSetWritable Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.542 sec Running org.apache.hadoop.io.TestSecureIOUtils Tests run: 4, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.623 sec Running org.apache.hadoop.io.TestBytesWritable Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.118 sec Running org.apache.hadoop.io.file.tfile.TestTFileNoneCodecsByteArrays Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.954 sec Running org.apache.hadoop.io.file.tfile.TestTFileStreams Tests run: 19, Failures: 0, Errors: 0, Skipped: 0,
Re: [DISCUSS] Switch to log4j 2
moving to SLF4J as an API is independent —it's just a better API for logging than commons-logging, was already a dependency and doesn't force anyone to switch to a new log back end. On 15 August 2014 03:34, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote: Hi, Steve has started discussion titled use SLF4J APIs in new modules? as a related topic. http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201404.mbox/%3cca+4kjvv_9cmmtdqzcgzy-chslyb1wkgdunxs7wrheslwbuh...@mail.gmail.com%3E It sounds good to me to use asynchronous logging when we log INFO. One concern is that asynchronous logging makes debugging difficult - I don't know log4j 2 well, but I suspect that ordering of logging can be changed even if WARN or FATAL are logged with synchronous logger. Thanks, - Tsuyoshi On Thu, Aug 14, 2014 at 6:44 AM, Arpit Agarwal aagar...@hortonworks.com wrote: I don't recall whether this was discussed before. I often find our INFO logging to be too sparse for useful diagnosis. A high performance logging framework will encourage us to log more. Specifically, Asynchronous Loggers look interesting. https://logging.apache.org/log4j/2.x/manual/async.html#Performance What does the community think of switching to log4j 2 in a Hadoop 2.x release? -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Switch to log4j 2
Using asynchronous loggers for improved performance sounds reasonable. However, IMO we already log too much at INFO level (particularly YARN). Logging more at DEBUG level and lowering the overhead of enabling DEBUG logging is preferable. One concern is the defaults. Based on what I read on the log4j2 page shared, we might want to keep our audit logging synchronous and make all other logging asynchronous. Is there a way to easily configure it this way; otherwise, what is the dev cost we are looking at? On Wed, Aug 13, 2014 at 2:44 PM, Arpit Agarwal aagar...@hortonworks.com wrote: I don't recall whether this was discussed before. I often find our INFO logging to be too sparse for useful diagnosis. A high performance logging framework will encourage us to log more. Specifically, Asynchronous Loggers look interesting. https://logging.apache.org/log4j/2.x/manual/async.html#Performance What does the community think of switching to log4j 2 in a Hadoop 2.x release? -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HADOOP-10971) Flag to make `hadoop fs -ls` print filenames only
Ryan Williams created HADOOP-10971:
--
Summary: Flag to make `hadoop fs -ls` print filenames only
Key: HADOOP-10971
URL: https://issues.apache.org/jira/browse/HADOOP-10971
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Affects Versions: 2.3.0
Reporter: Ryan Williams
It would be useful to have a flag that made {{hadoop fs -ls}} only print
filenames, instead of full {{stat}} info. The {{-C}} flag from GNU {{ls}} is
the closest analog to this behavior that I've found, so I propose that as the
flag.
Per [this stackoverflow answer|http://stackoverflow.com/a/21574829], I've
reluctantly added a {{hadoop-ls-C}} wrapper that expands to {{hadoop fs -ls
$@ | sed 1d | perl -wlne'print +(split ,$_,8)\[7\]'}} to a few projects
I've worked on, and it would obviously be nice to have hadoop save me (and
others) from such hackery.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: [VOTE] Merge fs-encryption branch to trunk
With 4 binding +1s, 3 non-binding +1s, no -1s, the vote passes. Thanks everyone who gave feedback at this stage, particularly Sanjay and Suresh. I should add that this vote will run for the standard 7 days for a non-release vote, so will close at 12PM Pacific on August 15th. On Fri, Aug 8, 2014 at 11:45 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I'd like to call a vote to merge the fs-encryption branch to trunk. Development of this feature has been ongoing since March on HDFS-6134 and HADOOP-10150, totally approximately 50 commits. The fs-encryption branch introduces support for transparent, end-to-end encryption within an encryption zone. Each file stored within an encryption zone is automatically encrypted and decrypted with a unique key. These per-file keys are encrypted with an encryption key only accessible by the client, ensuring that only the client is able to decrypt sensitive data. Furthermore, there is support for native, hardware-accelerated AES encryption. For further details, please see the design doc on HDFS-6134. In terms of merge readiness, we've posted some successful consolidated patches to the JIRA for Jenkins runs. distcp and fs -cp support has also recently been completed, allowing users to securely copy encrypted files without first decrypting them. There is ongoing work to add support for WebHDFS, HttpFS, and other alternative access methods. Stephen Chu has also posted a test plan, and has already identified a few issues that have been fixed. Design and development of this feature was also a cross-company effort with many different contributors. I'd like to thank Charles Lamb, Yi Liu, Uma Maheswara Rao G, Colin McCabe, and Juan Yu for their code contributions and reviews. Alejandro Abdelnur was also instrumental, doing a lot of the design work and as well as writing most of the Hadoop Key Mangement Server (KMS). Finally, I'd like to thank everyone who gave feedback on the JIRAs. This includes Owen, Sanjay, Larry, Mike Y, ATM, Todd, Nicholas, and Andy, among others. With that, here's my +1 to merge this to trunk. Thanks, Andrew
Re: [VOTE] Merge fs-encryption branch to trunk
+1 (binding) We have made some great progress in the last few days on some of the issues I raised. I have posted a summary of the followup items that are needed on the Jira today. I am +1ing expecting the team will complete Items 1 (distcp/cp) and 2 (webhdfs) promptly. Before we publish transparent encryption in a 2.x release for pubic consumption, let us at least complete item 1 (ie distcp and cp) and the flag to turn this feature on/of. This is a great work; thanks team for contributing this important feature. sanjay On Aug 14, 2014, at 1:05 AM, sanjay Radia san...@hortonworks.com wrote: While I was originally skeptical of transparent encryption, I like the value proposition of transparent encryption. HDFS has several layers, protocols and tools. While the HDFS core part seems to be well done in the Jira, inserting the matching transparency in the other tools or protocols need to be worked through. I have the following areas of concern: - Common protocols like webhdfs should continue to work (the design doc marks this as a goal), This issue is being discussed in the Jira but it appears that webhdfs does not currently work with encrypted files: Andrew say that Regarding webhdfs, it's not a recommended deployment and that he will modify the documentation to match that. Aljeandro say Both httpfs and webhdfs will work just fine but then in the same paragraph says this could fail some security audits. We need to resolve this quickly. Webhdfs is heavily used by many Hadoop users. - Common tools should like cp, distcp and HAR should continue to work with non-encrypted and encrypted files in an automatic fashion. This issue has been heavily discussed in the Jira and at the meeting. The /.reserved./.raw mechanism appears to be a step in the right direction for distcp and cp, however this work has not reached its conclusion in my opinion; Charles are I are going through the use cases and I think we are close to a clean solution for distcp and cp. HAR still needs a concrete proposal. - KMS scalability in medium to large clusters. This can perhaps be addressed by getting the keys ahead of time when a job is submitted. Without this the KMS will need to be as highly available and scalable as the NN. I think this is future implementation work but we need to at least determine if this is indeed possible in case we need to modify some of the APIs right now to support that. There are some other minor things under discussion, and I still need to go through the new APIs. Unfortunately at this stage I cannot give a +1 for this merge; I hope to change this in the next day or - I am working with the Jira's team. Alejandoro, Charles, Andrew, Atm, ... to resolve the above as quickly as possible. Sanjay (binding) On Aug 8, 2014, at 11:45 AM, Andrew Wang andrew.w...@cloudera.com wrote: Hi all, I'd like to call a vote to merge the fs-encryption branch to trunk. Development of this feature has been ongoing since March on HDFS-6134 and HADOOP-10150, totally approximately 50 commits. . Thanks, Andrew -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
Thanks for initiating the thread Arun. Can we add YARN-1051 https://issues.apache.org/jira/browse/YARN-1051 to the list? We have most of the patches for the sub-JIRAs under review and have committed a couple. -Subru -- Forwarded message -- From: Arun C Murthy a...@hortonworks.com Date: Tue, Aug 12, 2014 at 1:34 PM Subject: Thinking ahead to hadoop-2.6 To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org, hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org, mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org, yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
