Update: checkstyle, shellcheck, and whitespace tests

2015-04-30 Thread Allen Wittenauer 

Hey gang.

With the commit of HADOOP-11866, the output for checkstyle and 
whitespace has been greatly enhanced to actually be useful now.  Instead of 
getting weird output, you'll see the file, the line number and (in the case of 
checkstyle) error.  Note that line numbers are after the patch applied. So any 
fuzzing done by patch will offset the line numbers if your source tree isn't up 
to date.

Shellcheck output has also been fixed to show errors that might have 
been missed before.  Additionally, I've filed an INFRA ticket to try to get 
shellcheck installed on the Jenkins nodes so that we can start seeing errors 
from it as well.

Thanks.

Re: Planning Hadoop 2.6.1 release

2015-04-30 Thread Chris Nauroth 
Thank you, Arpit.  In addition, I suggest we include the following:

HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread.
HADOOP-11648. Set DomainSocketWatcher thread name explicitly
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm

HADOOP-11604 and 11648 are not critical by themselves, but they are
pre-requisites to getting a clean cherry-pick of 11802, which we believe
finally fixes the root cause of this issue.


--Chris Nauroth




On 4/30/15, 3:55 PM, "Arpit Agarwal"  wrote:

>HDFS candidates for back-porting to Hadoop 2.6.1. The first two were
>requested in [1].
>
>HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
>should be non static
>HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
>synchronization
>
>HDFS-7009. Active NN and standby NN have different live nodes.
>HDFS-7035. Make adding a new data directory to the DataNode an atomic and
>improve error handling
>HDFS-7425. NameNode block deletion logging uses incorrect appender.
>HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
>block files are present in the same volume.
>HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
>HDFS-7503. Namenode restart after large deletions can cause slow
>processReport.
>HDFS-7575. Upgrade should generate a unique storage ID for each volume.
>HDFS-7579. Improve log reporting during block report rpc failure.
>HDFS-7587. Edit log corruption can happen if append fails with a quota
>violation.
>HDFS-7596. NameNode should prune dead storages from storageMap.
>HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
>in the blocksMap on NameNode restart.
>HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
>DataNode to register successfully with only one NameNode.
>HDFS-7733. NFS: readdir/readdirplus return null directory attribute on
>failure.
>HDFS-7831. Fix the starting index and end condition of the loop in
>FileDiffList.findEarlierSnapshotBlocks().
>HDFS-7885. Datanode should not trust the generation stamp provided by
>client.
>HDFS-7960. The full block report should prune zombie storages even if
>they're not empty.
>HDFS-8072. Reserved RBW space is not released if client terminates while
>writing block.
>HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
>finalize upgrade.
>
>
>Arpit
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>
>
>
>On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli" 
>wrote:
>
>>There were several requests on the user lists [1] for a 2.6.1 release. I
>>got many offline comments too.
>>
>>Planning to do a 2.6.1 release in a few weeks time. We already have a
>>bunch
>>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>>tickets.
>>
>>We need to collectively come up with a list of critical issues. We can
>>use
>>the JIRA Target Version field for the same. I see some but not a whole
>>lot
>>of new work for this release, most of it is likely going to be pulling in
>>critical patches from 2.7.1/2.8 etc.
>>
>>Thoughts?
>>
>>Thanks
>>+Vinod
>>
>>[1] Will Hadoop 2.6.1 be released soon?
>>http://markmail.org/thread/zlsr6prejyogdyvh
>>[2] 2.6.1 pending tickets
>>https://issues.apache.org/jira/issues/?filter=12331711
>>
>
>

Re: Planning Hadoop 2.6.1 release

2015-04-30 Thread Arpit Agarwal 
HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested 
in [1].

HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be 
non static
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt 
synchronization

HDFS-7009. Active NN and standby NN have different live nodes.
HDFS-7035. Make adding a new data directory to the DataNode an atomic and 
improve error handling
HDFS-7425. NameNode block deletion logging uses incorrect appender.
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block 
files are present in the same volume.
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
HDFS-7503. Namenode restart after large deletions can cause slow processReport.
HDFS-7575. Upgrade should generate a unique storage ID for each volume.
HDFS-7579. Improve log reporting during block report rpc failure.
HDFS-7587. Edit log corruption can happen if append fails with a quota 
violation.
HDFS-7596. NameNode should prune dead storages from storageMap.
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the 
blocksMap on NameNode restart.
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode 
to register successfully with only one NameNode.
HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure.
HDFS-7831. Fix the starting index and end condition of the loop in 
FileDiffList.findEarlierSnapshotBlocks().
HDFS-7885. Datanode should not trust the generation stamp provided by client.
HDFS-7960. The full block report should prune zombie storages even if they're 
not empty.
HDFS-8072. Reserved RBW space is not released if client terminates while 
writing block.
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize 
upgrade.


Arpit

[1] Will Hadoop 2.6.1 be released soon? 
http://markmail.org/thread/zlsr6prejyogdyvh



On 4/27/15, 11:47 AM, "Vinod Kumar Vavilapalli"  wrote:

>There were several requests on the user lists [1] for a 2.6.1 release. I
>got many offline comments too.
>
>Planning to do a 2.6.1 release in a few weeks time. We already have a bunch
>of tickets committed to 2.7.1. I created a filter [2] to tracking pending
>tickets.
>
>We need to collectively come up with a list of critical issues. We can use
>the JIRA Target Version field for the same. I see some but not a whole lot
>of new work for this release, most of it is likely going to be pulling in
>critical patches from 2.7.1/2.8 etc.
>
>Thoughts?
>
>Thanks
>+Vinod
>
>[1] Will Hadoop 2.6.1 be released soon?
>http://markmail.org/thread/zlsr6prejyogdyvh
>[2] 2.6.1 pending tickets
>https://issues.apache.org/jira/issues/?filter=12331711
>

[jira] [Created] (HADOOP-11895) Support running a specific subset of tests in test-patch.sh.

2015-04-30 Thread Chris Nauroth (JIRA) 
Chris Nauroth created HADOOP-11895:
--

 Summary: Support running a specific subset of tests in 
test-patch.sh.
 Key: HADOOP-11895
 URL: https://issues.apache.org/jira/browse/HADOOP-11895
 Project: Hadoop Common
  Issue Type: Improvement
  Components: scripts
Reporter: Chris Nauroth


test-patch.sh currently runs {{mvn test}} in the sub-modules that are changed 
by the patch.  Due to long-running tests, this can slow down development for 
certain kinds of patches.  This issue proposes an enhancement to test-patch.sh 
to support a faster mode of execution that applies the patch and runs {{mvn 
test -Dtest=...}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-common-trunk-Java8 #182

2015-04-30 Thread Apache Jenkins Server 
See

[jira] [Created] (HADOOP-11894) Bump the version of HTrace to 3.2.0-incubating

2015-04-30 Thread Masatake Iwasaki (JIRA) 
Masatake Iwasaki created HADOOP-11894:
-

 Summary: Bump the version of HTrace to 3.2.0-incubating
 Key: HADOOP-11894
 URL: https://issues.apache.org/jira/browse/HADOOP-11894
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Masatake Iwasaki
Assignee: Masatake Iwasaki


* update pom.xml
* update documentation
* replace {{addKVAnnotation(byte[] key, byte[] value)}} with 
{{addKVAnnotation(String key, String value)}}
* replace {{SpanReceiverHost#getUniqueLocalTraceFileName}} with 
{{LocalFileSpanReceiver#getUniqueLocalTraceFileName}}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Hadoop-Common-trunk #1481

2015-04-30 Thread Apache Jenkins Server 
See 

Changes:

[jing9] HDFS-8283. DataStreamer cleanup and some minor improvement. Contributed 
by Tsz Wo Nicholas Sze.

[wheat9] HDFS-8269. getBlockLocations() does not resolve the .reserved path and 
generates incorrect edit logs when updating the atime. Contributed by Haohui 
Mai.

[tgraves] YARN-3517. RM web ui for dumping scheduler logs should be for admins 
only (Varun Vasudev via tgraves)

[jianhe] YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be 
scheduled. Contributed by Anubhav Dhoot

[wang] HDFS-8214. Secondary NN Web UI shows wrong date for Last Checkpoint. 
Contributed by Charles Lamb.

[devaraj] MAPREDUCE-6339. Job history file is not flushed correctly because

--
[...truncated 5122 lines...]
Running org.apache.hadoop.util.TestFileBasedIPList
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.181 sec - in 
org.apache.hadoop.util.TestFileBasedIPList
Running org.apache.hadoop.util.TestGenericsUtil
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.338 sec - in 
org.apache.hadoop.util.TestGenericsUtil
Running org.apache.hadoop.util.TestFindClass
Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.778 sec - in 
org.apache.hadoop.util.TestFindClass
Running org.apache.hadoop.util.TestLightWeightGSet
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.178 sec - in 
org.apache.hadoop.util.TestLightWeightGSet
Running org.apache.hadoop.util.TestPureJavaCrc32
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.581 sec - in 
org.apache.hadoop.util.TestPureJavaCrc32
Running org.apache.hadoop.util.TestStringUtils
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.339 sec - in 
org.apache.hadoop.util.TestStringUtils
Running org.apache.hadoop.util.TestProtoUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.267 sec - in 
org.apache.hadoop.util.TestProtoUtil
Running org.apache.hadoop.util.TestSignalLogger
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.187 sec - in 
org.apache.hadoop.util.TestSignalLogger
Running org.apache.hadoop.util.TestDiskChecker
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.217 sec - in 
org.apache.hadoop.util.TestDiskChecker
Running org.apache.hadoop.util.TestShutdownThreadsHelper
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.18 sec - in 
org.apache.hadoop.util.TestShutdownThreadsHelper
Running org.apache.hadoop.util.TestCacheableIPList
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.282 sec - in 
org.apache.hadoop.util.TestCacheableIPList
Running org.apache.hadoop.util.TestLineReader
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.207 sec - in 
org.apache.hadoop.util.TestLineReader
Running org.apache.hadoop.util.TestIdentityHashStore
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.197 sec - in 
org.apache.hadoop.util.TestIdentityHashStore
Running org.apache.hadoop.util.TestClasspath
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.34 sec - in 
org.apache.hadoop.util.TestClasspath
Running org.apache.hadoop.util.TestApplicationClassLoader
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.247 sec - in 
org.apache.hadoop.util.TestApplicationClassLoader
Running org.apache.hadoop.util.TestShell
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.23 sec - in 
org.apache.hadoop.util.TestShell
Running org.apache.hadoop.util.TestShutdownHookManager
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec - in 
org.apache.hadoop.util.TestShutdownHookManager
Running org.apache.hadoop.util.TestHttpExceptionUtils
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.692 sec - in 
org.apache.hadoop.util.TestHttpExceptionUtils
Running org.apache.hadoop.util.TestJarFinder
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.78 sec - in 
org.apache.hadoop.util.TestJarFinder
Running org.apache.hadoop.util.hash.TestHash
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.462 sec - in 
org.apache.hadoop.util.hash.TestHash
Running org.apache.hadoop.util.TestLightWeightCache
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.778 sec - in 
org.apache.hadoop.util.TestLightWeightCache
Running org.apache.hadoop.util.TestNativeCodeLoader
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.174 sec - in 
org.apache.hadoop.util.TestNativeCodeLoader
Running org.apache.hadoop.util.TestReflectionUtils
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.651 sec - in 
org.apache.hadoop.util.TestReflectionUtils
Running org.apache.hadoop.crypto.TestCryptoStreams
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.174 sec - 
in org.apache.hadoop.crypto.TestCryptoStreams
Running org.apache.hadoop.crypto.TestCryptoStreamsForLoca