Build failed in Jenkins: Hadoop-Common-0.23-Build #1098
See https://builds.apache.org/job/Hadoop-Common-0.23-Build/1098/ -- [...truncated 8263 lines...] Running org.apache.hadoop.io.TestBloomMapFile Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec Running org.apache.hadoop.io.TestObjectWritableProtos Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.324 sec Running org.apache.hadoop.io.TestTextNonUTF8 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec Running org.apache.hadoop.io.nativeio.TestNativeIO Tests run: 9, Failures: 0, Errors: 0, Skipped: 9, Time elapsed: 0.159 sec Running org.apache.hadoop.io.TestSortedMapWritable Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.193 sec Running org.apache.hadoop.io.TestMapFile Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.631 sec Running org.apache.hadoop.io.TestUTF8 Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.322 sec Running org.apache.hadoop.io.TestBoundedByteArrayOutputStream Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 sec Running org.apache.hadoop.io.retry.TestRetryProxy Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.201 sec Running org.apache.hadoop.io.retry.TestFailoverProxy Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.192 sec Running org.apache.hadoop.io.TestSetFile Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.023 sec Running org.apache.hadoop.io.serializer.TestWritableSerialization Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.32 sec Running org.apache.hadoop.io.serializer.TestSerializationFactory Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.282 sec Running org.apache.hadoop.io.serializer.avro.TestAvroSerialization Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.537 sec Running org.apache.hadoop.util.TestGenericOptionsParser Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.688 sec Running org.apache.hadoop.util.TestReflectionUtils Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec Running org.apache.hadoop.util.TestJarFinder Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.704 sec Running org.apache.hadoop.util.TestPureJavaCrc32 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.3 sec Running org.apache.hadoop.util.TestHostsFileReader Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.184 sec Running org.apache.hadoop.util.TestShutdownHookManager Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.144 sec Running org.apache.hadoop.util.TestDiskChecker Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.487 sec Running org.apache.hadoop.util.TestStringUtils Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.138 sec Running org.apache.hadoop.util.TestGenericsUtil Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.26 sec Running org.apache.hadoop.util.TestAsyncDiskService Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.125 sec Running org.apache.hadoop.util.TestProtoUtil Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec Running org.apache.hadoop.util.TestDataChecksum Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.184 sec Running org.apache.hadoop.util.TestRunJar Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.126 sec Running org.apache.hadoop.util.TestOptions Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec Running org.apache.hadoop.util.TestShell Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.196 sec Running org.apache.hadoop.util.TestIndexedSort Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.625 sec Running org.apache.hadoop.util.TestStringInterner Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 sec Running org.apache.hadoop.record.TestRecordVersioning Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.142 sec Running org.apache.hadoop.record.TestBuffer Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec Running org.apache.hadoop.record.TestRecordIO Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.161 sec Running org.apache.hadoop.security.TestGroupFallback Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.424 sec Running org.apache.hadoop.security.TestGroupsCaching Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.278 sec Running org.apache.hadoop.security.TestProxyUserFromEnv Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.366 sec Running org.apache.hadoop.security.TestUserGroupInformation Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.662 sec Running org.apache.hadoop.security.TestJNIGroupsMapping Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.138 sec Running
[jira] [Created] (HADOOP-11188) hadoop-azure: automatically expand page blobs when they become full
Eric Hanson created HADOOP-11188: Summary: hadoop-azure: automatically expand page blobs when they become full Key: HADOOP-11188 URL: https://issues.apache.org/jira/browse/HADOOP-11188 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Eric Hanson Right now, page blobs are initialized to a fixed size (fs.azure.page.blob.size) and cannot be expanded. This task is to make them automatically expand when they get to be nearly full. Design: if a write occurs that does not have enough room in the file to finish, then flush all preceding operations, extend the file, and complete the write. This will be synchronized (to have exclusive access) in access to PageBlobOutputStream so there won't be race conditions. The file will be extended by fs.azure.page.blob.extension.size bytes, which must be a multiple of 512. The internal default for fs.azure.page.blob.extension size will be 128 * 1024 * 1024. The minimum extension size will be 4 * 1024 * 1024 which is the maximum write size, so the new write will finish. Extension will stop when the file size reaches 1TB. The final extension may be less than fs.azure.page.blob.extension.size if the remainder (1TB - current_file_size) is smaller than fs.azure.page.blob.extension.size. An alternative to this is to make the default size 1TB. This is much simpler to implement. It's a one-line change. Or even simpler, don't change it at all because it is adequate for HBase. Rationale for this file size extension feature: 1) be able to download files to local disk easily with CloudXplorer and similar tools. Downloading a 1TB page blob is not practical if you don't have 1TB disk space since on the local side it expands to the full file size, locally filled with zeros where there is no valid data. 2) don't make customers uncomfortable when they see large 1TB files. They often ask if they have to pay for it, even though they only pay for the space actually used in the page blob. I think rationale 2 is a relatively minor issue, because 98% of customers for HBase will never notice. They will just use it and not look at what kind of files are used for the logs. They don't pay for the unused space, so it is not a problem for them. We can document this. Also, if they use hadoop fs -ls, they will see the actual size of the files since I put in a fix for that. Rationale 1 is a minor issue because you cannot interpret the data on your local file system anyway due to the data format. So really, the only reason to copy data locally in its binary format would be if you are moving it around or archiving it. Copying a 1TB page blob from one location in the cloud to another is pretty fast with smart copy utilities that don't actually move the 0-filled parts of the file. Nevertheless, this is a convenience feature for users. They won't have to worry about setting fs.azure.page.blob.size under normal circumstances and can make the files grow as big as they want. If we make the change to extend the file size on the fly, that introduces new possible error or failure modes for HBase. We should included retry logic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11189) TestDNFencing.testQueueingWithAppend failed often in latest test
Yongjun Zhang created HADOOP-11189:
--
Summary: TestDNFencing.testQueueingWithAppend failed often in
latest test
Key: HADOOP-11189
URL: https://issues.apache.org/jira/browse/HADOOP-11189
Project: Hadoop Common
Issue Type: Bug
Components: ha
Reporter: Yongjun Zhang
Using tool from HADOOP-11045, got the following report:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
PreCommit-HDFS-Build -n 1
Recently FAILED builds in url:
https://builds.apache.org//job/PreCommit-HDFS-Build
THERE ARE 9 builds (out of 9) that have failed tests in the past 1 days, as
listed below:
===https://builds.apache.org/job/PreCommit-HDFS-Build/8390/testReport
(2014-10-10 05:20:58)
Failed test:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
Failed test:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.testFencingStress
Failed test:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testOpenFilesWithMultipleSnapshots
===https://builds.apache.org/job/PreCommit-HDFS-Build/8389/testReport
(2014-10-10 01:10:58)
No failed tests in testReport, check job's Console Output for why it was
reported failed
===https://builds.apache.org/job/PreCommit-HDFS-Build/8388/testReport
(2014-10-10 00:30:54)
Failed test:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
Failed test:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.testFencingStress
..
Among 9 runs examined, all failed tests #failedRuns: testName:
7:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
6:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.testFencingStress
3:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testOpenFilesWithMultipleSnapshots
1: org.apache.hadoop.hdfs.server.namenode.TestEditLog.testFailedOpen
1: org.apache.hadoop.hdfs.server.namenode.TestEditLog.testSyncBatching
..
{code}
TestDNFencingWithReplication.testFencingStress was reported as HDFS-7221.
Creating this jira for TestDNFencing.testQueueingWithAppend.
Symptom:
{code}
Failed
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
Failing for the past 1 build (Since Failed#8390 )
Took 2.9 sec.
Error Message
expected:18 but was:12
Stacktrace
java.lang.AssertionError: expected:18 but was:12
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend(TestDNFencing.java:448)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11189) TestDNFencing.testQueueingWithAppend failed often in latest test
[
https://issues.apache.org/jira/browse/HADOOP-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang resolved HADOOP-11189.
Resolution: Duplicate
TestDNFencing.testQueueingWithAppend failed often in latest test
Key: HADOOP-11189
URL: https://issues.apache.org/jira/browse/HADOOP-11189
Project: Hadoop Common
Issue Type: Bug
Components: ha
Reporter: Yongjun Zhang
Using tool from HADOOP-11045, got the following report:
{code}
[yzhang@localhost jenkinsftf]$ ./determine-flaky-tests-hadoop.py -j
PreCommit-HDFS-Build -n 1
Recently FAILED builds in url:
https://builds.apache.org//job/PreCommit-HDFS-Build
THERE ARE 9 builds (out of 9) that have failed tests in the past 1 days,
as listed below:
..
Among 9 runs examined, all failed tests #failedRuns: testName:
7:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
6:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication.testFencingStress
3:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot.testOpenFilesWithMultipleSnapshots
1: org.apache.hadoop.hdfs.server.namenode.TestEditLog.testFailedOpen
1: org.apache.hadoop.hdfs.server.namenode.TestEditLog.testSyncBatching
..
{code}
TestDNFencingWithReplication.testFencingStress was reported as HDFS-7221.
Creating this jira for TestDNFencing.testQueueingWithAppend.
Symptom:
{code}
Failed
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend
Failing for the past 1 build (Since Failed#8390 )
Took 2.9 sec.
Error Message
expected:18 but was:12
Stacktrace
java.lang.AssertionError: expected:18 but was:12
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencing.testQueueingWithAppend(TestDNFencing.java:448)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11190) Potentially stale value is used in SelfRenewingLease ctor
Ted Yu created HADOOP-11190:
---
Summary: Potentially stale value is used in SelfRenewingLease ctor
Key: HADOOP-11190
URL: https://issues.apache.org/jira/browse/HADOOP-11190
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is w.r.t. threadNumber, shown in the code around line 102:
{code}
renewer.setName(AzureLeaseRenewer- + threadNumber++);
{code}
Since there is no synchronization involved, potentially stale value may be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11191) NativeAzureFileSystem#close() should be synchronized
Ted Yu created HADOOP-11191:
---
Summary: NativeAzureFileSystem#close() should be synchronized
Key: HADOOP-11191
URL: https://issues.apache.org/jira/browse/HADOOP-11191
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void close() throws IOException {
in.close();
closed = true;
}
{code}
The other methods, such as seek(), are synchronized.
close() should be as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11192) Change old subversion links to git
Ravi Prakash created HADOOP-11192: - Summary: Change old subversion links to git Key: HADOOP-11192 URL: https://issues.apache.org/jira/browse/HADOOP-11192 Project: Hadoop Common Issue Type: Bug Reporter: Ravi Prakash e.g. hadoop-project/src/site/site.xml still references SVN. We should probably check our wiki's and other documentation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11193) Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_fstat returns uninitialized mode on Windows
Xiaoyu Yao created HADOOP-11193: --- Summary: Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_fstat returns uninitialized mode on Windows Key: HADOOP-11193 URL: https://issues.apache.org/jira/browse/HADOOP-11193 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.5.1 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao This was caught by TestNativeIO#testFstat test. Looks like an uninitialized variable mode below in NativeIO.c#Java_org_apache_hadoop_io_nativeio_NativeIO_00024POSIX_fstat around line ~278. ifdef WINDOWS LPWSTR owner = NULL; LPWSTR group = NULL; int mode; jstring jstr_owner = NULL; ... Test Report: --- T E S T S --- --- T E S T S --- Running org.apache.hadoop.io.nativeio.TestNativeIO Tests run: 18, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 2.214 sec FAILURE! - in org.apache.hadoop.io.native io.TestNativeIO testFstat(org.apache.hadoop.io.nativeio.TestNativeIO) Time elapsed: 0.514 sec FAILURE! java.lang.AssertionError: Stat mode field should indicate a regular file expected:32768 but was:53248 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.io.nativeio.TestNativeIO.testFstat(TestNativeIO.java:96) Results : Failed tests: TestNativeIO.testFstat:96 Stat mode field should indicate a regular file expected:32768 but was:53248 Tests run: 18, Failures: 1, Errors: 0, Skipped: 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
