[jira] [Commented] (HDFS-8373) Ec files can't be deleted into Trash because of that Trash isn't EC zone.
[
https://issues.apache.org/jira/browse/HDFS-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711665#comment-14711665
]
Zhe Zhang commented on HDFS-8373:
-
Good catch [~brahmareddy]. Once HDFS-8833 is finalized let's close this one as
duplicate.
Ec files can't be deleted into Trash because of that Trash isn't EC zone.
-
Key: HDFS-8373
URL: https://issues.apache.org/jira/browse/HDFS-8373
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: GAO Rui
Assignee: Brahma Reddy Battula
Labels: EC
When EC files were deleted, they would be moved into {{Trash}} directory.
But, EC files can only be placed under EC zone. So, EC files which have been
deleted can not be moved to {{Trash}} directory.
Problem could be solved by creating a EC zone(floder) inside {{Trash}} to
contain deleted EC files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8943) Read apis in ByteRangeInputStream does not read all the bytes specified when chunked transfer-encoding is used in the server
[
https://issues.apache.org/jira/browse/HDFS-8943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711664#comment-14711664
]
Shradha Revankar commented on HDFS-8943:
Thanks, I've created this https://issues.apache.org/jira/browse/HBASE-14307
Read apis in ByteRangeInputStream does not read all the bytes specified when
chunked transfer-encoding is used in the server
Key: HDFS-8943
URL: https://issues.apache.org/jira/browse/HDFS-8943
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
Attachments: HDFS-8943.000.patch
With the default Webhdfs server implementation the read apis in
ByteRangeInputStream work as expected reading the correct number of bytes for
these apis :
{{public int read(byte b[], int off, int len)}}
{{public int read(long position, byte[] buffer, int offset, int length)}}
But when a custom Webhdfs server implementation is plugged in which uses
chunked Transfer-encoding, these apis read only the first chunk. Simple fix
would be to loop and read till bytes specified similar to {{readfully()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8915) TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
[ https://issues.apache.org/jira/browse/HDFS-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711680#comment-14711680 ] Anu Engineer commented on HDFS-8915: This was a tracking bug to see if this failure happens in the apache tree. Since the only failure was reported when we had a VM outage, resolving this issue as won't fix. This test has been running without any issues for a week. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins - Key: HDFS-8915 URL: https://issues.apache.org/jira/browse/HDFS-8915 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.8.0 Reporter: Anu Engineer Assignee: Anu Engineer This test was added as part of HDFS-8883, There is a race condition in the test and it has failed *once* in the Apache Jenkins run. Here is the stack FAILED: org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount Error Message: Expected number of blocked thread not found expected:3 but was:1 Stack Trace: java.lang.AssertionError: Expected number of blocked thread not found expected:3 but was:1 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261) From cursory code reading , even though we call into readlock.lock() there is no guarantee that our thread is put in the wait queue. A proposed fix could be to check for any thread in the lock queue instead of all 3, or disable the test. It could also indicate an issue with the test infra-structure but any test open to variations in result due to infra-structure issues creates noise in tests so we are better off fixing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-1148) Convert FSDataset to ReadWriteLock
[ https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yong Zhang updated HDFS-1148: - Attachment: HDFS-1148.003.patch Convert FSDataset to ReadWriteLock -- Key: HDFS-1148 URL: https://issues.apache.org/jira/browse/HDFS-1148 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, performance Reporter: Todd Lipcon Assignee: Yong Zhang Attachments: HDFS-1148.001.patch, HDFS-1148.002.patch, HDFS-1148.003.patch, hdfs-1148-old.txt, hdfs-1148-trunk.txt, patch-HDFS-1148-rel0.20.2.txt In benchmarking HDFS-941 I noticed that for the random read workload, the FSDataset lock is highly contended. After converting it to a ReentrantReadWriteLock, I saw a ~25% improvement on both latency and ops/second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: HDFS-8894-01.patch Set SO_KEEPALIVE on DN server sockets - Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts Assignee: kanaka kumar avvaru Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch SO_KEEPALIVE is not set on things like datastreamer sockets which can cause lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Create edit log files with old layout version for upgrade testing
[
https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711658#comment-14711658
]
Zhe Zhang commented on HDFS-8846:
-
The change is test-only and is not related to {{TestEditLog}}. It also passes
locally.
Create edit log files with old layout version for upgrade testing
-
Key: HDFS-8846
URL: https://issues.apache.org/jira/browse/HDFS-8846
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.1
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch,
HDFS-8846.02.patch, HDFS-8846.03.patch
Per discussion under HDFS-8480, we should create some edit log files with old
layout version, to test whether they can be correctly handled in upgrades.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-8915) TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins
[ https://issues.apache.org/jira/browse/HDFS-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer resolved HDFS-8915. Resolution: Invalid TestFSNamesystem.testFSLockGetWaiterCount fails intermittently in jenkins - Key: HDFS-8915 URL: https://issues.apache.org/jira/browse/HDFS-8915 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Affects Versions: 2.8.0 Reporter: Anu Engineer Assignee: Anu Engineer This test was added as part of HDFS-8883, There is a race condition in the test and it has failed *once* in the Apache Jenkins run. Here is the stack FAILED: org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount Error Message: Expected number of blocked thread not found expected:3 but was:1 Stack Trace: java.lang.AssertionError: Expected number of blocked thread not found expected:3 but was:1 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.apache.hadoop.hdfs.server.namenode.TestFSNamesystem.testFSLockGetWaiterCount(TestFSNamesystem.java:261) From cursory code reading , even though we call into readlock.lock() there is no guarantee that our thread is put in the wait queue. A proposed fix could be to check for any thread in the lock queue instead of all 3, or disable the test. It could also indicate an issue with the test infra-structure but any test open to variations in result due to infra-structure issues creates noise in tests so we are better off fixing it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-8938:
Attachment: HDFS-8938.005.patch
Refactor BlockManager in blockmanagement
Key: HDFS-8938
URL: https://issues.apache.org/jira/browse/HDFS-8938
Project: Hadoop HDFS
Issue Type: Task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch,
HDFS-8938.005.patch
This lira tracks the effort of refactoring the {{BlockManager}} in
{{hdfs.server.blockmanagement}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6515) testPageRounder (org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
[
https://issues.apache.org/jira/browse/HDFS-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711289#comment-14711289
]
Hadoop QA commented on HDFS-6515:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 34s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 36s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 1s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 161m 42s | Tests failed in hadoop-hdfs. |
| | | 231m 44s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12677586/HDFS-6515-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12112/console |
This message was automatically generated.
testPageRounder (org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
-
Key: HDFS-6515
URL: https://issues.apache.org/jira/browse/HDFS-6515
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0, 2.4.0, 2.4.1
Environment: Linux on PPC64
Tested with Hadoop 3.0.0 SNAPSHOT, on RHEL 6.5, on Ubuntu 14.04, on Fedora
19, using mvn -Dtest=TestFsDatasetCache#testPageRounder -X test
Reporter: Tony Reix
Priority: Blocker
Labels: BB2015-05-TBR, hadoop, test
Attachments: HDFS-6515-1.patch, HDFS-6515-2.patch
I have an issue with test :
testPageRounder
(org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
on Linux/PowerPC.
On Linux/Intel, test runs fine.
On Linux/PowerPC, I have:
testPageRounder(org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache)
Time elapsed: 64.037 sec ERROR!
java.lang.Exception: test timed out after 6 milliseconds
Looking at details, I see that some Failed to cache messages appear in the
traces. Only 10 on Intel, but 186 on PPC64.
On PPC64, it looks like some thread is waiting for something that never
happens, generating a TimeOut.
I'm now using IBM JVM, however I've just checked that the issue also appears
with OpenJDK.
I'm now using Hadoop latest, however, the issue appeared within Hadoop 2.4.0 .
I need help for understanding what the test is doing, what traces are
expected, in order to understand what/where is the root cause.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711326#comment-14711326
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #296 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/296/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[
https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711399#comment-14711399
]
Hadoop QA commented on HDFS-8581:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 52s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 24s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 9s | Tests passed in hadoop-hdfs.
|
| | | 207m 24s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752211/HDFS-8581.2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12115/console |
This message was automatically generated.
count cmd calculate wrong when huge files exist in one folder
-
Key: HDFS-8581
URL: https://issues.apache.org/jira/browse/HDFS-8581
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Reporter: tongshiquan
Assignee: J.Andreina
Priority: Minor
Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch
If one directory such as /result exists about 20 files, then when
execute hdfs dfs -count /, the result will go wrong. For all directories
whose name after /result, file num will not be included.
My cluster see as below, /result_1433858936 is the directory exist huge
files, and files in /sparkJobHistory, /tmp, /user are not included
vm-221:/export1/BigData/current # hdfs dfs -ls /
15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled.
Found 9 items
-rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10
/PRE_CREATE_DIR.SUCCESS
drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume
drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase
drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt
drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history
drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10
/result_1433858936
drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory
drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp
drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user
vm-221:/export1/BigData/current #
vm-221:/export1/BigData/current # hdfs dfs -count /
15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled.
1043 171536 1756375688 /
vm-221:/export1/BigData/current #
vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS
15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled.
01 0 /PRE_CREATE_DIR.SUCCESS
vm-221:/export1/BigData/current #
vm-221:/export1/BigData/current # hdfs dfs -count /flume
15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled.
10 0 /flume
vm-221:/export1/BigData/current #
[jira] [Updated] (HDFS-7765) FSOutputSummer throwing ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HDFS-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-7765:
-
Assignee: Janmejay Singh (was: Surendra Singh Lilhore)
FSOutputSummer throwing ArrayIndexOutOfBoundsException
--
Key: HDFS-7765
URL: https://issues.apache.org/jira/browse/HDFS-7765
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.6.0
Environment: Centos 6, Open JDK 7, Amazon EC2, Accumulo 1.6.2RC4
Reporter: Keith Turner
Assignee: Janmejay Singh
Attachments:
0001-PATCH-HDFS-7765-FSOutputSummer-throwing-ArrayIndexOu.patch,
HDFS-7765.patch
While running an Accumulo test, saw exceptions like the following while
trying to write to write ahead log in HDFS.
The exception occurrs at
[FSOutputSummer.java:76|https://github.com/apache/hadoop/blob/release-2.6.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSOutputSummer.java#L76]
which is attempting to update a byte array.
{noformat}
2015-02-06 19:46:49,769 [log.DfsLogger] WARN : Exception syncing
java.lang.reflect.InvocationTargetException
java.lang.ArrayIndexOutOfBoundsException: 4608
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
at java.io.DataOutputStream.write(DataOutputStream.java:88)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
at
org.apache.accumulo.tserver.logger.LogFileKey.write(LogFileKey.java:87)
at org.apache.accumulo.tserver.log.DfsLogger.write(DfsLogger.java:526)
at
org.apache.accumulo.tserver.log.DfsLogger.logFileData(DfsLogger.java:540)
at
org.apache.accumulo.tserver.log.DfsLogger.logManyTablets(DfsLogger.java:573)
at
org.apache.accumulo.tserver.log.TabletServerLogger$6.write(TabletServerLogger.java:373)
at
org.apache.accumulo.tserver.log.TabletServerLogger.write(TabletServerLogger.java:274)
at
org.apache.accumulo.tserver.log.TabletServerLogger.logManyTablets(TabletServerLogger.java:365)
at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:1667)
at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.closeUpdate(TabletServer.java:1754)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.trace.instrument.thrift.RpcServerInvocationHandler.invoke(RpcServerInvocationHandler.java:46)
at
org.apache.accumulo.server.util.RpcWrapper$1.invoke(RpcWrapper.java:47)
at com.sun.proxy.$Proxy22.closeUpdate(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2370)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$closeUpdate.getResult(TabletClientService.java:2354)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:168)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:516)
at
org.apache.accumulo.server.util.CustomNonBlockingServer$1.run(CustomNonBlockingServer.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:744)
2015-02-06 19:46:49,769 [log.DfsLogger] WARN : Exception syncing
java.lang.reflect.InvocationTargetException
2015-02-06 19:46:49,772 [log.DfsLogger] ERROR:
java.lang.ArrayIndexOutOfBoundsException: 4609
java.lang.ArrayIndexOutOfBoundsException: 4609
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:76)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:50)
at java.io.DataOutputStream.write(DataOutputStream.java:88)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
at
org.apache.accumulo.tserver.logger.LogFileKey.write(LogFileKey.java:87)
at
[jira] [Commented] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711437#comment-14711437
]
Hadoop QA commented on HDFS-8771:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 47s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 1
new checkstyle issues (total was 3, now 4). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 26s | Tests failed in hadoop-hdfs. |
| | | 207m 47s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFSNamesystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752216/HDFS-8771-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12116/console |
This message was automatically generated.
If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
send another RPC calls to Journalnodes
Key: HDFS-8771
URL: https://issues.apache.org/jira/browse/HDFS-8771
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takuya Fukudome
Assignee: kanaka kumar avvaru
Attachments: HDFS-8771-01.patch
In our cluster, edits has became huge(about 50GB) accidentally and our
Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
Namenode couldn't send other RPC calls to Journalnodes because
{{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
It will cause namenode shutting down.
I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712445#comment-14712445
]
Hadoop QA commented on HDFS-8951:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 28s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 31s | The applied patch generated
63 new checkstyle issues (total was 16, now 79). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 27s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 192m 47s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 243m 54s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752353/HDFS-8951.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12127/console |
This message was automatically generated.
Move shortcircuit to hdfs-client
Key: HDFS-8951
URL: https://issues.apache.org/jira/browse/HDFS-8951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch,
HDFS-8951.002.patch
This jira tracks the effort of moving the {{shortcircuit}} package into the
hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[
https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712448#comment-14712448
]
Hadoop QA commented on HDFS-8950:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 13s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 8s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 166m 42s | Tests failed in hadoop-hdfs. |
| | | 211m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752359/HDFS-8950.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a4d9acc |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12128/console |
This message was automatically generated.
NameNode refresh doesn't remove DataNodes
-
Key: HDFS-8950
URL: https://issues.apache.org/jira/browse/HDFS-8950
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, namenode
Affects Versions: 2.6.0
Reporter: Daniel Templeton
Assignee: Daniel Templeton
Fix For: 2.8.0
Attachments: HDFS-8950.001.patch, HDFS-8950.002.patch
If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN
refresh, it doesn't remove it actually and the NN UI keeps showing that node.
It may try to allocate some blocks to that DN as well during an MR job. This
issue is independent from DN decommission.
To reproduce:
1. Add a DN to dfs_hosts_allow
2. Refresh NN
3. Start DN. Now NN starts seeing DN.
4. Stop DN
5. Remove DN from dfs_hosts_allow
6. Refresh NN - NN is still reporting DN as being used by HDFS.
This is different from decom because there DN is added to exclude list in
addition to being removed from allowed list, and in that case everything
works correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
Xu Chen created HDFS-8954: - Summary: Not all the one block in RAM_DISK use lazy_persist policy Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , is that a bug ? or my config issue? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8955) Support 'hedged' write in DFSClient
bijaya created HDFS-8955: Summary: Support 'hedged' write in DFSClient Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: bijaya Assignee: Liang Xie Fix For: 2.4.0 This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize dfs.dfsclient.quorum.read.threshold.millis dfs.dfsclient.quorum.read.threadpool.size to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Release Note: If a write from a block is slow, start up another parallel, 'hedged' write against a different set of replica. We need to get different set of replica/data pipeline from NN. We then take the result of which ever write returns first (the outstanding write is cancelled). This 'hedged' write feature will help rein in the outliers, the odd write that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set codedfs.client.hedged.write.threadpool.size/code to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent writes in your client. Then set codedfs.client.hedged.write.threshold.millis/code to the number of milliseconds to wait before starting up a 'hedged' write. For example, if you set this property to 10, then if a write has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedWriteOps + hedgeWriteOpsWin -- how many times the hedged write 'beat' the original write + hedgedWriteOpsInCurThread -- how many times we went to do a hedged write but we had to run it in the current thread because dfs.client.hedged.write.threadpool.size was at a maximum. was: If a read from a block is slow, start up another parallel, 'hedged' read against a different block replica. We then take the result of which ever read returns first (the outstanding read is cancelled). This 'hedged' read feature will help rein in the outliers, the odd read that takes a long time because it hit a bad patch on the disc, etc. This feature is off by default. To enable this feature, set codedfs.client.hedged.read.threadpool.size/code to a positive number. The threadpool size is how many threads to dedicate to the running of these 'hedged', concurrent reads in your client. Then set codedfs.client.hedged.read.threshold.millis/code to the number of milliseconds to wait before starting up a 'hedged' read. For example, if you set this property to 10, then if a read has not returned within 10 milliseconds, we will start up a new read against a different block replica. This feature emits new metrics: + hedgedReadOps + hedgeReadOpsWin -- how many times the hedged read 'beat' the original read + hedgedReadOpsInCurThread -- how many times we went to do a hedged read but we had to run it in the current thread because dfs.client.hedged.read.threadpool.size was at a maximum. Support 'hedged' write in DFSClient --- Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: bijaya Assignee: bijaya Fix For: 2.4.0 This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize dfs.dfsclient.quorum.read.threshold.millis dfs.dfsclient.quorum.read.threadpool.size to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8956) Not able to start Datanode
sreelakshmi created HDFS-8956: - Summary: Not able to start Datanode Key: HDFS-8956 URL: https://issues.apache.org/jira/browse/HDFS-8956 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.7.0 Environment: Centos Reporter: sreelakshmi Data node service is not started on one of the data nodes, java.net.bind exception is thrown. Verified that ports 50010,50070 and 50075 are not in use by any other application. 15/08/26 01:50:15 INFO http.HttpServer2: HttpServer.start() threw a non Bind IOException java.net.BindException: Port in use: localhost:0 at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:919) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856) at org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:779) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1134) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:434) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2404) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2291) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2338) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2515) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2539) Caused by: java.net.BindException: Cannot assign requested address -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8897) Loadbalancer always exits with : java.io.IOException: Another Balancer is running.. Exiting ...
[
https://issues.apache.org/jira/browse/HDFS-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712382#comment-14712382
]
Xu Chen commented on HDFS-8897:
---
[~Alexandre LINTE] I was encounter this issue too , when I use hdfs balancer
command , did you to do this patch ?
Loadbalancer always exits with : java.io.IOException: Another Balancer is
running.. Exiting ...
Key: HDFS-8897
URL: https://issues.apache.org/jira/browse/HDFS-8897
Project: Hadoop HDFS
Issue Type: Bug
Components: balancer mover
Affects Versions: 2.7.1
Environment: Centos 6.6
Reporter: LINTE
When balancer is launched, it should test if there is already a
/system/balancer.id file in HDFS.
When the file doesn't exist, the balancer don't want to run :
15/08/14 16:35:12 INFO balancer.Balancer: namenodes = [hdfs://sandbox/,
hdfs://sandbox]
15/08/14 16:35:12 INFO balancer.Balancer: parameters =
Balancer.Parameters[BalancingPolicy.Node, threshold=10.0, max idle iteration
= 5, number of nodes to be excluded = 0, number of nodes to be included = 0]
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
Bytes Being Moved
15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
30mins, 0sec
15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
15/08/14 16:35:14 INFO balancer.KeyManager: Block token params received from
NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
15/08/14 16:35:14 INFO block.BlockTokenSecretManager: Setting block keys
15/08/14 16:35:14 INFO balancer.KeyManager: Update block keys every 2hrs,
30mins, 0sec
java.io.IOException: Another Balancer is running.. Exiting ...
Aug 14, 2015 4:35:14 PM Balancing took 2.408 seconds
Looking at the audit log file when trying to run the balancer, the balancer
create the /system/balancer.id and then delete it on exiting ...
2015-08-14 16:37:45,844 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
src=/system/balancer.id dst=nullperm=null proto=rpc
2015-08-14 16:37:45,900 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=create
src=/system/balancer.id dst=nullperm=hdfs:hadoop:rw-r-
proto=rpc
2015-08-14 16:37:45,919 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
src=/system/balancer.id dst=nullperm=null proto=rpc
2015-08-14 16:37:46,090 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
src=/system/balancer.id dst=nullperm=null proto=rpc
2015-08-14 16:37:46,112 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=getfileinfo
src=/system/balancer.id dst=nullperm=null proto=rpc
2015-08-14 16:37:46,117 INFO FSNamesystem.audit: allowed=true
ugi=hdfs@SANDBOX.HADOOP (auth:KERBEROS) ip=/x.x.x.x cmd=delete
src=/system/balancer.id dst=nullperm=null proto=rpc
The error seems to be located in
org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
The function checkAndMarkRunning return null even if the /system/balancer.id
doesn't exist before entering this function; if it exists, then it is deleted
and the balancer exit with the same error.
private OutputStream checkAndMarkRunning() throws IOException {
try {
if (fs.exists(idPath)) {
// try appending to it so that it will fail fast if another balancer
is
// running.
IOUtils.closeStream(fs.append(idPath));
fs.delete(idPath, true);
}
final FSDataOutputStream fsout = fs.create(idPath, false);
// mark balancer idPath to be deleted during filesystem closure
fs.deleteOnExit(idPath);
if (write2IdFile) {
fsout.writeBytes(InetAddress.getLocalHost().getHostName());
fsout.hflush();
}
return fsout;
} catch(RemoteException e) {
if(AlreadyBeingCreatedException.class.getName().equals(e.getClassName())){
return null;
} else {
throw e;
}
}
}
Regards
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regard was:I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , is that a bug ? or my config issue? Not all the one block in RAM_DISK use lazy_persist policy - Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regard -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya reassigned HDFS-8955: Assignee: bijaya (was: Liang Xie) Support 'hedged' write in DFSClient --- Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: bijaya Assignee: bijaya Fix For: 2.4.0 This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize dfs.dfsclient.quorum.read.threshold.millis dfs.dfsclient.quorum.read.threadpool.size to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! when I use other machine without datanode installed , it is worked fine all DN tmpfs is fill enough Regards was: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards Not balance block in RAM_DISK use lazy_persist policy on datanode local copy - Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! when I use other machine without datanode installed , it is worked fine all DN tmpfs is fill enough Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Summary: Not balance block in RAM_DISK use lazy_persist policy on datanode local copy (was: Not all the one block in RAM_DISK use lazy_persist policy) Not balance block in RAM_DISK use lazy_persist policy on datanode local copy - Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Description: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards was: I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regard Not all the one block in RAM_DISK use lazy_persist policy - Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: balancer mover Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Affects Version/s: (was: 3.0.0) 2.6.0 Support 'hedged' write in DFSClient --- Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0 Reporter: bijaya Assignee: bijaya We do have hedged read which serves redundancy on read failures due to bad sector/patch in disk. We need to have similar feature for hdfs write. This feature may come with cost but its something to must have for use case which needs to guarantee write success regardless of degraded disk health. Defination of degraded disk is highly debatable but this is what I would define. Degraded disk is the disk which fails to read and write intermittently -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Description: We do have hedged read which serves redundancy on read failures due to bad sector/patch in disk. We need to have similar feature for hdfs write. This feature may come with cost but its something to must have for use case which needs to guarantee write success regardless of degraded disk health. Defination of degraded disk is highly debatable but this is what I would define. Degraded disk is the disk which fails to read and write intermittently (was: This is a placeholder of hdfs related stuff backport from https://issues.apache.org/jira/browse/HBASE-7509 The quorum read ability should be helpful especially to optimize read outliers we can utilize dfs.dfsclient.quorum.read.threshold.millis dfs.dfsclient.quorum.read.threadpool.size to enable/disable the hedged read ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we could export the interested metric valus into client system(e.g. HBase's regionserver metric). The core logic is in pread code path, we decide to goto the original fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per the above config items.) Support 'hedged' write in DFSClient --- Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0 Reporter: bijaya Assignee: bijaya We do have hedged read which serves redundancy on read failures due to bad sector/patch in disk. We need to have similar feature for hdfs write. This feature may come with cost but its something to must have for use case which needs to guarantee write success regardless of degraded disk health. Defination of degraded disk is highly debatable but this is what I would define. Degraded disk is the disk which fails to read and write intermittently -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8955) Support 'hedged' write in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bijaya updated HDFS-8955: - Fix Version/s: (was: 2.4.0) Support 'hedged' write in DFSClient --- Key: HDFS-8955 URL: https://issues.apache.org/jira/browse/HDFS-8955 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0 Reporter: bijaya Assignee: bijaya We do have hedged read which serves redundancy on read failures due to bad sector/patch in disk. We need to have similar feature for hdfs write. This feature may come with cost but its something to must have for use case which needs to guarantee write success regardless of degraded disk health. Defination of degraded disk is highly debatable but this is what I would define. Degraded disk is the disk which fails to read and write intermittently -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-4750) Support NFSv3 interface to HDFS
[ https://issues.apache.org/jira/browse/HDFS-4750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712494#comment-14712494 ] Brandon Li commented on HDFS-4750: -- Overlapped writes happen usually for the first block. For example, a file has 100 bytes. User write this file to NFS mount point. After a while (the 100 bytes still in OS buffer cache), the user appends another 100 bytes. In this case, the NFS client will rewrite the first block (0-199th byte), and thus we see an overlapped write. These kind of case should be well handled by current code (but, still might have bugs there :-( ) Based on your description, the problem seems not the above case. There is one case, we currently have not way to control it on the server side: 1. client sends write 0-99, we cache this write and do the write asynchronously 2. client sends another write 180-299, we cache it too, can't write since there is a hole 3. client sends another write 100-199, we will do the write since it's a sequential write 4. after we finish writing (0-99) and (100-199), we see an overlapped write (180-299) in the cache. This is where you see the error message since we are expecting another sequential write (200-xxx) This kind of overlapped write happens very rarely. In case it happens, we have multiple copies of the same range (180-198 in the above example). The data could be different. When they are different, it could be hard to know which one is really expected by the client. Support NFSv3 interface to HDFS --- Key: HDFS-4750 URL: https://issues.apache.org/jira/browse/HDFS-4750 Project: Hadoop HDFS Issue Type: New Feature Components: nfs Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Attachments: HADOOP-NFS-Proposal.pdf, HDFS-4750.patch, nfs-trunk.patch Access HDFS is usually done through HDFS Client or webHDFS. Lack of seamless integration with client’s file system makes it difficult for users and impossible for some applications to access HDFS. NFS interface support is one way for HDFS to have such easy integration. This JIRA is to track the NFS protocol support for accessing HDFS. With HDFS client, webHDFS and the NFS interface, HDFS will be easier to access and be able support more applications and use cases. We will upload the design document and the initial implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8954) Not all the one block in RAM_DISK use lazy_persist policy
[ https://issues.apache.org/jira/browse/HDFS-8954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Chen updated HDFS-8954: -- Component/s: (was: balancer mover) datanode Not all the one block in RAM_DISK use lazy_persist policy - Key: HDFS-8954 URL: https://issues.apache.org/jira/browse/HDFS-8954 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Xu Chen Priority: Critical I was try to use Heterogeneous Storage feature and I config RAM_DISK volume for each DN size of 20g , and set one directory policy of lazy_persist at HDFS , then command copyFromLocal to load a 79g file to the directory HDFS , it is have 593 block , and only 141 block in RAM_DISK , I use command df to look tmpfs usage it only my execute command DN fill enough the tmpfs , other DN tmpfs is empty ! that a bug ? or my config issue? Regards -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712541#comment-14712541
]
Zhe Zhang commented on HDFS-8853:
-
Thanks for the update [~andreina]. Since there are many failures (likely to be
unrelated though), I just triggered Jenkins again.
Erasure Coding: Provide ECSchema validation when creating ECZone
Key: HDFS-8853
URL: https://issues.apache.org/jira/browse/HDFS-8853
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: J.Andreina
Attachments: HDFS-8853-HDFS-7285-01.patch,
HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-merge-02.patch,
HDFS-8853-HDFS-7285-merge-03.patch, HDFS-8853-HDFS-7285-merge-04.patch
Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
doesn't have any validation that the given {{ecSchema}} is available in
{{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
then will create the ECZone with {{null}} schema. IMHO we could improve this
by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8581: - Attachment: HDFS-8581.3.patch Updated the patch. Please review. count cmd calculate wrong when huge files exist in one folder - Key: HDFS-8581 URL: https://issues.apache.org/jira/browse/HDFS-8581 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Reporter: tongshiquan Assignee: J.Andreina Priority: Minor Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch, HDFS-8581.3.patch If one directory such as /result exists about 20 files, then when execute hdfs dfs -count /, the result will go wrong. For all directories whose name after /result, file num will not be included. My cluster see as below, /result_1433858936 is the directory exist huge files, and files in /sparkJobHistory, /tmp, /user are not included vm-221:/export1/BigData/current # hdfs dfs -ls / 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. Found 9 items -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 /PRE_CREATE_DIR.SUCCESS drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 /result_1433858936 drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count / 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. 1043 171536 1756375688 / vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. 01 0 /PRE_CREATE_DIR.SUCCESS vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /flume 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. 10 0 /flume vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /hbase 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. 36 18 14807 /hbase vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /hyt 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. 10 0 /hyt vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /mr-history 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. 30 0 /mr-history vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. 1001 171517 1756360881 /result_1433858936 vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. 13 21785 /sparkJobHistory vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /tmp 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. 176 35958 /tmp vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /user 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712520#comment-14712520
]
J.Andreina commented on HDFS-8853:
--
Testcase failures are not related to this patch. Please review.
Erasure Coding: Provide ECSchema validation when creating ECZone
Key: HDFS-8853
URL: https://issues.apache.org/jira/browse/HDFS-8853
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: J.Andreina
Attachments: HDFS-8853-HDFS-7285-01.patch,
HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-merge-02.patch,
HDFS-8853-HDFS-7285-merge-03.patch, HDFS-8853-HDFS-7285-merge-04.patch
Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
doesn't have any validation that the given {{ecSchema}} is available in
{{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
then will create the ECZone with {{null}} schema. IMHO we could improve this
by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710763#comment-14710763
]
Hadoop QA commented on HDFS-8909:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 35s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 12 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 13s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 41s | The patch appears to introduce 4
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 4s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 20m 10s | Tests failed in hadoop-hdfs. |
| | | 62m 22s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestErasureCodingZones |
| | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeMXBean |
| | hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestEditLogRace |
| | hadoop.hdfs.server.namenode.ha.TestStateTransitionFailure |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
| | hadoop.hdfs.TestDFSRename |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.TestDatanodeConfig |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.TestBlockReaderFactory |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.TestFSOutputSummer |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.tools.TestJMXGet |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.server.datanode.TestDiskError |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.fs.contract.hdfs.TestHDFSContractRootDirectory |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.fs.TestUrlStreamHandler |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.TestDFSRemove |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.server.namenode.TestStartupProgressServlet |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.fs.contract.hdfs.TestHDFSContractSeek |
| | hadoop.hdfs.server.datanode.TestHSync |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.web.TestJsonUtil |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| |
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710844#comment-14710844
]
Hadoop QA commented on HDFS-8951:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 18s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 55s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 26s | The applied patch generated
61 new checkstyle issues (total was 16, now 77). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 68m 23s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 118m 38s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.server.datanode.TestStorageReport |
| | hadoop.hdfs.web.TestWebHDFSForHA |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.TestFetchImage |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestReadOnlySharedStorage |
| | hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.hdfs.web.TestHttpsFileSystem |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
| |
hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters |
| | hadoop.hdfs.server.namenode.TestNameNodeResourceChecker |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica |
| | hadoop.hdfs.TestClientBlockVerification |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages |
| | hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
| | hadoop.hdfs.server.namenode.TestCreateEditsLog |
| | hadoop.tools.TestJMXGet |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.TestParallelRead |
| | hadoop.hdfs.TestHDFSTrash |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.blockmanagement.TestBlockReportRateLimiting |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.TestBlockUnderConstruction |
| | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| | hadoop.hdfs.TestWriteRead |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.web.TestFSMainOperationsWebHdfs |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.crypto.TestHdfsCryptoStreams |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestFSOutputSummer |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.TestListCorruptFileBlocks |
| |
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710857#comment-14710857
]
Yi Liu commented on HDFS-8900:
--
Thanks Andrew, Colin and Brahma.
The findbugs doesn't exist. (The existing findbugs were in pre-patch).
Committed to trunk and branch-2.
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-8900:
-
Resolution: Fixed
Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change)
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[
https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710770#comment-14710770
]
Zhe Zhang commented on HDFS-7285:
-
[~vinayrpet] Could you take a look at the results comparing the two
rebase/merge efforts, and see if we should delete the {{HDFS-7285-REBASE}}
branch? Thanks!
Erasure Coding Support inside HDFS
--
Key: HDFS-7285
URL: https://issues.apache.org/jira/browse/HDFS-7285
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Weihua Jiang
Assignee: Zhe Zhang
Attachments: Compare-consolidated-20150824.diff,
Consolidated-20150707.patch, Consolidated-20150806.patch,
Consolidated-20150810.patch, ECAnalyzer.py, ECParser.py,
HDFS-7285-initial-PoC.patch, HDFS-7285-merge-consolidated-01.patch,
HDFS-7285-merge-consolidated-trunk-01.patch,
HDFS-7285-merge-consolidated.trunk.03.patch,
HDFS-7285-merge-consolidated.trunk.04.patch,
HDFS-EC-Merge-PoC-20150624.patch, HDFS-EC-merge-consolidated-01.patch,
HDFS-bistriped.patch, HDFSErasureCodingDesign-20141028.pdf,
HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf,
HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf,
HDFSErasureCodingSystemTestPlan-20150824.pdf, fsimage-analysis-20150105.pdf
Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice
of data reliability, comparing to the existing HDFS 3-replica approach. For
example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks,
with storage overhead only being 40%. This makes EC a quite attractive
alternative for big data storage, particularly for cold data.
Facebook had a related open source project called HDFS-RAID. It used to be
one of the contribute packages in HDFS but had been removed since Hadoop 2.0
for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends
on MapReduce to do encoding and decoding tasks; 2) it can only be used for
cold files that are intended not to be appended anymore; 3) the pure Java EC
coding implementation is extremely slow in practical use. Due to these, it
might not be a good idea to just bring HDFS-RAID back.
We (Intel and Cloudera) are working on a design to build EC into HDFS that
gets rid of any external dependencies, makes it self-contained and
independently maintained. This design lays the EC feature on the storage type
support and considers compatible with existing HDFS features like caching,
snapshot, encryption, high availability and etc. This design will also
support different EC coding schemes, implementations and policies for
different deployment scenarios. By utilizing advanced libraries (e.g. Intel
ISA-L library), an implementation can greatly improve the performance of EC
encoding/decoding and makes the EC solution even more attractive. We will
post the design document soon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710775#comment-14710775 ] Zhe Zhang commented on HDFS-8838: - Thanks Walter for the discussion. If you are still +1 on the latest patch, it's OK to commit to the HDFS-7285 branch now. Tolerate datanode failures in DFSStripedOutputStream when the data length is small -- Key: HDFS-8838 URL: https://issues.apache.org/jira/browse/HDFS-8838 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: HDFS-8838-HDFS-7285-000.patch, HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, h8838_20150804-HDFS-7285.patch, h8838_20150809.patch Currently, DFSStripedOutputStream cannot tolerate datanode failures when the data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7116) Add a command to get the balancer bandwidth
[ https://issues.apache.org/jira/browse/HDFS-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710820#comment-14710820 ] Rakesh R commented on HDFS-7116: It looks like test case failures are not related to the proposed patch, please ignore it ! Add a command to get the balancer bandwidth --- Key: HDFS-7116 URL: https://issues.apache.org/jira/browse/HDFS-7116 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover Reporter: Akira AJISAKA Assignee: Rakesh R Attachments: HDFS-7116-00.patch, HDFS-7116-01.patch, HDFS-7116-02.patch, HDFS-7116-03.patch, HDFS-7116-04.patch, HDFS-7116-05.patch, HDFS-7116-06.patch Now reading logs is the only way to check how the balancer bandwidth is set. It would be useful for administrators if they can get the value of the same. This jira to discuss implement the way to access the balancer bandwidth value of the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710910#comment-14710910
]
Hadoop QA commented on HDFS-8155:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 16s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:red}-1{color} | javac | 7m 50s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 3s | Site still builds. |
| {color:red}-1{color} | checkstyle | 2m 40s | The applied patch generated
36 new checkstyle issues (total was 151, now 187). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 7 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 21s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 163m 36s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 27s | Tests passed in
hadoop-hdfs-client. |
| | | 219m 54s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752142/HDFS-8155.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 57c7ae1 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12106/console |
This message was automatically generated.
Support OAuth2 in WebHDFS
-
Key: HDFS-8155
URL: https://issues.apache.org/jira/browse/HDFS-8155
Project: Hadoop HDFS
Issue Type: New Feature
Components: webhdfs
Reporter: Jakob Homan
Assignee: Jakob Homan
Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch
WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710920#comment-14710920
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8348 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8348/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[
https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710773#comment-14710773
]
Zhe Zhang commented on HDFS-8838:
-
{{HDFS-7285-merge}} and {{HDFS-7285-REBASE}} are intermediate branches. I have
just updated the main {{HDFS-7285}} branch, so let's use it from now on. Sorry
about the confusion.
Tolerate datanode failures in DFSStripedOutputStream when the data length is
small
--
Key: HDFS-8838
URL: https://issues.apache.org/jira/browse/HDFS-8838
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: HDFS-8838-HDFS-7285-000.patch,
HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch,
HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch,
h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch,
h8838_20150804-HDFS-7285.patch, h8838_20150809.patch
Currently, DFSStripedOutputStream cannot tolerate datanode failures when the
data length is small. We fix the bugs here and add more tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8941:
---
Status: Patch Available (was: Open)
DistributedFileSystem listCorruptFileBlocks API should resolve relative path
Key: HDFS-8941
URL: https://issues.apache.org/jira/browse/HDFS-8941
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8941-00.patch
Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
path relative to the workingDir. This jira is to discuss and provide the
implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8941) DistributedFileSystem listCorruptFileBlocks API should resolve relative path
[
https://issues.apache.org/jira/browse/HDFS-8941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8941:
---
Status: Open (was: Patch Available)
DistributedFileSystem listCorruptFileBlocks API should resolve relative path
Key: HDFS-8941
URL: https://issues.apache.org/jira/browse/HDFS-8941
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8941-00.patch
Presently {{DFS#listCorruptFileBlocks(path)}} API is not resolving the given
path relative to the workingDir. This jira is to discuss and provide the
implementation of the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710718#comment-14710718
]
Hadoop QA commented on HDFS-8900:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 50s | Pre-patch trunk has 2 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 1
new checkstyle issues (total was 516, now 511). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 36s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 163m 27s | Tests passed in hadoop-hdfs.
|
| | | 209m 9s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752124/HDFS-8900.005.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3b00eae |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12103/console |
This message was automatically generated.
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8939) Test(S)WebHdfsFileContextMainOperations failing on branch-2
[
https://issues.apache.org/jira/browse/HDFS-8939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710671#comment-14710671
]
Hadoop QA commented on HDFS-8939:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 23m 15s | Pre-patch branch-2 has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 6m 8s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 55s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 16s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 51s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 1m 52s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 42s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 216m 16s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDistributedFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752106/HDFS-8939-branch-2.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | branch-2 / 137bde0 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/artifact/patchprocess/branch-2FindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/artifact/patchprocess/branch-2FindbugsWarningshadoop-hdfs-client.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12101/console |
This message was automatically generated.
Test(S)WebHdfsFileContextMainOperations failing on branch-2
---
Key: HDFS-8939
URL: https://issues.apache.org/jira/browse/HDFS-8939
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.8.0
Reporter: Jakob Homan
Assignee: Jakob Homan
Fix For: 2.8.0
Attachments: HDFS-8939-branch-2.001.patch,
HDFS-8939-branch-2.002.patch
After HDFS-8180, TestWebHdfsFileContextMainOperations and
TestSWebHdfsFileContextMainOperations are failing with runtime NPEs while
instantiating the wrapped WebHDFSFileSystems because {{getDefaultPort}} is
trying to access a conf that was never provided. In the constructor both
both WebHdfs and SWebhdfs the underlying (S)WebHdfsFileSystems are
instantiated in the constructor and never have a chance to have their
{{setConf}} methods called:
{code} SWebHdfs(URI theUri, Configuration conf)
throws IOException, URISyntaxException {
super(theUri, new SWebHdfsFileSystem(), conf, SCHEME, false);
}r{code}
The test passes on trunk because HDFS-5321 removed the call to the
Configuration instance as part of {{getDefaultPort}}. HDFS-5321 was applied
to branch-2 but reverted in HDFS-6632, so there's a bit of a difference in
how branch-2 versus trunk handles default values (branch-2 pulls them from
configs if specified, trunk just returns the hard-coded value from the
constants file).
I've fixed this behave like trunk and return just the hard-coded value, which
causes the test to pass.
There is no WebHdfsFileSystem that takes a Config, which would be another
way to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710695#comment-14710695 ] Li Bo commented on HDFS-8838: - hi, Walter Su 我看了下branch， 跟7285相关的有 remotes/origin/HDFS-7285 remotes/origin/HDFS-7285-REBASE remotes/origin/HDFS-7285-merge 这几者目前是什么关系？ Thanks Tolerate datanode failures in DFSStripedOutputStream when the data length is small -- Key: HDFS-8838 URL: https://issues.apache.org/jira/browse/HDFS-8838 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: HDFS-8838-HDFS-7285-000.patch, HDFS-8838-HDFS-7285-20150809-test.patch, HDFS-8838-HDFS-7285-20150809.patch, HDFS-8838-HDFS-7285-20150821.patch, h8838_20150729.patch, h8838_20150731-HDFS-7285.patch, h8838_20150731.log, h8838_20150731.patch, h8838_20150804-HDFS-7285.patch, h8838_20150809.patch Currently, DFSStripedOutputStream cannot tolerate datanode failures when the data length is small. We fix the bugs here and add more tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710735#comment-14710735
]
Hadoop QA commented on HDFS-8248:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 48s | Pre-patch trunk has 2 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 512, now 510). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 12s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 66m 41s | Tests failed in hadoop-hdfs. |
| | | 111m 33s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestEditLogRace |
| | hadoop.hdfs.TestDFSRename |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithNodeGroup |
| | hadoop.hdfs.TestDatanodeConfig |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.namenode.TestXAttrConfigFlag |
| | hadoop.hdfs.TestBlockReaderFactory |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.TestFSOutputSummer |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.web.TestWebHdfsWithAuthenticationFilter |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.TestDFSRemove |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.server.namenode.TestGenericJournalConf |
| | hadoop.hdfs.TestWriteConfigurationToDFS |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.web.TestJsonUtil |
| | hadoop.hdfs.server.namenode.TestNNThroughputBenchmark |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| | hadoop.hdfs.TestPipelines |
| | hadoop.hdfs.tools.TestGetGroups |
| | hadoop.hdfs.TestFileCreationClient |
| | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockId |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestRollingUpgradeRollback |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename |
| | hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.protocolPB.TestPBHelper |
| | hadoop.hdfs.server.namenode.TestMalformedURLs |
| | hadoop.hdfs.server.namenode.TestFSImageWithXAttr |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| |
[jira] [Commented] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710744#comment-14710744 ] Vinayakumar B commented on HDFS-5851: - This Jira should be marked as duplicate of HDFS-6581 ? Support memory as a storage medium -- Key: HDFS-5851 URL: https://issues.apache.org/jira/browse/HDFS-5851 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf Memory can be used as a storage medium for smaller/transient files for fast write throughput. More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[
https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711714#comment-14711714
]
Hadoop QA commented on HDFS-8894:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 28s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 1
new checkstyle issues (total was 150, now 150). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 17s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 0m 24s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 0m 23s | Tests failed in hadoop-hdfs. |
| | | 43m 34s | |
\\
\\
|| Reason || Tests ||
| Failed build | hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752268/HDFS-8894-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12118/console |
This message was automatically generated.
Set SO_KEEPALIVE on DN server sockets
-
Key: HDFS-8894
URL: https://issues.apache.org/jira/browse/HDFS-8894
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.7.1
Reporter: Nathan Roberts
Assignee: kanaka kumar avvaru
Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch
SO_KEEPALIVE is not set on things like datastreamer sockets which can cause
lingering ESTABLISHED sockets when there is a network glitch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711775#comment-14711775
]
Haohui Mai commented on HDFS-8951:
--
Thanks for the work.
{code}
+package org.apache.hadoop.hdfs.util;
+
+import org.slf4j.Logger;
+
+import java.io.IOException;
+
+public class IOUtils {
+
+ /**
+ * Close the Closeable objects and bignore/b any {@link IOException} or
+ * null pointers. Must only be used for cleanup in exception handlers.
+ *
+ * @param log the log to record problems to at debug level. Can be null.
+ * @param closeables the objects to close
+ */
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
+for (java.io.Closeable c : closeables) {
+ if (c != null) {
+try {
+ c.close();
+} catch(Throwable e) {
+ if (log != null log.isDebugEnabled()) {
+log.debug(Exception in closing + c, e);
+ }
+}
+ }
+}
+ }
+
+}
{code}
The change is not required. It's better to change the callers to use
try-with-resource statement in Java 7.
{code}
-final DataChecksum checksum =
BlockMetadataHeader.readDataChecksum(srcMeta);
+final DataChecksum checksum = BlockMetadataHeader.readDataChecksum(srcMeta,
+DFSUtil.getIoFileBufferSize(new HdfsConfiguration()));
{code}
Creating a new configuration object will parse the XML configuration every
single time. The caller needs to use the {{Configuration}} object in the
{{FSDatasetImpl}} class. Therefore the changes need to be propagated all the
way along the call chain.
Move shortcircuit to hdfs-client
Key: HDFS-8951
URL: https://issues.apache.org/jira/browse/HDFS-8951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch
This jira tracks the effort of moving the {{shortcircuit}} package into the
hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: (was: HDFS-8894-01.patch) Set SO_KEEPALIVE on DN server sockets - Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts Assignee: kanaka kumar avvaru Attachments: HDFS-8894-01.patch SO_KEEPALIVE is not set on things like datastreamer sockets which can cause lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711003#comment-14711003
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1037 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1037/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711045#comment-14711045
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2253 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2253/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8836:
--
Status: Patch Available (was: Open)
Attached a patch to add a new flag {{-skip-empty-file}} and check the file
length before copy content.
Skip newline on empty files with getMerge -nl
-
Key: HDFS-8836
URL: https://issues.apache.org/jira/browse/HDFS-8836
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.7.1, 2.6.0
Reporter: Jan Filipiak
Assignee: kanaka kumar avvaru
Priority: Trivial
Attachments: HDFS-8836-01.patch
Hello everyone,
I recently was in the need of using the new line option -nl with getMerge
because the files I needed to merge simply didn't had one. I was merging all
the files from one directory and unfortunately this directory also included
empty files, which effectively led to multiple newlines append after some
files. I needed to remove them manually afterwards.
In this situation it is maybe good to have another argument that allows
skipping empty files.
Thing one could try to implement this feature:
The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
return the number of bytes copied which would be convenient as one could
skip append the new line when 0 bytes where copied or one would check the
file size before.
I posted this Idea on the mailing list
http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8892) ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
[
https://issues.apache.org/jira/browse/HDFS-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710959#comment-14710959
]
Ravikumar commented on HDFS-8892:
-
I am fine if this runs in a separate thread as you said…
My app opens hadoop files only once. The block input-streams, once opened need
to stay that way forever until app instructs to close it. The close call
happens only for deleting the file or transferring the ownership of file to
another server {Like Hbase regions}
With current ShortCircuitCache, I am not able to find any knobs to tune for
such kind of behaviour.
I am planning to set all params of LRU cache high so that ShortCircuitCache
never closes any open input-stream by itself. All the stream-close calls are
handled only through invalid-slot-checker
ShortCircuitCache.CacheCleaner can add Slot.isInvalid() check too
-
Key: HDFS-8892
URL: https://issues.apache.org/jira/browse/HDFS-8892
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.7.1
Reporter: Ravikumar
Assignee: kanaka kumar avvaru
Priority: Minor
Currently CacheCleaner thread checks only for cache-expiry times. It would be
nice if it handles an invalid-slot too in an extra-pass of evictable map…
for(ShortCircuitReplica replica:evictable.values()) {
if(!scr.getSlot().isValid()) {
purge(replica);
}
}
//Existing code...
int numDemoted = demoteOldEvictableMmaped(curMs);
int numPurged = 0;
Long evictionTimeNs = Long.valueOf(0);
….
…..
Apps like HBase can tweak the expiry/staleness/cache-size params in
DFS-Client, so that ShortCircuitReplica will never be closed except when Slot
is declared invalid.
I assume slot-invalidation will happen during block-invalidation/deletes
{Primarily triggered by compaction/shard-takeover etc..}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8853) Erasure Coding: Provide ECSchema validation when creating ECZone
[
https://issues.apache.org/jira/browse/HDFS-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina updated HDFS-8853:
-
Attachment: HDFS-8853-HDFS-7285-05.patch
Thanks [~vinayrpet].
Have updated the patch fixing the comments.
Please review.
Erasure Coding: Provide ECSchema validation when creating ECZone
Key: HDFS-8853
URL: https://issues.apache.org/jira/browse/HDFS-8853
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: J.Andreina
Attachments: HDFS-8853-HDFS-7285-01.patch,
HDFS-8853-HDFS-7285-05.patch, HDFS-8853-HDFS-7285-merge-02.patch,
HDFS-8853-HDFS-7285-merge-03.patch, HDFS-8853-HDFS-7285-merge-04.patch
Presently the {{DFS#createErasureCodingZone(path, ecSchema, cellSize)}}
doesn't have any validation that the given {{ecSchema}} is available in
{{ErasureCodingSchemaManager#activeSchemas}} list. Now, if it doesn't exists
then will create the ECZone with {{null}} schema. IMHO we could improve this
by doing necessary basic sanity checks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Patch Available (was: Open) Set SO_KEEPALIVE on DN server sockets - Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts Assignee: kanaka kumar avvaru Attachments: HDFS-8894-01.patch SO_KEEPALIVE is not set on things like datastreamer sockets which can cause lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8938) Refactor BlockManager in blockmanagement
[
https://issues.apache.org/jira/browse/HDFS-8938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711041#comment-14711041
]
Hadoop QA commented on HDFS-8938:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 38s | Pre-patch trunk has 2 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 311, now 295). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings, and fixes 1 pre-existing warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 50s | Tests failed in hadoop-hdfs. |
| | | 207m 54s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752145/HDFS-8938.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / af78767 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12109/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12109/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12109/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12109/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12109/console |
This message was automatically generated.
Refactor BlockManager in blockmanagement
Key: HDFS-8938
URL: https://issues.apache.org/jira/browse/HDFS-8938
Project: Hadoop HDFS
Issue Type: Task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Attachments: HDFS-8938.000.patch, HDFS-8938.001.patch,
HDFS-8938.002.patch, HDFS-8938.003.patch, HDFS-8938.004.patch
This lira tracks the effort of refactoring the {{BlockManager}} in
{{hdfs.server.blockmanagement}} package.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8763) Incremental blockreport order may replicate unnecessary block
[
https://issues.apache.org/jira/browse/HDFS-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711052#comment-14711052
]
Brahma Reddy Battula commented on HDFS-8763:
Change of {{dfs.namenode.replication.min}} to 3 (not 2) might solve this
scenario.
Changing the min replication can lead to new problems. Like safemode on
restart, write failures if not enough datanodes, etc.
So better to increase the replication check interval. Thats enough,I feel.
Incremental blockreport order may replicate unnecessary block
-
Key: HDFS-8763
URL: https://issues.apache.org/jira/browse/HDFS-8763
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS
Affects Versions: 2.4.0
Reporter: jiangyu
Assignee: Walter Su
Priority: Minor
For our cluster, the NameNode is always very busy, so for every incremental
block report , the contention of lock is heavy.
The logic of incremental block report is as follow, client send block to dn1
and dn1 mirror to dn2, dn2 mirror to dn3. After finish this block, all
datanode will report the newly received block to namenode. In NameNode side,
all will go to the method processIncrementalBlockReport in BlockManager
class. But the status of the block reported from dn2,dn3 is RECEIVING_BLOCK,
for dn1 is RECEIED_BLOCK. It is okay if dn2, dn3 report before dn1(that is
common), but in some busy environment, it is easy to find dn1 report before
dn2 or dn3, let’s assume dn2 report first, dn1 report second, and dn3 report
third.
So dn1 will addStoredBlock and find the replica of this block is not reach
the the original number(which is 3), and the block will add to
neededReplications construction and soon ask some node in pipeline (dn1 or
dn2)to replica it dn4 . After sometime, dn4 and dn3 all report this block,
then choose one node to invalidate.
Here is one log i found in our cluster:
2015-07-08 01:05:34,675 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/logs/***_bigdata_spam/logs/application_1435099124107_470749/xx.xx.4.62_45454.tmp.
BP-1386326728-xx.xx.2.131-1382089338395
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-a7c0f8f6-2399-4980-9479-efa08487b7b3:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-c75145a0-ed63-4180-87ee-d48ccaa647c5:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW]]}
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.75:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:34,689 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.62:50010 is added to
blk_3194502674_2121080184{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-15a4dc8e-5b7d-449f-a941-6dced45e6f07:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-74ed264f-da43-4cc3-9fa9-164ba99f752a:NORMAL|RBW],
ReplicaUnderConstruction[[DISK]DS-56121ce1-8991-45b3-95bc-2a5357991512:NORMAL|RBW]]}
size 0
2015-07-08 01:05:35,003 INFO BlockStateChange: BLOCK* ask xx.xx.4.62:50010 to
replicate blk_3194502674_2121080184 to datanode(s) xx.xx.4.65:50010
2015-07-08 01:05:35,403 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.7.73:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* addStoredBlock:
blockMap updated: xx.xx.4.65:50010 is added to blk_3194502674_2121080184 size
67750
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
blk_3194502674_2121080184 to xx.xx.7.75:50010
2015-07-08 01:05:35,833 INFO BlockStateChange: BLOCK* chooseExcessReplicates:
(xx.xx.7.75:50010, blk_3194502674_2121080184) is added to invalidated blocks
set
2015-07-08 01:05:35,852 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
InvalidateBlocks: ask xx.xx.7.75:50010 to delete [blk_3194502674_2121080184,
blk_3194497594_2121075104]
Some day, the number of this situation can be 40, that is not good for
the performance and waste network band.
Our base version is hadoop 2.4 and i have checked hadoop 2.7.1 didn’t find
any difference.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8520) Patch for PPC64 block size
[ https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710968#comment-14710968 ] Tony Reix commented on HDFS-8520: - Hi Andrew Sorry of answering so late. I was busy with other staff. My understanding of this defect is: - I had analyzed the code and had found hardcoded 4096 values for PageSize in many places, which does cover mainly basic x86 case but not other architectures - on Linux on Power, PageSize is different - I had proposed to use the getOperatingSystemPageSize() routine instead of hardcoded 4096 value in order to make the test generic (work on all targets with no change) - You proposed instead to hardcode 4096 in the existing test and to add a new test with 64KB as hardcoded value (which would cover x86 Linux Large Page case) I guess that this would mean to add some code to check, before running these 2 tests, which ones are applicable on the targeted architecture. What about other architectures that would not use 4096 nor 64KB ? What is the status now for Hadoop 2.7.2 ? Are you planning to make the code more homogeneous and to enable it to handle cases (like Linux on Power) where basic PageSize is NOT 4096 ? It is very difficult for me to propose a patch since: 1) there are a lot of places where 4096 and PageSize appear, with no clear clue if that deals with my issue 2) your approach is very different from mine: hardcode values instead of making code generic Moreover, the way the tests fail is a pain, since they end with TimeOut. In my opinion, a huge work of rewriting test code so that it deals by default with very different architectures (including x86 4KB and 64KB cases) would be the best approach. I also would suggest to add a Linux/PPC64 machine in the development/test environment in order to handle different architectures at soon as possible. Patch for PPC64 block size -- Key: HDFS-8520 URL: https://issues.apache.org/jira/browse/HDFS-8520 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Environment: RHEL 7.1 /PPC64 Reporter: Tony Reix Assignee: Tony Reix Attachments: HDFS-8520-2.patch, HDFS-8520.patch The attached patch enables Hadoop to work on PPC64. That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64. There are changes in 3 files: - hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE The patch has been built on branch-2.7 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8836: -- Attachment: HDFS-8836-01.patch Skip newline on empty files with getMerge -nl - Key: HDFS-8836 URL: https://issues.apache.org/jira/browse/HDFS-8836 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0, 2.7.1 Reporter: Jan Filipiak Assignee: kanaka kumar avvaru Priority: Trivial Attachments: HDFS-8836-01.patch Hello everyone, I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied or one would check the file size before. I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Status: Open (was: Patch Available) Set SO_KEEPALIVE on DN server sockets - Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts Assignee: kanaka kumar avvaru Attachments: HDFS-8894-01.patch SO_KEEPALIVE is not set on things like datastreamer sockets which can cause lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710947#comment-14710947
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #309 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/309/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8248) Store INodeId instead of the INodeFile object in BlockInfoContiguous
[
https://issues.apache.org/jira/browse/HDFS-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712068#comment-14712068
]
Hadoop QA commented on HDFS-8248:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 49s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 56s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 26s | The applied patch generated 3
new checkstyle issues (total was 512, now 510). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 8s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 8s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 0m 22s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 159m 41s | Tests failed in hadoop-hdfs. |
| | | 202m 7s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.fs.TestHdfsNativeCodeLoader |
| | hadoop.hdfs.server.blockmanagement.TestSequentialBlockId |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752141/HDFS-8248.007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12122/console |
This message was automatically generated.
Store INodeId instead of the INodeFile object in BlockInfoContiguous
Key: HDFS-8248
URL: https://issues.apache.org/jira/browse/HDFS-8248
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-8248.000.patch, HDFS-8248.001.patch,
HDFS-8248.002.patch, HDFS-8248.003.patch, HDFS-8248.004.patch,
HDFS-8248.005.patch, HDFS-8248.006.patch, HDFS-8248.007.patch
Currently the namespace and the block manager are tightly coupled together.
There are two couplings in terms of implementation:
1. The {{BlockInfoContiguous}} stores a reference of the {{INodeFile}} that
owns the block, so that the block manager can look up the corresponding file
when replicating blocks, recovering from pipeline failures, etc.
1. The {{INodeFile}} stores {{BlockInfoContiguous}} objects that the file
owns.
Decoupling the namespace and the block manager allows the BM to be separated
out from the Java heap or even as a standalone process. This jira proposes to
remove the first coupling by storing the id of the inode instead of the
object reference of {{INodeFile}} in the {{BlockInfoContiguous}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-1148) Convert FSDataset to ReadWriteLock
[
https://issues.apache.org/jira/browse/HDFS-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711727#comment-14711727
]
Hadoop QA commented on HDFS-1148:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 25s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 0s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 25s | The applied patch generated 1
new checkstyle issues (total was 121, now 111). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 1s | Post-patch findbugs
hadoop-hdfs-project/hadoop-hdfs compilation is broken. |
| {color:green}+1{color} | findbugs | 2m 1s | The patch does not introduce
any new Findbugs (version ) warnings. |
| {color:green}+1{color} | native | 0m 37s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 6m 39s | Tests failed in hadoop-hdfs. |
| | | 49m 49s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752270/HDFS-1148.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12117/console |
This message was automatically generated.
Convert FSDataset to ReadWriteLock
--
Key: HDFS-1148
URL: https://issues.apache.org/jira/browse/HDFS-1148
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Reporter: Todd Lipcon
Assignee: Yong Zhang
Attachments: HDFS-1148.001.patch, HDFS-1148.002.patch,
HDFS-1148.003.patch, hdfs-1148-old.txt, hdfs-1148-trunk.txt,
patch-HDFS-1148-rel0.20.2.txt
In benchmarking HDFS-941 I noticed that for the random read workload, the
FSDataset lock is highly contended. After converting it to a
ReentrantReadWriteLock, I saw a ~25% improvement on both latency and
ops/second.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-8950: - Target Version/s: 2.8.0 NameNode refresh doesn't remove DataNodes - Key: HDFS-8950 URL: https://issues.apache.org/jira/browse/HDFS-8950 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Daniel Templeton Assignee: Daniel Templeton If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission. To reproduce: 1. Add a DN to dfs_hosts_allow 2. Refresh NN 3. Start DN. Now NN starts seeing DN. 4. Stop DN 5. Remove DN from dfs_hosts_allow 6. Refresh NN - NN is still reporting DN as being used by HDFS. This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8951) Move shortcircuit to hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712075#comment-14712075
]
Hadoop QA commented on HDFS-8951:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 19m 59s | Pre-patch trunk has 6 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 10s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 16s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 31s | The applied patch generated
62 new checkstyle issues (total was 16, now 78). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 22s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 213m 29s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752146/HDFS-8951.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs-client.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12121/console |
This message was automatically generated.
Move shortcircuit to hdfs-client
Key: HDFS-8951
URL: https://issues.apache.org/jira/browse/HDFS-8951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Reporter: Mingliang Liu
Assignee: Mingliang Liu
Attachments: HDFS-8951.000.patch, HDFS-8951.001.patch
This jira tracks the effort of moving the {{shortcircuit}} package into the
hdfs-client module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8909) Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use BlockUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712083#comment-14712083
]
Hadoop QA commented on HDFS-8909:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 16s | Pre-patch HDFS-7285 has 4
extant Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 12 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 17s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 28s | The applied patch generated
14 new checkstyle issues (total was 1116, now 1086). |
| {color:green}+1{color} | whitespace | 0m 12s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 5s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 208m 14s | Tests failed in hadoop-hdfs. |
| | | 253m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestWriteStripedFileWithFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752122/HDFS-8909-HDFS-7285.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 6b6a63b |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/HDFS-7285FindbugsWarningshadoop-hdfs.html
|
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12119/console |
This message was automatically generated.
Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC to use
BlockUnderConstructionFeature
Key: HDFS-8909
URL: https://issues.apache.org/jira/browse/HDFS-8909
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Jing Zhao
Attachments: HDFS-8909-HDFS-7285.003.patch, HDFS-8909.000.patch,
HDFS-8909.001.patch, HDFS-8909.002.patch, HDFS-8909.003.patch
HDFS-8801 converts {{BlockInfoUC}} as a feature. We should consolidate
{{BlockInfoContiguousUC}} and {{BlockInfoStripedUC}} logics to use this
feature.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: 28531.patch Here's a patch for a unit test and fix. The HostFileManager now excludes any nodes not present in a non-empty includes file from the node report. All tests in the HDFS project pass. NameNode refresh doesn't remove DataNodes - Key: HDFS-8950 URL: https://issues.apache.org/jira/browse/HDFS-8950 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Daniel Templeton Assignee: Daniel Templeton Attachments: 28531.patch If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission. To reproduce: 1. Add a DN to dfs_hosts_allow 2. Refresh NN 3. Start DN. Now NN starts seeing DN. 4. Stop DN 5. Remove DN from dfs_hosts_allow 6. Refresh NN - NN is still reporting DN as being used by HDFS. This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712097#comment-14712097 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1038 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1038/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8885) ByteRangeInputStream used in webhdfs does not override available()
[
https://issues.apache.org/jira/browse/HDFS-8885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711935#comment-14711935
]
Chris Nauroth commented on HDFS-8885:
-
Hello [~srevanka]. Thank you for posting the patch. Here are a few notes.
# A {{ByteRangeInputStream}} starts out in {{SEEK}} status, which indicates the
underlying HTTP stream needs to be opened. If a caller calls {{available}}
before any {{read}} calls, then the result won't be correct, because it has not
yet determined the file length. I recommend calling {{getInputStream}} first,
which lazily opens the underlying HTTP stream if necessary.
# A similar situation occurs if a caller calls {{seek}} immediately before
{{available}}. Again, calling {{getInputStream}} first would handle this case
correctly.
# It's typical for {{available}} to throw an {{IOException}} if called after
the stream has been closed. Once again, a call to {{getInputStream}} would
cover this case, because that method is already implemented to throw an
{{IOException}} if the stream is in {{CLOSED}} state.
# Can you please look into adding unit tests in {{TestByteRangeInputStream}}?
ByteRangeInputStream used in webhdfs does not override available()
--
Key: HDFS-8885
URL: https://issues.apache.org/jira/browse/HDFS-8885
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.1
Reporter: Shradha Revankar
Assignee: Shradha Revankar
Priority: Minor
Attachments: HDFS-8885.000.patch, HDFS-8885.001.patch
ByteRangeInputStream has to override the method {{available()}} , the super
class implementation returns 0. Clients using the method {{available()}} to
rely on available bytes information will end up with errors during reads if
WebhdfsFileSystem is used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711827#comment-14711827 ] kanaka kumar avvaru commented on HDFS-8836: --- Updated patch for jenkin reported errors Skip newline on empty files with getMerge -nl - Key: HDFS-8836 URL: https://issues.apache.org/jira/browse/HDFS-8836 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0, 2.7.1 Reporter: Jan Filipiak Assignee: kanaka kumar avvaru Priority: Trivial Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch Hello everyone, I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied or one would check the file size before. I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711882#comment-14711882
]
Zhe Zhang commented on HDFS-8833:
-
{{TestFSNamesystem}} passes consistently on local node.
{{testWriteStripedFileWithDNFailure}} is flaky even without the change. I'm
debugging.
Erasure coding: store EC schema and cell size in INodeFile and eliminate
notion of EC zones
---
Key: HDFS-8833
URL: https://issues.apache.org/jira/browse/HDFS-8833
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
We have [discussed |
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
storing EC schema with files instead of EC zones and recently revisited the
discussion under HDFS-8059.
As a recap, the _zone_ concept has severe limitations including renaming and
nested configuration. Those limitations are valid in encryption for security
reasons and it doesn't make sense to carry them over in EC.
This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
simplicity, we should first implement it as an xattr and consider memory
optimizations (such as moving it to file header) as a follow-on. We should
also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8894: -- Attachment: HDFS-8894-01.patch Some problems in earlier build. Uploading same patch again for jenkins new build. Set SO_KEEPALIVE on DN server sockets - Key: HDFS-8894 URL: https://issues.apache.org/jira/browse/HDFS-8894 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Nathan Roberts Assignee: kanaka kumar avvaru Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch SO_KEEPALIVE is not set on things like datastreamer sockets which can cause lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-6955: -- Attachment: HDFS-6955-03.patch updated patch for required check style errors. Test failure doesn't seem to be related to this patch DN should reserve disk space for a full block when creating tmp files - Key: HDFS-6955 URL: https://issues.apache.org/jira/browse/HDFS-6955 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Arpit Agarwal Assignee: kanaka kumar avvaru Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch HDFS-6898 is introducing disk space reservation for RBW files to avoid running out of disk space midway through block creation. This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8155) Support OAuth2 in WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711874#comment-14711874
]
Chris Nauroth commented on HDFS-8155:
-
Hi [~jghoman]. Nice work! I have a few comments.
# {{ConfRefreshTokenBasedAccessTokenProvider}},
{{ConfRefreshTokenBasedAccessTokenProvider}}: There are no timeouts specified
in the calls to the refresh URL. Timeouts can be controlled by calling
{{client.setConnectTimeout}} and {{client.setReadTimeout}}.
# {{AccessTokenProvider}}: *Optional* - consider extending {{Configured}} so
that it inherits the implementations of {{getConf}} and {{setConf}} for free.
# WebHDFS.md: Typo: toekns instead of tokens
# Please address the javac and checkstyle warnings.
[~chris.douglas], it looks like the most recent version of the patch has
addressed your feedback. Do you agree?
Support OAuth2 in WebHDFS
-
Key: HDFS-8155
URL: https://issues.apache.org/jira/browse/HDFS-8155
Project: Hadoop HDFS
Issue Type: New Feature
Components: webhdfs
Reporter: Jakob Homan
Assignee: Jakob Homan
Attachments: HDFS-8155-1.patch, HDFS-8155.002.patch
WebHDFS should be able to accept OAuth2 credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8784) BlockInfo#numNodes should be numStorages
[ https://issues.apache.org/jira/browse/HDFS-8784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711936#comment-14711936 ] Arpit Agarwal commented on HDFS-8784: - The NameNode does not support more than one replica of a given block per DataNode, so node/storage are interchangeable and node is probably better. BlockInfo#numNodes should be numStorages Key: HDFS-8784 URL: https://issues.apache.org/jira/browse/HDFS-8784 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Jagadesh Kiran N Attachments: HDFS-8784-00.patch, HDFS-8784-01.patch The method actually returns the number of storages holding a block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8833) Erasure coding: store EC schema and cell size in INodeFile and eliminate notion of EC zones
[
https://issues.apache.org/jira/browse/HDFS-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711937#comment-14711937
]
Zhe Zhang commented on HDFS-8833:
-
The failure will be addressed by HDFS-8704.
Erasure coding: store EC schema and cell size in INodeFile and eliminate
notion of EC zones
---
Key: HDFS-8833
URL: https://issues.apache.org/jira/browse/HDFS-8833
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS-7285
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8833-HDFS-7285-merge.00.patch,
HDFS-8833-HDFS-7285-merge.01.patch, HDFS-8833-HDFS-7285.02.patch
We have [discussed |
https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14357754page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14357754]
storing EC schema with files instead of EC zones and recently revisited the
discussion under HDFS-8059.
As a recap, the _zone_ concept has severe limitations including renaming and
nested configuration. Those limitations are valid in encryption for security
reasons and it doesn't make sense to carry them over in EC.
This JIRA aims to store EC schema and cell size on {{INodeFile}} level. For
simplicity, we should first implement it as an xattr and consider memory
optimizations (such as moving it to file header) as a follow-on. We should
also disable changing EC policy on a non-empty file / dir in the first phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Status: Patch Available (was: Open)
Attached a patch to handle the {{purgeLogsOlderThan}} calls at {{JournalNode}}
in a separate single threaded executor service so that edits are not blocked
for the purge log completion.
If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
send another RPC calls to Journalnodes
Key: HDFS-8771
URL: https://issues.apache.org/jira/browse/HDFS-8771
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takuya Fukudome
Assignee: kanaka kumar avvaru
Attachments: HDFS-8771-01.patch
In our cluster, edits has became huge(about 50GB) accidentally and our
Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
Namenode couldn't send other RPC calls to Journalnodes because
{{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
It will cause namenode shutting down.
I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5696) Examples for httpfs REST API incorrect on apache.org
[ https://issues.apache.org/jira/browse/HDFS-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1479#comment-1479 ] Harsh J commented on HDFS-5696: --- Thanks for reporting this. Care to submit a fix changing the op values to the right one? I suspect these may be leftovers from the Hoop days. bq. Not sure what the convention should be for specifying the user.name. Use hdfs? or a name that is obviously an example? Since these are curl based examples that also likely assume no kerberos setup, why not $USER or $(whoami) instead of a hardcoded value? Examples for httpfs REST API incorrect on apache.org Key: HDFS-5696 URL: https://issues.apache.org/jira/browse/HDFS-5696 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.2.0 Environment: NA Reporter: Casey Brotherton Priority: Trivial The examples provided for the httpfs REST API are incorrect. http://hadoop.apache.org/docs/r2.2.0/hadoop-hdfs-httpfs/index.html http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-hdfs-httpfs/index.html From the documentation: * HttpFS is a separate service from Hadoop NameNode. HttpFS itself is Java web-application and it runs using a preconfigured Tomcat bundled with HttpFS binary distribution. HttpFS HTTP web-service API calls are HTTP REST calls that map to a HDFS file system operation. For example, using the curl Unix command: $ curl http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt returns the contents of the HDFS /user/foo/README.txt file. $ curl http://httpfs-host:14000/webhdfs/v1/user/foo?op=list returns the contents of the HDFS /user/foo directory in JSON format. $ curl -X POST http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirs creates the HDFS /user/foo.bar directory. *** The commands have incorrect operations. ( Verified through source code in HttpFSFileSystem.java ) In addition, although the webhdfs documentation specifies user.name as optional, on my cluster, each action required a user.name It should be included in the short examples to allow for the greatest chance of success. Three examples rewritten: curl -i -L http://httpfs-host:14000/webhdfs/v1/user/foo/README.txt?op=openuser.name=hdfsuser; curl -i http://httpfs-host:14000/webhdfs/v1/user/foo/?op=liststatususer.name=hdfsuser; curl -i -X PUT http://httpfs-host:14000/webhdfs/v1/user/foo/bar?op=mkdirsuser.name=hdfsuser; Not sure what the convention should be for specifying the user.name. Use hdfs? or a name that is obviously an example? It would also be beneficial if the HTTPfs page linked to the webhdfs documentation page in the text instead of just on the menu sidebar. http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711131#comment-14711131
]
Hadoop QA commented on HDFS-8836:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 50s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 58s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 0s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 8s | The applied patch generated 2
new checkstyle issues (total was 45, now 47). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 26s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 53s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 22m 16s | Tests failed in
hadoop-common. |
| | | 68m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.cli.TestCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752203/HDFS-8836-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12114/console |
This message was automatically generated.
Skip newline on empty files with getMerge -nl
-
Key: HDFS-8836
URL: https://issues.apache.org/jira/browse/HDFS-8836
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.6.0, 2.7.1
Reporter: Jan Filipiak
Assignee: kanaka kumar avvaru
Priority: Trivial
Attachments: HDFS-8836-01.patch
Hello everyone,
I recently was in the need of using the new line option -nl with getMerge
because the files I needed to merge simply didn't had one. I was merging all
the files from one directory and unfortunately this directory also included
empty files, which effectively led to multiple newlines append after some
files. I needed to remove them manually afterwards.
In this situation it is maybe good to have another argument that allows
skipping empty files.
Thing one could try to implement this feature:
The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
return the number of bytes copied which would be convenient as one could
skip append the new line when 0 bytes where copied or one would check the
file size before.
I posted this Idea on the mailing list
http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Attachment: HDFS-8771-01.patch
If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
send another RPC calls to Journalnodes
Key: HDFS-8771
URL: https://issues.apache.org/jira/browse/HDFS-8771
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takuya Fukudome
Assignee: kanaka kumar avvaru
Attachments: HDFS-8771-01.patch
In our cluster, edits has became huge(about 50GB) accidentally and our
Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
Namenode couldn't send other RPC calls to Journalnodes because
{{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
It will cause namenode shutting down.
I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8581) count cmd calculate wrong when huge files exist in one folder
[ https://issues.apache.org/jira/browse/HDFS-8581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J.Andreina updated HDFS-8581: - Attachment: HDFS-8581.2.patch Patch is updated. Please review. count cmd calculate wrong when huge files exist in one folder - Key: HDFS-8581 URL: https://issues.apache.org/jira/browse/HDFS-8581 Project: Hadoop HDFS Issue Type: Bug Components: HDFS Reporter: tongshiquan Assignee: J.Andreina Priority: Minor Attachments: HDFS-8581.1.patch, HDFS-8581.2.patch If one directory such as /result exists about 20 files, then when execute hdfs dfs -count /, the result will go wrong. For all directories whose name after /result, file num will not be included. My cluster see as below, /result_1433858936 is the directory exist huge files, and files in /sparkJobHistory, /tmp, /user are not included vm-221:/export1/BigData/current # hdfs dfs -ls / 15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled. Found 9 items -rw-r--r-- 3 hdfs supergroup 0 2015-06-08 12:10 /PRE_CREATE_DIR.SUCCESS drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume drwx-- - hbase hadoop 0 2015-06-10 15:25 /hbase drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 /result_1433858936 drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count / 15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled. 1043 171536 1756375688 / vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS 15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled. 01 0 /PRE_CREATE_DIR.SUCCESS vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /flume 15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled. 10 0 /flume vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /hbase 15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled. 36 18 14807 /hbase vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /hyt 15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled. 10 0 /hyt vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /mr-history 15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled. 30 0 /mr-history vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936 15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled. 1001 171517 1756360881 /result_1433858936 vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory 15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled. 13 21785 /sparkJobHistory vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /tmp 15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled. 176 35958 /tmp vm-221:/export1/BigData/current # vm-221:/export1/BigData/current # hdfs dfs -count /user 15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled. 121 19077 /user -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1474#comment-1474
]
Hudson commented on HDFS-8900:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #304 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/304/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8900) Compact XAttrs to optimize memory footprint.
[
https://issues.apache.org/jira/browse/HDFS-8900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1475#comment-1475
]
Hudson commented on HDFS-8900:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2234 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2234/])
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) (yliu: rev
eee0d4563c62647cfaaed6605ee713aaf69add78)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFormat.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberMap.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestXAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrStorage.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SerialNumberManager.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrPermissionFilter.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/XAttrFeature.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/XAttrHelper.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
Compact XAttrs to optimize memory footprint.
Key: HDFS-8900
URL: https://issues.apache.org/jira/browse/HDFS-8900
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: 2.8.0
Attachments: HDFS-8900.001.patch, HDFS-8900.002.patch,
HDFS-8900.003.patch, HDFS-8900.004.patch, HDFS-8900.005.patch
{code}
private final ImmutableListXAttr xAttrs;
{code}
Currently we use above in XAttrFeature, it's not efficient from memory point
of view, since {{ImmutableList}} and {{XAttr}} have object memory overhead,
and each object has memory alignment.
We can use a {{byte[]}} in XAttrFeature and do some compact in {{XAttr}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8836) Skip newline on empty files with getMerge -nl
[ https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kanaka kumar avvaru updated HDFS-8836: -- Attachment: HDFS-8836-02.patch Skip newline on empty files with getMerge -nl - Key: HDFS-8836 URL: https://issues.apache.org/jira/browse/HDFS-8836 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.6.0, 2.7.1 Reporter: Jan Filipiak Assignee: kanaka kumar avvaru Priority: Trivial Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch Hello everyone, I recently was in the need of using the new line option -nl with getMerge because the files I needed to merge simply didn't had one. I was merging all the files from one directory and unfortunately this directory also included empty files, which effectively led to multiple newlines append after some files. I needed to remove them manually afterwards. In this situation it is maybe good to have another argument that allows skipping empty files. Thing one could try to implement this feature: The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't return the number of bytes copied which would be convenient as one could skip append the new line when 0 bytes where copied or one would check the file size before. I posted this Idea on the mailing list http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E but I didn't really get many responses, so I thought I my try this way. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-8771:
--
Attachment: HDFS-8771-02.patch
If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
send another RPC calls to Journalnodes
Key: HDFS-8771
URL: https://issues.apache.org/jira/browse/HDFS-8771
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takuya Fukudome
Assignee: kanaka kumar avvaru
Attachments: HDFS-8771-01.patch, HDFS-8771-02.patch
In our cluster, edits has became huge(about 50GB) accidentally and our
Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
Namenode couldn't send other RPC calls to Journalnodes because
{{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
It will cause namenode shutting down.
I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8771) If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not send another RPC calls to Journalnodes
[
https://issues.apache.org/jira/browse/HDFS-8771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711839#comment-14711839
]
kanaka kumar avvaru commented on HDFS-8771:
---
updated patch for checkstyle and white space erros reported in jenkins
If IPCLoggerChannel#purgeLogsOlderThan takes too long, Namenode could not
send another RPC calls to Journalnodes
Key: HDFS-8771
URL: https://issues.apache.org/jira/browse/HDFS-8771
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takuya Fukudome
Assignee: kanaka kumar avvaru
Attachments: HDFS-8771-01.patch, HDFS-8771-02.patch
In our cluster, edits has became huge(about 50GB) accidentally and our
Jounalnodes' disks were busy, therefore {{purgeLogsOlderThan}} took more than
30secs. If {{IPCLoggerChannel#purgeLogsOlderThan}} takes too much time,
Namenode couldn't send other RPC calls to Journalnodes because
{{o.a.h.hdfs.qjournal.client.IPCLoggerChannel}}'s executor is single thread.
It will cause namenode shutting down.
I think IPCLoggerChannel#purgeLogsOlderThan should not block other RPC calls
like sendEdits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8953) DataNode Metrics logging
kanaka kumar avvaru created HDFS-8953: - Summary: DataNode Metrics logging Key: HDFS-8953 URL: https://issues.apache.org/jira/browse/HDFS-8953 Project: Hadoop HDFS Issue Type: Improvement Reporter: kanaka kumar avvaru Assignee: kanaka kumar avvaru HDFS-8880 added metrics logging at NameNode. Similarly, this JIRA is to add a separate logger for metrics at DN -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711950#comment-14711950
]
Hadoop QA commented on HDFS-8836:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 22m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 23s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 3m 6s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 10s | The applied patch generated 2
new checkstyle issues (total was 45, now 46). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 6s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 23m 8s | Tests failed in
hadoop-common. |
| | | 73m 40s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752298/HDFS-8836-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/artifact/patchprocess/testrun_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12123/console |
This message was automatically generated.
Skip newline on empty files with getMerge -nl
-
Key: HDFS-8836
URL: https://issues.apache.org/jira/browse/HDFS-8836
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.6.0, 2.7.1
Reporter: Jan Filipiak
Assignee: kanaka kumar avvaru
Priority: Trivial
Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch
Hello everyone,
I recently was in the need of using the new line option -nl with getMerge
because the files I needed to merge simply didn't had one. I was merging all
the files from one directory and unfortunately this directory also included
empty files, which effectively led to multiple newlines append after some
files. I needed to remove them manually afterwards.
In this situation it is maybe good to have another argument that allows
skipping empty files.
Thing one could try to implement this feature:
The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
return the number of bytes copied which would be convenient as one could
skip append the new line when 0 bytes where copied or one would check the
file size before.
I posted this Idea on the mailing list
http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Fix Version/s: 2.8.0 Status: Patch Available (was: In Progress) Ready for review NameNode refresh doesn't remove DataNodes - Key: HDFS-8950 URL: https://issues.apache.org/jira/browse/HDFS-8950 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Daniel Templeton Assignee: Daniel Templeton Fix For: 2.8.0 Attachments: HDFS-8950.001.patch If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission. To reproduce: 1. Add a DN to dfs_hosts_allow 2. Refresh NN 3. Start DN. Now NN starts seeing DN. 4. Stop DN 5. Remove DN from dfs_hosts_allow 6. Refresh NN - NN is still reporting DN as being used by HDFS. This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: HDFS-8950.002.patch Just noticed an extraneous import in the patch. (Swing?!? WTF?) Removed and reposting patch. NameNode refresh doesn't remove DataNodes - Key: HDFS-8950 URL: https://issues.apache.org/jira/browse/HDFS-8950 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Daniel Templeton Assignee: Daniel Templeton Fix For: 2.8.0 Attachments: HDFS-8950.001.patch, HDFS-8950.002.patch If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission. To reproduce: 1. Add a DN to dfs_hosts_allow 2. Refresh NN 3. Start DN. Now NN starts seeing DN. 4. Stop DN 5. Remove DN from dfs_hosts_allow 6. Refresh NN - NN is still reporting DN as being used by HDFS. This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712233#comment-14712233 ] Hudson commented on HDFS-8846: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2254 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2254/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712265#comment-14712265 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #297 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/297/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712277#comment-14712277 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2235 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2235/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8950) NameNode refresh doesn't remove DataNodes
[ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated HDFS-8950: --- Attachment: (was: 28531.patch) NameNode refresh doesn't remove DataNodes - Key: HDFS-8950 URL: https://issues.apache.org/jira/browse/HDFS-8950 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.6.0 Reporter: Daniel Templeton Assignee: Daniel Templeton If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh, it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate some blocks to that DN as well during an MR job. This issue is independent from DN decommission. To reproduce: 1. Add a DN to dfs_hosts_allow 2. Refresh NN 3. Start DN. Now NN starts seeing DN. 4. Stop DN 5. Remove DN from dfs_hosts_allow 6. Refresh NN - NN is still reporting DN as being used by HDFS. This is different from decom because there DN is added to exclude list in addition to being removed from allowed list, and in that case everything works correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712126#comment-14712126 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #305 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/305/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8846) Add a unit test for INotify functionality across a layout version upgrade
[ https://issues.apache.org/jira/browse/HDFS-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712135#comment-14712135 ] Hudson commented on HDFS-8846: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #310 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/310/]) HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) (cmccabe: rev a4d9acc51d1a977bc333da17780c00c72e8546f1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-252-dfs-dir.tgz Add a unit test for INotify functionality across a layout version upgrade - Key: HDFS-8846 URL: https://issues.apache.org/jira/browse/HDFS-8846 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Reporter: Zhe Zhang Assignee: Zhe Zhang Fix For: 2.8.0 Attachments: HDFS-8846.00.patch, HDFS-8846.01.patch, HDFS-8846.02.patch, HDFS-8846.03.patch Per discussion under HDFS-8480, we should create some edit log files with old layout version, to test whether they can be correctly handled in upgrades. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
