[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962926#comment-14962926
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8660 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8660/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated HDFS-9237:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
3.0.0
Target Version/s: 3.0.0, 2.8.0
Status: Resolved (was: Patch Available)
Committed this to trunk and branch-2. Thanks [~brahmareddy] for your review and
thanks [~liuml07] for your review.
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962970#comment-14962970
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #549 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/549/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962898#comment-14962898
]
Tsuyoshi Ozawa commented on HDFS-9237:
--
+1, checking this in.
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962932#comment-14962932
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #564 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/564/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[
https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963165#comment-14963165
]
Hadoop QA commented on HDFS-8287:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 7s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:red}-1{color} | javac | 8m 38s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 55s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 41s | The applied patch generated 3
new checkstyle issues (total was 8, now 11). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 13s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 25s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 0m 31s | Tests passed in
hadoop-hdfs-client. |
| | | 49m 14s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767336/HDFS-8287.13.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7f0e1eb |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/13047/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13047/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13047/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13047/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13047/console |
This message was automatically generated.
> DFSStripedOutputStream.writeChunk should not wait for writing parity
> -
>
> Key: HDFS-8287
> URL: https://issues.apache.org/jira/browse/HDFS-8287
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Kai Sasaki
> Attachments: HDFS-8287-HDFS-7285.00.patch,
> HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch,
> HDFS-8287-HDFS-7285.03.patch, HDFS-8287-HDFS-7285.04.patch,
> HDFS-8287-HDFS-7285.05.patch, HDFS-8287-HDFS-7285.06.patch,
> HDFS-8287-HDFS-7285.07.patch, HDFS-8287-HDFS-7285.08.patch,
> HDFS-8287-HDFS-7285.09.patch, HDFS-8287-HDFS-7285.10.patch,
> HDFS-8287-HDFS-7285.11.patch, HDFS-8287-HDFS-7285.WIP.patch,
> HDFS-8287-performance-report.pdf, HDFS-8287.12.patch, HDFS-8287.13.patch,
> h8287_20150911.patch, jstack-dump.txt
>
>
> When a stripping cell is full, writeChunk computes and generates parity
> packets. It sequentially calls waitAndQueuePacket so that user client cannot
> continue to write data until it finishes.
> We should allow user client to continue writing instead but not blocking it
> when writing parity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963150#comment-14963150
]
Hudson commented on HDFS-9237:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #512 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/512/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-815) FileContext tests fail on Windows
[
https://issues.apache.org/jira/browse/HDFS-815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-815.
-
Resolution: Cannot Reproduce
Fix Version/s: 2.7.2
These don't appear to happen any more, so closing as cannot reproduce.
> FileContext tests fail on Windows
> -
>
> Key: HDFS-815
> URL: https://issues.apache.org/jira/browse/HDFS-815
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
> Environment: Windows
>Reporter: Konstantin Shvachko
> Fix For: 2.7.2
>
>
> The following FileContext-related tests are failing on windows because of
> incorrect use "test.build.data" system property for setting hdfs paths, which
> end up containing "C:" as a path component, which hdfs does not support.
> {code}
> org.apache.hadoop.fs.TestFcHdfsCreateMkdir
> org.apache.hadoop.fs.TestFcHdfsPermission
> org.apache.hadoop.fs.TestHDFSFileContextMainOperations
> org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
Steve Loughran created HDFS-9263:
Summary: tests are using /test/build/data; breaking Jenkins
Key: HDFS-9263
URL: https://issues.apache.org/jira/browse/HDFS-9263
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Environment: Jenkins
Reporter: Steve Loughran
Priority: Blocker
Some of the HDFS tests are using the path {{test/build/data}} to store files,
so leaking files which fail the new post-build RAT test checks on Jenkins (and
dirtying all development systems with paths which {{mvn clean}} will miss.
fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Status: Open (was: Patch Available) > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch, > HDFS-9070-trunk.04.patch, HDFS-9070-trunk.05.patch, HDFS-9070-trunk.06.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8631) WebHDFS : Support list/setQuota
[
https://issues.apache.org/jira/browse/HDFS-8631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963013#comment-14963013
]
J.Andreina commented on HDFS-8631:
--
Thanks [~surendrasingh] for the patch.
It looks great.
I have few comments :
1.
Both in NamespaceQuotaParam and in StoragespaceQuotaParam, can update below as
default value:
{code}
public static final String DEFAULT =
String.valueOf(HdfsConstants.QUOTA_DONT_SET);
instead of
public static final String DEFAULT = String.valueOf(Long.MAX_VALUE);
{code}
2.
Both in NamespaceQuotaParam and in StoragespaceQuotaParam, can update the
constructor code as below for more readability
{code}
super(DOMAIN, value, HdfsConstants.QUOTA_RESET, null);
instead of
super(DOMAIN, value, -1L, null);
{code}
3.
Inorder to make test run faster, start 0 datanodes in both testcases, as there
is no file is been written in tests.
4.
Can update the webhdfs document for supporting list/setQuota.
> WebHDFS : Support list/setQuota
> ---
>
> Key: HDFS-8631
> URL: https://issues.apache.org/jira/browse/HDFS-8631
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: nijel
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-8631-001.patch
>
>
> User is able do quota management from filesystem object. Same operation can
> be allowed trough REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Attachment: HDFS-9070-trunk.07.patch Hi, [~andreina]. I have uploaded a new patch, please let me know if it makes sense to you. Thanks > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch, > HDFS-9070-trunk.04.patch, HDFS-9070-trunk.05.patch, HDFS-9070-trunk.06.patch, > HDFS-9070-trunk.07.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963109#comment-14963109
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2498 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2498/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963125#comment-14963125
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2449 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2449/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9241) HDFS clients can't construct HdfsConfiguration instances
[
https://issues.apache.org/jira/browse/HDFS-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963045#comment-14963045
]
Steve Loughran commented on HDFS-9241:
--
I see your point, but I will note that it loses part of the benefit of the
whole idea of a leaner client "you miss out all the server-side dependencies"
> HDFS clients can't construct HdfsConfiguration instances
>
>
> Key: HDFS-9241
> URL: https://issues.apache.org/jira/browse/HDFS-9241
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
> Attachments: HDFS-9241.000.patch
>
>
> the changes for the hdfs client classpath make instantiating
> {{HdfsConfiguration}} from the client impossible; it only lives server side.
> This breaks any app which creates one.
> I know people will look at the {{@Private}} tag and say "don't do that then",
> but it's worth considering precisely why I, at least, do this: it's the only
> way to guarantee that the hdfs-default and hdfs-site resources get on the
> classpath, including all the security settings. It's precisely the use case
> which {{HdfsConfigurationLoader.init();}} offers internally to the hdfs code.
> What am I meant to do now?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HDFS-8287: - Attachment: HDFS-8287.13.patch > DFSStripedOutputStream.writeChunk should not wait for writing parity > - > > Key: HDFS-8287 > URL: https://issues.apache.org/jira/browse/HDFS-8287 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo Nicholas Sze >Assignee: Kai Sasaki > Attachments: HDFS-8287-HDFS-7285.00.patch, > HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch, > HDFS-8287-HDFS-7285.03.patch, HDFS-8287-HDFS-7285.04.patch, > HDFS-8287-HDFS-7285.05.patch, HDFS-8287-HDFS-7285.06.patch, > HDFS-8287-HDFS-7285.07.patch, HDFS-8287-HDFS-7285.08.patch, > HDFS-8287-HDFS-7285.09.patch, HDFS-8287-HDFS-7285.10.patch, > HDFS-8287-HDFS-7285.11.patch, HDFS-8287-HDFS-7285.WIP.patch, > HDFS-8287-performance-report.pdf, HDFS-8287.12.patch, HDFS-8287.13.patch, > h8287_20150911.patch, jstack-dump.txt > > > When a stripping cell is full, writeChunk computes and generates parity > packets. It sequentially calls waitAndQueuePacket so that user client cannot > continue to write data until it finishes. > We should allow user client to continue writing instead but not blocking it > when writing parity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962991#comment-14962991
]
Hudson commented on HDFS-9237:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1286 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1286/])
HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. (ozawa: rev
4c9c3955ae6cb5c3b67c8127eb7fa5449a0286d9)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureToleration.java
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity
[
https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963006#comment-14963006
]
Kai Sasaki commented on HDFS-8287:
--
[~rakeshr] Great. Thank you so much for reviewing.
{quote}
Any specific reason to use cachedThreadPool?
{quote}
Currently, there is no reason to use {{cachedThreadPool}} here. We can use
{{newFixedThreadPool}} because parity gen task is always threw only 1 at the
maximum. And these thread pools are used only by {{DFSStripedOutputStream}}. So
I think it might be better to keep inside {{DFSStripedOutputStream}} not
{{DFSClient}}.
> DFSStripedOutputStream.writeChunk should not wait for writing parity
> -
>
> Key: HDFS-8287
> URL: https://issues.apache.org/jira/browse/HDFS-8287
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Kai Sasaki
> Attachments: HDFS-8287-HDFS-7285.00.patch,
> HDFS-8287-HDFS-7285.01.patch, HDFS-8287-HDFS-7285.02.patch,
> HDFS-8287-HDFS-7285.03.patch, HDFS-8287-HDFS-7285.04.patch,
> HDFS-8287-HDFS-7285.05.patch, HDFS-8287-HDFS-7285.06.patch,
> HDFS-8287-HDFS-7285.07.patch, HDFS-8287-HDFS-7285.08.patch,
> HDFS-8287-HDFS-7285.09.patch, HDFS-8287-HDFS-7285.10.patch,
> HDFS-8287-HDFS-7285.11.patch, HDFS-8287-HDFS-7285.WIP.patch,
> HDFS-8287-performance-report.pdf, HDFS-8287.12.patch, h8287_20150911.patch,
> jstack-dump.txt
>
>
> When a stripping cell is full, writeChunk computes and generates parity
> packets. It sequentially calls waitAndQueuePacket so that user client cannot
> continue to write data until it finishes.
> We should allow user client to continue writing instead but not blocking it
> when writing parity.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9260) Improve performance and GC friendliness of startup and FBRs
[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963055#comment-14963055
]
Walter Su commented on HDFS-9260:
-
Excellent work [~sfriberg]!
1. Per your tests, the result looks good. But I'm more interested how does the
new datastructure incorporate with HDFS-7836.
2. To avoid moving entries in {{DatanodeStorageInfo}}, an easy way is to make
blockReport a serializable HashMap. Instead of iterating blockReport, we
iterate {{DatanodeStorageInfo}} with O(1) comparing block from blockReport.
> Improve performance and GC friendliness of startup and FBRs
> ---
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
>Affects Versions: 2.7.1
>Reporter: Staffan Friberg
>Assignee: Staffan Friberg
> Attachments: HDFS Block and Replica Management 20151013.pdf,
> HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change and also some help
> investigating/understanding a few outstanding issues if we are interested in
> moving forward with this.
> There seems to be some timing issues I hit when testing the patch, not sure
> if it is a bug in the patch or something else (most likely the earlier)...
> Tests that fail for me:
>The issues seems to be that the blocks are not on any storage, so no
> replication can occur causing the tests to fail in different ways.
>TestDecomission.testDecommision
>If I add a little sleep after the cleanup/delete things seem to work
>TestDFSStripedOutputStreamWithFailure
>A couple of tests fails in this class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9249) NPE thrown if an IOException is thrown in NameNode.
[
https://issues.apache.org/jira/browse/HDFS-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-9249:
--
Attachment: HDFS-9249.003.patch
Reset authentication method at end of the test to clean up.
> NPE thrown if an IOException is thrown in NameNode.
> -
>
> Key: HDFS-9249
> URL: https://issues.apache.org/jira/browse/HDFS-9249
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-9249.001.patch, HDFS-9249.002.patch,
> HDFS-9249.003.patch
>
>
> This issue was found when running test case
> TestBackupNode.testCheckpointNode, but upon closer look, the problem is not
> due to the test case.
> Looks like an IOException was thrown in
> try {
> initializeGenericKeys(conf, nsId, namenodeId);
> initialize(conf);
> try {
> haContext.writeLock();
> state.prepareToEnterState(haContext);
> state.enterState(haContext);
> } finally {
> haContext.writeUnlock();
> }
> causing the namenode to stop, but the namesystem was not yet properly
> instantiated, causing NPE.
> I tried to reproduce locally, but to no avail.
> Because I could not reproduce the bug, and the log does not indicate what
> caused the IOException, I suggest make this a supportability JIRA to log the
> exception for future improvement.
> Stacktrace
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:210)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:827)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:89)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1474)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.startBackupNode(TestBackupNode.java:102)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:130)
> The last few lines of log:
> 2015-10-14 19:45:07,807 INFO namenode.NameNode
> (NameNode.java:createNameNode(1422)) - createNameNode [-checkpoint]
> 2015-10-14 19:45:07,807 INFO impl.MetricsSystemImpl
> (MetricsSystemImpl.java:init(158)) - CheckpointNode metrics system started
> (again)
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(402)) - fs.defaultFS is
> hdfs://localhost:37835
> 2015-10-14 19:45:07,808 INFO namenode.NameNode
> (NameNode.java:setClientNamenodeAddress(422)) - Clients are to use
> localhost:37835 to access this namenode/service.
> 2015-10-14 19:45:07,810 INFO hdfs.MiniDFSCluster
> (MiniDFSCluster.java:shutdown(1708)) - Shutting down the Mini HDFS Cluster
> 2015-10-14 19:45:07,810 INFO namenode.FSNamesystem
> (FSNamesystem.java:stopActiveServices(1298)) - Stopping services started for
> active state
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:endCurrentLogSegment(1228)) - Ending log segment 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5306)) - NameNodeEditLogRoller was interrupted, exiting
> 2015-10-14 19:45:07,811 INFO namenode.FSEditLog
> (FSEditLog.java:printStatistics(703)) - Number of transactions: 3 Total time
> for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of
> syncs: 4 SyncTimes(ms): 2 1
> 2015-10-14 19:45:07,811 INFO namenode.FSNamesystem
> (FSNamesystem.java:run(5373)) - LazyPersistFileScrubber was interrupted,
> exiting
> 2015-10-14 19:45:07,822 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/edits_inprogress_001
> ->
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1/current/edits_001-003
> 2015-10-14 19:45:07,835 INFO namenode.FileJournalManager
> (FileJournalManager.java:finalizeLogSegment(142)) - Finalizing edits file
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name2/current/edits_inprogress_001
> ->
> /data/jenkins/workspace/CDH5.5.0-Hadoop-HDFS-2.6.0/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name2/current/edits_001-003
> 2015-10-14 19:45:07,836 INFO
[jira] [Commented] (HDFS-6589) TestDistributedFileSystem.testAllWithNoXmlDefaults failed intermittently
[
https://issues.apache.org/jira/browse/HDFS-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963306#comment-14963306
]
Wei-Chiu Chuang commented on HDFS-6589:
---
Unfortunately I can not reproduce the exception.
Initially I though it's a race condition, so I inserted Thread.yield() and also
tried to add Thread.sleep(), but still unable to reproduce it. I studied the
code but can not find a possible execution path for this to happen.
> TestDistributedFileSystem.testAllWithNoXmlDefaults failed intermittently
>
>
> Key: HDFS-6589
> URL: https://issues.apache.org/jira/browse/HDFS-6589
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Wei-Chiu Chuang
>
> https://builds.apache.org/job/PreCommit-HDFS-Build/7207 is clean
> https://builds.apache.org/job/PreCommit-HDFS-Build/7208 has the following
> failure. The code is essentially the same.
> Running the same test locally doesn't reproduce. A flaky test there.
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testDFSClient(TestDistributedFileSystem.java:263)
> at
> org.apache.hadoop.hdfs.TestDistributedFileSystem.testAllWithNoXmlDefaults(TestDistributedFileSystem.java:651)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963318#comment-14963318
]
Jagadesh Kiran N commented on HDFS-9263:
Thanks [~steve_l] for reporting the same, i am planning to work on this., if
you have already started work on this, please let me know
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Priority: Blocker
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9264) Minor cleanup of operations on FsVolumeList#volumes
[
https://issues.apache.org/jira/browse/HDFS-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-9264:
Status: Patch Available (was: Open)
> Minor cleanup of operations on FsVolumeList#volumes
> ---
>
> Key: HDFS-9264
> URL: https://issues.apache.org/jira/browse/HDFS-9264
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Attachments: HDFS-9264.01.patch
>
>
> We can use {{CopyOnWriteArrayList}} to simplify the operations on
> {{FsVolumeList#volumes}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9264) Minor cleanup of operations on FsVolumeList#volumes
[
https://issues.apache.org/jira/browse/HDFS-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-9264:
Attachment: HDFS-9264.01.patch
> Minor cleanup of operations on FsVolumeList#volumes
> ---
>
> Key: HDFS-9264
> URL: https://issues.apache.org/jira/browse/HDFS-9264
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Walter Su
>Assignee: Walter Su
>Priority: Minor
> Attachments: HDFS-9264.01.patch
>
>
> We can use {{CopyOnWriteArrayList}} to simplify the operations on
> {{FsVolumeList#volumes}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HDFS-7464) TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java 8
[
https://issues.apache.org/jira/browse/HDFS-7464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reopened HDFS-7464:
---
I am seeing this today with Java 7
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
Running org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
Tests run: 11, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 15.205 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA
testRefreshSuperUserGroupsConfiguration(org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA)
Time elapsed: 0.808 sec <<< FAILURE!
java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File
Exception between local host is: "weichiu-MBP.local/172.16.1.61"; destination
host is: "localhost":10872; : java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:235)
Results :
Failed tests:
TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration:235
refreshSuperUserGroupsConfiguration: End of File Exception between local host
is: "weichiu-MBP.local/172.16.1.61"; destination host is: "localhost":10872; :
java.io.EOFException; For more details see:
http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1>
> TestDFSAdminWithHA#testRefreshSuperUserGroupsConfiguration fails against Java
> 8
> ---
>
> Key: HDFS-7464
> URL: https://issues.apache.org/jira/browse/HDFS-7464
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
>
> From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/23/ :
> {code}
> REGRESSION:
> org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration
> Error Message:
> refreshSuperUserGroupsConfiguration: End of File Exception between local host
> is: "asf908.gq1.ygridcore.net/67.195.81.152"; destination host is:
> "localhost":12700; : java.io.EOFException; For more details see:
> http://wiki.apache.org/hadoop/EOFException expected:<0> but was:<-1>
> Stack Trace:
> java.lang.AssertionError: refreshSuperUserGroupsConfiguration: End of File
> Exception between local host is: "asf908.gq1.ygridcore.net/67.195.81.152";
> destination host is: "localhost":12700; : java.io.EOFException; For more
> details see: http://wiki.apache.org/hadoop/EOFException expected:<0> but
> was:<-1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdminWithHA.testRefreshSuperUserGroupsConfiguration(TestDFSAdminWithHA.java:228)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963432#comment-14963432 ] Dave Marion commented on HDFS-8486: --- Thanks for the quick response! > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.addendum.patch, HDFS-8486-branch-2.6.patch, > HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Attachment: HDFS-9263-001.patch
This covers -common and a couple of other places too, but it's where HDFS did
the most
# uses of test.build.data have been unified
# many buildups of test dirs now use something random, rather than a hard-coded
path like "dfs". This includes minidfs cluster...which should improve
parallelism on test runs.
# I've kept the bits of code that feed the paths into {{Path}} constructors
independent of the ones that want {{File}} instances. Why? Not sure what
happens in windows and whether the path stuff really wants forward-slash paths
(but then...what happens if maven has set test.build.data to an absolute path
in the local FS?)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-9263:
-
Status: Patch Available (was: Open)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9264) Minor cleanup of operations on FsVolumeList#volumes
Walter Su created HDFS-9264:
---
Summary: Minor cleanup of operations on FsVolumeList#volumes
Key: HDFS-9264
URL: https://issues.apache.org/jira/browse/HDFS-9264
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Walter Su
Assignee: Walter Su
Priority: Minor
We can use {{CopyOnWriteArrayList}} to simplify the operations on
{{FsVolumeList#volumes}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963398#comment-14963398 ] Dave Marion commented on HDFS-8486: --- Does this also affect 2.5.0? If so, can someone provide a patch for it? The branch-2.6 patches don't apply cleanly and the code is different. > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.addendum.patch, HDFS-8486-branch-2.6.patch, > HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N reassigned HDFS-9263:
--
Assignee: Jagadesh Kiran N
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963429#comment-14963429 ] Arpit Agarwal commented on HDFS-8486: - 2.5.0 is not affected. > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.addendum.patch, HDFS-8486-branch-2.6.patch, > HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8486) DN startup may cause severe data loss
[ https://issues.apache.org/jira/browse/HDFS-8486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963431#comment-14963431 ] Dave Marion commented on HDFS-8486: --- Thanks for the quick response! > DN startup may cause severe data loss > - > > Key: HDFS-8486 > URL: https://issues.apache.org/jira/browse/HDFS-8486 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 0.23.1, 2.0.0-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.6.1, 2.7.1 > > Attachments: HDFS-8486-branch-2.6.02.patch, > HDFS-8486-branch-2.6.addendum.patch, HDFS-8486-branch-2.6.patch, > HDFS-8486.patch, HDFS-8486.patch > > > A race condition between block pool initialization and the directory scanner > may cause a mass deletion of blocks in multiple storages. > If block pool initialization finds a block on disk that is already in the > replica map, it deletes one of the blocks based on size, GS, etc. > Unfortunately it _always_ deletes one of the blocks even if identical, thus > the replica map _must_ be empty when the pool is initialized. > The directory scanner starts at a random time within its periodic interval > (default 6h). If the scanner starts very early it races to populate the > replica map, causing the block pool init to erroneously delete blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963449#comment-14963449 ] Allen Wittenauer commented on HDFS-8914: We don't update the documentation for a released version of Hadoop as it is part of the release artifact. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963464#comment-14963464
]
Steve Loughran commented on HDFS-9263:
--
I'd already pulled out the IDE, though it'll take some time hardening —how
about you review the code
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Jagadesh Kiran N
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9249) NPE thrown if an IOException is thrown in NameNode.
[
https://issues.apache.org/jira/browse/HDFS-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963480#comment-14963480
]
Hadoop QA commented on HDFS-9249:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 27s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 9m 20s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 29s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 35s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 38s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 27s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 66m 12s | Tests failed in hadoop-hdfs. |
| | | 117m 51s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767356/HDFS-9249.003.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 7f0e1eb |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13048/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13048/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13048/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13048/console |
This message was automatically generated.
> NPE thrown if an IOException is thrown in NameNode.
> -
>
> Key: HDFS-9249
> URL: https://issues.apache.org/jira/browse/HDFS-9249
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.1
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
> Labels: supportability
> Attachments: HDFS-9249.001.patch, HDFS-9249.002.patch,
> HDFS-9249.003.patch
>
>
> This issue was found when running test case
> TestBackupNode.testCheckpointNode, but upon closer look, the problem is not
> due to the test case.
> Looks like an IOException was thrown in
> try {
> initializeGenericKeys(conf, nsId, namenodeId);
> initialize(conf);
> try {
> haContext.writeLock();
> state.prepareToEnterState(haContext);
> state.enterState(haContext);
> } finally {
> haContext.writeUnlock();
> }
> causing the namenode to stop, but the namesystem was not yet properly
> instantiated, causing NPE.
> I tried to reproduce locally, but to no avail.
> Because I could not reproduce the bug, and the log does not indicate what
> caused the IOException, I suggest make this a supportability JIRA to log the
> exception for future improvement.
> Stacktrace
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:906)
> at org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:210)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:827)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:89)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1474)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.startBackupNode(TestBackupNode.java:102)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:130)
> The last few lines of log:
> 2015-10-14 19:45:07,807 INFO namenode.NameNode
> (NameNode.java:createNameNode(1422)) - createNameNode [-checkpoint]
> 2015-10-14
[jira] [Created] (HDFS-9265) Use of undefined behavior in remote_block_reader causing deterministic crashes.
James Clampffer created HDFS-9265:
-
Summary: Use of undefined behavior in remote_block_reader causing
deterministic crashes.
Key: HDFS-9265
URL: https://issues.apache.org/jira/browse/HDFS-9265
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: James Clampffer
Priority: Blocker
The remote block reader relies on undefined behavior in how it uses
enable_shared_from_this.
http://en.cppreference.com/w/cpp/memory/enable_shared_from_this
The spec states a shared_ptr to an object inheriting from
enable_shared_from_this must be live before calling make_shared_from_this.
Calling make_shared_from_this without an existing shared_ptr is undefined
behavior and causes deterministic crashes when the code is built with GCC.
example:
class foo : public enable_shared_from_this {/*bar*/};
safe:
auto ptr1 = std::make_shared();
auto ptr2 = foo->make_shared_from_this();
broken:
foo *ptr = new foo();
auto ptr2 = foo->make_shared_from_this(); //no existing live shared_ptr
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963549#comment-14963549 ] Allen Wittenauer commented on HDFS-8914: a) it needs to get reviewed and committed b) when there is a new release > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963546#comment-14963546 ] Ravindra Babu commented on HDFS-8914: - When this patch will be deployed? > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-9263:
---
Assignee: Steve Loughran (was: Jagadesh Kiran N)
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9208) Disabling atime may fail clients like distCp
[
https://issues.apache.org/jira/browse/HDFS-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963503#comment-14963503
]
Chris Nauroth commented on HDFS-9208:
-
[~kihwal], thank you for the patch. I am +1 for making this change.
The test failures look unrelated, and I can't repro them locally. I also
manually tested {{hdfs dfs -copyFromLocal -p}} and {{hadoop distcp -pt}} with a
NameNode that has atime disabled. You can remove the now-unused import of
{{DFS_NAMENODE_ACCESSTIME_PRECISION_KEY}} to fix the Checkstyle warning. After
that, I think the patch will be ready.
> Disabling atime may fail clients like distCp
>
>
> Key: HDFS-9208
> URL: https://issues.apache.org/jira/browse/HDFS-9208
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9208.patch
>
>
> When atime is disabled, {{setTimes()}} throws an exception if the passed-in
> atime is not -1. But since atime is not -1, distCp fails when it tries to
> set the mtime and atime.
> There are several options:
> 1) make distCp check for 0 atime and call {{setTimes()}} with -1. I am not
> very enthusiastic about it.
> 2) make NN also accept 0 atime in addition to -1, when the atime support is
> disabled.
> 3) support setting mtime & atime regardless of the atime support. The main
> reason why atime is disabled is to avoid edit logging/syncing during
> {{getBlockLocations()}} read calls. Explicit setting can be allowed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963545#comment-14963545 ] Ravindra Babu commented on HDFS-8914: - When this patch will deployed? > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9266) hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HDFS-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963689#comment-14963689 ] Nemanja Matkovic commented on HDFS-9266: Cannot make this task to sub-task of HADOOP-11890 as this one is in HDFS, previous one is in hadoop-common... > hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HDFS-9266 > URL: https://issues.apache.org/jira/browse/HDFS-9266 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Original Estimate: 48h > Remaining Estimate: 48h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963714#comment-14963714 ] Ravindra Babu commented on HDFS-8914: - so it this patch reviewed currently? I see test report from Hadoop QA. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963792#comment-14963792 ] Ravindra Babu commented on HDFS-8914: - Still I did not get the process. Can you please submit this patch for review? > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963812#comment-14963812 ] Allen Wittenauer commented on HDFS-8914: The process is relatively simple: a) contributor submits a patch b) QA bot does static analysis to cover some of the review work c) committer reads through patch after QA bot says it is good d) committer gives a +1 if it is good, -1 or general feedback if it has issues e) if good, committer commits against source tree for releases that the committer feels it should be in, etc f) patch shows up in next release that contains that patch So even if this were to get committed right this second, it likely won't show up on the website for a very long time. Also keep in mind that there are literally hundreds of JIRAs in patch available state. It can take a while for someone to get to it. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963857#comment-14963857 ] Ravindra Babu commented on HDFS-8914: - Current report of QA says -1 due to white space in the end. What's next? I did not see contributor of patch fixing the issue later from 18th August. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9077) webhdfs client requires SPNEGO to do renew
[
https://issues.apache.org/jira/browse/HDFS-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964025#comment-14964025
]
Hadoop QA commented on HDFS-9077:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 22m 20s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 9m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 3m 21s | The applied patch generated 2
new checkstyle issues (total was 58, now 60). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 5m 22s | The patch appears to introduce 1
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 34s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 52m 52s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 35s | Tests passed in
hadoop-hdfs-client. |
| | | 111m 44s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-client |
| Failed unit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767391/HDFS-9077.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6144e01 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13054/console |
This message was automatically generated.
> webhdfs client requires SPNEGO to do renew
> --
>
> Key: HDFS-9077
> URL: https://issues.apache.org/jira/browse/HDFS-9077
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
>Assignee: HeeSoo Kim
> Attachments: HDFS-9077.patch
>
>
> Simple bug.
> webhdfs (the file system) doesn't pass delegation= in its REST call to renew
> the same token. This forces a SPNEGO (or other auth) instead of just
> renewing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API
[ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964024#comment-14964024 ] James Clampffer commented on HDFS-8766: --- Thanks Bob and Haohui for the feedback. Bob, I'll make those changes first so we can get that landed, then right after do the following: 1) I'm going to open up another jira for cli utilities in an effort to eventually make a c++ implementation of the hadoop CLI tools. I'll turn this stub into a unix-like cat executable so that it can be used for other things as well. Then move onto stat and other simple fs utils. 2) I can get rid of the INVALIDATE_MEMBER_PTR macro and use unique pointers, but I'd like to add a deleter to the smart pointers that nulls the pointer value on destruction. This sort of debugging tool would be more useful in the pipeline and block reader implementations because those are where most of the dangling pointers and references tend to originate. 3) Opened HDFS-9265 for this. It's more broken than I originally thought which explains all the double deletes that often manifest as calls to pure virtual methods I've been seeing in crashes. Inputstream's state owns a unique_ptr to the reader, and the reader is handing out shared_ptrs of itself to the continuation pipeline. So it's double deleting by design rather than a memory stomp. Fixing the make_shared_from_this usage alone isn't enough. I can take a stab at refactoring and fixing this sometime this week. 4) This is the conventional posix signature, this was intentional because google's guide says follow existing conventions if they are already well known. 5) Agree with Bob, he's already going to be refactoring a lot so I'll keep everything in the same place and let him move that. So the patch will cut out the c_api_test executable (1), add unique_ptrs (2) + get rid of INVALIDATE_MEMBER_PTR, omit (3) because it's in another jira, not change (4) because it falls into "exceptions to naming rules", and not change (5) because it's going to be handled in HDFS-9144. I'll also get rid of hdfs.h and use the one in the tree. Haohui, can you think of anything else I'm going to need to change? I think the current implementation, once rebased, should be sufficient to get on the development branch so Bob can begin HDFS-9144, and then I'll submit another patch with the changes I mentioned above tomorrow or Wednesday. > Implement a libhdfs(3) compatible API > - > > Key: HDFS-8766 > URL: https://issues.apache.org/jira/browse/HDFS-8766 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8766.HDFS-8707.000.patch, > HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, > HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, > HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch > > > Add a synchronous API that is compatible with the hdfs.h header used in > libhdfs and libhdfs3. This will make it possible for projects using > libhdfs/libhdfs3 to relink against libhdfspp with minimal changes. > This also provides a pure C interface that can be linked against projects > that aren't built in C++11 mode for various reasons but use the same > compiler. It also allows many other programming languages to access > libhdfspp through builtin FFI interfaces. > The libhdfs API is very similar to the posix file API which makes it easier > for programs built using posix filesystem calls to be modified to access HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9265) Use of undefined behavior in remote_block_reader causing deterministic crashes.
[
https://issues.apache.org/jira/browse/HDFS-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-9265:
--
Description:
The remote block reader relies on undefined behavior in how it uses
enable_shared_from_this.
http://en.cppreference.com/w/cpp/memory/enable_shared_from_this
The spec states a shared_ptr to an object inheriting from
enable_shared_from_this must be live before calling make_shared_from_this.
Calling make_shared_from_this without an existing shared_ptr is undefined
behavior and causes deterministic crashes when the code is built with GCC.
example:
class foo : public enable_shared_from_this {/*bar*/};
safe:
auto ptr1 = std::make_shared();
auto ptr2 = foo->make_shared_from_this();
broken:
foo *ptr = new foo();
auto ptr2 = foo->make_shared_from_this(); //no existing live shared_ptr
In order to fix the input stream should call std::make_shared and hang onto a
shared_ptr rather than unique_ptr to the block reader. The block reader will
then be free to call make_shared_from this as much as it wants without issue.
I think this will fix some double deletes and pure virtual method call
exceptions as a bonus. Both the shared_ptr and unique pointer think they own
the reader, whichever calls delete on the reader second will either segfault if
the memory has been reused or end up calling a pure virtual dtor which throws
(because after the first dtor executes vptr points to the base class vtable and
gcc stubs those in to throw).
was:
The remote block reader relies on undefined behavior in how it uses
enable_shared_from_this.
http://en.cppreference.com/w/cpp/memory/enable_shared_from_this
The spec states a shared_ptr to an object inheriting from
enable_shared_from_this must be live before calling make_shared_from_this.
Calling make_shared_from_this without an existing shared_ptr is undefined
behavior and causes deterministic crashes when the code is built with GCC.
example:
class foo : public enable_shared_from_this {/*bar*/};
safe:
auto ptr1 = std::make_shared();
auto ptr2 = foo->make_shared_from_this();
broken:
foo *ptr = new foo();
auto ptr2 = foo->make_shared_from_this(); //no existing live shared_ptr
> Use of undefined behavior in remote_block_reader causing deterministic
> crashes.
> ---
>
> Key: HDFS-9265
> URL: https://issues.apache.org/jira/browse/HDFS-9265
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Priority: Blocker
>
> The remote block reader relies on undefined behavior in how it uses
> enable_shared_from_this.
> http://en.cppreference.com/w/cpp/memory/enable_shared_from_this
> The spec states a shared_ptr to an object inheriting from
> enable_shared_from_this must be live before calling make_shared_from_this.
> Calling make_shared_from_this without an existing shared_ptr is undefined
> behavior and causes deterministic crashes when the code is built with GCC.
> example:
> class foo : public enable_shared_from_this {/*bar*/};
> safe:
> auto ptr1 = std::make_shared();
> auto ptr2 = foo->make_shared_from_this();
> broken:
> foo *ptr = new foo();
> auto ptr2 = foo->make_shared_from_this(); //no existing live shared_ptr
> In order to fix the input stream should call std::make_shared and hang onto a
> shared_ptr rather than unique_ptr to the block reader. The block reader will
> then be free to call make_shared_from this as much as it wants without issue.
>
> I think this will fix some double deletes and pure virtual method call
> exceptions as a bonus. Both the shared_ptr and unique pointer think they own
> the reader, whichever calls delete on the reader second will either segfault
> if the memory has been reused or end up calling a pure virtual dtor which
> throws (because after the first dtor executes vptr points to the base class
> vtable and gcc stubs those in to throw).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9098) Erasure coding: emulate race conditions among striped streamers in write pipeline
[
https://issues.apache.org/jira/browse/HDFS-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9098:
Target Version/s: 3.0.0
Status: Patch Available (was: Open)
> Erasure coding: emulate race conditions among striped streamers in write
> pipeline
> -
>
> Key: HDFS-9098
> URL: https://issues.apache.org/jira/browse/HDFS-9098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> Apparently the interleaving of events among {{StripedDataStreamer}}'s is very
> tricky to handle. [~walter.k.su] and [~jingzhao] have discussed several race
> conditions under HDFS-9040.
> Let's use FaultInjector to emulate different combinations of interleaved
> events.
> In particular, we should consider inject delays in the following places:
> # {{Streamer#endBlock}}
> # {{Streamer#locateFollowingBlock}}
> # {{Streamer#updateBlockForPipeline}}
> # {{Streamer#updatePipeline}}
> # {{OutputStream#writeChunk}}
> # {{OutputStream#close}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9098) Erasure coding: emulate race conditions among striped streamers in write pipeline
[
https://issues.apache.org/jira/browse/HDFS-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9098:
Attachment: HDFS-9098.wip.patch
WIP patch to demonstrate the idea. It leverages ideas from the [IMUnit |
http://mir.cs.illinois.edu/imunit/] paper and [sync_point testing |
https://github.com/cloudera/kudu/blob/master/src/kudu/util/sync_point.cc] in
Kudu.
The logic of {{syncPoint}} is still hacky because it needs to serve both as a
synchronization point and a fault injector. I'm working on a better structure.
The added {{TestStripedDataStreamers}} has a very simple test to emulate the
case where a second failure happens during the {{updateBlockForPipeline}} for
the first failure. I think ideally we need to create {{BEFORE}} and {{AFTER}}
events, like {{BEFORE_UPDATE_BLOCK_FOR_PIPELINE}} and
{{AFTER_UPDATE_BLOCK_FOR_PIPELINE}}. The {{WRITE_CHUNK}} event is also a little
tricky. We need to emulate the {{writeChunk}} for a specific offset.
> Erasure coding: emulate race conditions among striped streamers in write
> pipeline
> -
>
> Key: HDFS-9098
> URL: https://issues.apache.org/jira/browse/HDFS-9098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9098.wip.patch
>
>
> Apparently the interleaving of events among {{StripedDataStreamer}}'s is very
> tricky to handle. [~walter.k.su] and [~jingzhao] have discussed several race
> conditions under HDFS-9040.
> Let's use FaultInjector to emulate different combinations of interleaved
> events.
> In particular, we should consider inject delays in the following places:
> # {{Streamer#endBlock}}
> # {{Streamer#locateFollowingBlock}}
> # {{Streamer#updateBlockForPipeline}}
> # {{Streamer#updatePipeline}}
> # {{OutputStream#writeChunk}}
> # {{OutputStream#close}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9208) Disabling atime may fail clients like distCp
[
https://issues.apache.org/jira/browse/HDFS-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964062#comment-14964062
]
Hadoop QA commented on HDFS-9208:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 18m 19s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 25s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 32s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 14s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 49m 12s | Tests failed in hadoop-hdfs. |
| | | 95m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestDecommission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767469/HDFS-9208.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6144e01 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13055/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13055/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13055/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13055/console |
This message was automatically generated.
> Disabling atime may fail clients like distCp
>
>
> Key: HDFS-9208
> URL: https://issues.apache.org/jira/browse/HDFS-9208
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9208.patch, HDFS-9208.v2.patch
>
>
> When atime is disabled, {{setTimes()}} throws an exception if the passed-in
> atime is not -1. But since atime is not -1, distCp fails when it tries to
> set the mtime and atime.
> There are several options:
> 1) make distCp check for 0 atime and call {{setTimes()}} with -1. I am not
> very enthusiastic about it.
> 2) make NN also accept 0 atime in addition to -1, when the atime support is
> disabled.
> 3) support setting mtime & atime regardless of the atime support. The main
> reason why atime is disabled is to avoid edit logging/syncing during
> {{getBlockLocations()}} read calls. Explicit setting can be allowed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8766) Implement a libhdfs(3) compatible API
[ https://issues.apache.org/jira/browse/HDFS-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963963#comment-14963963 ] Bob Hansen commented on HDFS-8766: -- Haohui - thanks for those explicit and actionable items. I'm unsure if you see all of them as show-stoppers and some as small nits; it would be nice if you could clarify that when we have the narrow band of Jira to communicate. 1a. While copied code is The Devil, I believe strongly that in the current state of the code, we should not put forward declarations into a header file that we don't implement. That will just lead to linker errors later and sad consumers.Perhaps we should explicitly rename it to "hdfspp.h" to disambiguate it. 1b. I think we can make forward progress by landing this code with the integration tests as-is and having a separate Jira to integrate the integration test functionality with the existing tests. Bringing that scope into this bug is a major effort, and deserves its own Jira. 2. As we discussed, we will be losing a valuable debugging aid that has sussed out issues in integration testing already. I don't think we should ship with this in-place, but I would like to keep it for the short term. Again, let's make a "Before we merge with trunk" Jira to capture that so we can both reap the benefits of it during the high-risk portion of our development and not merge it into trunk. 3. This code won't pass integration test without that fix. As a matter of good form, it is good to break changes into atomic work units that stand alone, but as we discussed, we should be flexible during this phase of the development lifecycle so we don't get held up on issues of form. Can we accept that it's an improvement and accept? 4. I'm sure this is a minor nit that we can fix as we go, but I hope we're all on board that this is not a show-stopper. 5. Our plan is to take care of that with HDFS-9144 after this changeset lands. This is good; that will be better. The code does need to be re-factored to land on top of HDFS-9207. James - can you put together a patch that will apply to the current state of the HDFS-8707 branch (and perhaps fix the pread --> Pread renaming while you're there) and submit a patch that we can land? Haohui - can we land this patch with your points #1, 2, and 5 captured in new Jiras? > Implement a libhdfs(3) compatible API > - > > Key: HDFS-8766 > URL: https://issues.apache.org/jira/browse/HDFS-8766 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-8766.HDFS-8707.000.patch, > HDFS-8766.HDFS-8707.001.patch, HDFS-8766.HDFS-8707.002.patch, > HDFS-8766.HDFS-8707.003.patch, HDFS-8766.HDFS-8707.004.patch, > HDFS-8766.HDFS-8707.005.patch, HDFS-8766.HDFS-8707.006.patch > > > Add a synchronous API that is compatible with the hdfs.h header used in > libhdfs and libhdfs3. This will make it possible for projects using > libhdfs/libhdfs3 to relink against libhdfspp with minimal changes. > This also provides a pure C interface that can be linked against projects > that aren't built in C++11 mode for various reasons but use the same > compiler. It also allows many other programming languages to access > libhdfspp through builtin FFI interfaces. > The libhdfs API is very similar to the posix file API which makes it easier > for programs built using posix filesystem calls to be modified to access HDFS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9267) TestDiskError should get stored replicas through FsDatasetTestUtils.
Lei (Eddy) Xu created HDFS-9267:
---
Summary: TestDiskError should get stored replicas through
FsDatasetTestUtils.
Key: HDFS-9267
URL: https://issues.apache.org/jira/browse/HDFS-9267
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Affects Versions: 2.7.1
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
{{TestDiskError#testReplicationError}} scans local directories to verify blocks
and metadata files, which leaks the details of {{FsDataset}} implementation.
This JIRA will abstract the "scanning" operation to {{FsDatasetTestUtils}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9241) HDFS clients can't construct HdfsConfiguration instances
[
https://issues.apache.org/jira/browse/HDFS-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964044#comment-14964044
]
Colin Patrick McCabe commented on HDFS-9241:
bq. I see your point, but I will note that it loses part of the benefit of the
whole idea of a leaner client "you miss out all the server-side dependencies"
As we commented when the project started, the client is not "thin"-- most of
the problematic dependencies like Jackson, Guava, Jetty, Jersey, Netty, Xerces,
Protobuf are included in the client anyway. So the client/server split
provides no meaningful dependency isolation with or without this change.
> HDFS clients can't construct HdfsConfiguration instances
>
>
> Key: HDFS-9241
> URL: https://issues.apache.org/jira/browse/HDFS-9241
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
> Attachments: HDFS-9241.000.patch
>
>
> the changes for the hdfs client classpath make instantiating
> {{HdfsConfiguration}} from the client impossible; it only lives server side.
> This breaks any app which creates one.
> I know people will look at the {{@Private}} tag and say "don't do that then",
> but it's worth considering precisely why I, at least, do this: it's the only
> way to guarantee that the hdfs-default and hdfs-site resources get on the
> classpath, including all the security settings. It's precisely the use case
> which {{HdfsConfigurationLoader.init();}} offers internally to the hdfs code.
> What am I meant to do now?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9267) TestDiskError should get stored replicas through FsDatasetTestUtils.
[
https://issues.apache.org/jira/browse/HDFS-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-9267:
Attachment: HDFS-9267.00.patch
> TestDiskError should get stored replicas through FsDatasetTestUtils.
>
>
> Key: HDFS-9267
> URL: https://issues.apache.org/jira/browse/HDFS-9267
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-9267.00.patch
>
>
> {{TestDiskError#testReplicationError}} scans local directories to verify
> blocks and metadata files, which leaks the details of {{FsDataset}}
> implementation.
> This JIRA will abstract the "scanning" operation to {{FsDatasetTestUtils}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9267) TestDiskError should get stored replicas through FsDatasetTestUtils.
[
https://issues.apache.org/jira/browse/HDFS-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-9267:
Status: Patch Available (was: Open)
> TestDiskError should get stored replicas through FsDatasetTestUtils.
>
>
> Key: HDFS-9267
> URL: https://issues.apache.org/jira/browse/HDFS-9267
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-9267.00.patch
>
>
> {{TestDiskError#testReplicationError}} scans local directories to verify
> blocks and metadata files, which leaks the details of {{FsDataset}}
> implementation.
> This JIRA will abstract the "scanning" operation to {{FsDatasetTestUtils}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963717#comment-14963717 ] Allen Wittenauer commented on HDFS-8914: No, it isn't reviewed. You might find https://wiki.apache.org/hadoop/HowToContribute#Contributing_your_work informative. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9208) Disabling atime may fail clients like distCp
[
https://issues.apache.org/jira/browse/HDFS-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-9208:
-
Attachment: HDFS-9208.v2.patch
Thanks for the review, Chris. Attaching a new patch that removes the unused
import.
> Disabling atime may fail clients like distCp
>
>
> Key: HDFS-9208
> URL: https://issues.apache.org/jira/browse/HDFS-9208
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9208.patch, HDFS-9208.v2.patch
>
>
> When atime is disabled, {{setTimes()}} throws an exception if the passed-in
> atime is not -1. But since atime is not -1, distCp fails when it tries to
> set the mtime and atime.
> There are several options:
> 1) make distCp check for 0 atime and call {{setTimes()}} with -1. I am not
> very enthusiastic about it.
> 2) make NN also accept 0 atime in addition to -1, when the atime support is
> disabled.
> 3) support setting mtime & atime regardless of the atime support. The main
> reason why atime is disabled is to avoid edit logging/syncing during
> {{getBlockLocations()}} read calls. Explicit setting can be allowed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963918#comment-14963918
]
Hadoop QA commented on HDFS-8647:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 13s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 6 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 22s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 10s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 5s | The patch has 4 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | common tests | 10m 5s | Tests failed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 50m 44s | Tests failed in hadoop-hdfs. |
| | | 108m 48s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.fs.TestLocalFsFCStatistics |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767395/HDFS-8647-007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6144e01 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13053/console |
This message was automatically generated.
> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -
>
> Key: HDFS-8647
> URL: https://issues.apache.org/jira/browse/HDFS-8647
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch,
> HDFS-8647-003.patch, HDFS-8647-004.patch, HDFS-8647-004.patch,
> HDFS-8647-005.patch, HDFS-8647-006.patch, HDFS-8647-007.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy
> such as upgrade domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as
> useDelHint, blockHasEnoughRacks. That means when we have new block placement
> policy, we need to modify BlockManager to account for the new policy. Ideally
> BlockManager should ask BlockPlacementPolicy object instead. That will allow
> us to provide new BlockPlacementPolicy without changing BlockManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9266) hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 literals
Nemanja Matkovic created HDFS-9266: -- Summary: hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 literals Key: HDFS-9266 URL: https://issues.apache.org/jira/browse/HDFS-9266 Project: Hadoop HDFS Issue Type: Task Reporter: Nemanja Matkovic Assignee: Nemanja Matkovic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963898#comment-14963898 ] Lars Francke commented on HDFS-8914: I missed the -1, thanks for the heads up. I'll try to upload a new patch soon. > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9241) HDFS clients can't construct HdfsConfiguration instances
[
https://issues.apache.org/jira/browse/HDFS-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963652#comment-14963652
]
Mingliang Liu commented on HDFS-9241:
-
I think you two [~wheat9] and [~steve_l] have made good points. I'll update the
patch soon.
> HDFS clients can't construct HdfsConfiguration instances
>
>
> Key: HDFS-9241
> URL: https://issues.apache.org/jira/browse/HDFS-9241
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
> Attachments: HDFS-9241.000.patch
>
>
> the changes for the hdfs client classpath make instantiating
> {{HdfsConfiguration}} from the client impossible; it only lives server side.
> This breaks any app which creates one.
> I know people will look at the {{@Private}} tag and say "don't do that then",
> but it's worth considering precisely why I, at least, do this: it's the only
> way to guarantee that the hdfs-default and hdfs-site resources get on the
> classpath, including all the security settings. It's precisely the use case
> which {{HdfsConfigurationLoader.init();}} offers internally to the hdfs code.
> What am I meant to do now?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9237) NPE at TestDataNodeVolumeFailureToleration#tearDown
[
https://issues.apache.org/jira/browse/HDFS-9237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963667#comment-14963667
]
Brahma Reddy Battula commented on HDFS-9237:
Thanks [~ozawa] for review and commit!!
> NPE at TestDataNodeVolumeFailureToleration#tearDown
> ---
>
> Key: HDFS-9237
> URL: https://issues.apache.org/jira/browse/HDFS-9237
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Fix For: 3.0.0, 2.8.0
>
> Attachments: HDFS-9237.patch
>
>
> {noformat}
> Stack Trace:
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration.tearDown(TestDataNodeVolumeFailureToleration.java:79)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963662#comment-14963662
]
Brahma Reddy Battula commented on HDFS-8647:
[~mingma]
{quote}BlockManager should swap addedNode and delNodeHint.,{quote}
yes you are correct..
{quote}After the fix, the only failed unit test left is
{{TestBalancer#testBalancerWithPinnedBlocks}}. Can you please investigate if it
is related to the patch?{quote}
.{code}cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(capacities.length)
.hosts(new String[]{"localhost", "localhost"})
.racks(new String[]{"rack0",
"rack1"}).simulatedCapacities(capacities).build(){code}
2 DNs are started with "rack1". Ideally we should not create 2 DNs with the
same hostname.And Pinning depends on favoredNodes.DFSClient#create(..) only
uses host:port, if favoredNodes is created by {{new InetSocketAddress(ip,
port)}}
DFSClient will attempt a reverse lookup locally to get host:port, instead of
sending ip:port directly to NameNode.
.
MiniDFSCluster use fake hostname "host1.foo.com" to start DataNodes.DFSClient
doesn't use StaticMapping. So if DFSClient do reverse lookup, "127.0.0.1:8020"
becomes "localhost:8020".
{quote}{{public}} can be removed from {{public DatanodeStorageInfo
chooseReplicaToDelete}}.{quote}
it's overidden in test {{TestDnfencing#chooseReplicaToDelete}} .hence I make
this method visible for test.
Remaning changes are done and uploaded the patchHope I address all of your
comments..
> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -
>
> Key: HDFS-8647
> URL: https://issues.apache.org/jira/browse/HDFS-8647
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch,
> HDFS-8647-003.patch, HDFS-8647-004.patch, HDFS-8647-004.patch,
> HDFS-8647-005.patch, HDFS-8647-006.patch, HDFS-8647-007.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy
> such as upgrade domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as
> useDelHint, blockHasEnoughRacks. That means when we have new block placement
> policy, we need to modify BlockManager to account for the new policy. Ideally
> BlockManager should ask BlockPlacementPolicy object instead. That will allow
> us to provide new BlockPlacementPolicy without changing BlockManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9252) Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp.
[
https://issues.apache.org/jira/browse/HDFS-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-9252:
Attachment: HDFS-9252.01.patch
Thanks a lot for the feedbacks, [~cmccabe].
bq. It seems like
blockFile.getCanonicalPath().equals(listdir[j].getCanonicalPath())
Done
bq. Maybe it would be clearer if these methods were named getStoredDataLength
and getStoredGenerationStamp?
Done
bq. n assertEquals, the thing that we "expect" to see should come first, not
second.
The data length and genstemp read from disk are the expected values here, and
the in-memory {{ExternedBlock}}s are the actual value. It is also consistent
with the above data length tests.
Would appreciate much to have another review. Thanks!
> Change TestFileTruncate to use FsDatasetTestUtils to get block file size and
> genstamp.
> --
>
> Key: HDFS-9252
> URL: https://issues.apache.org/jira/browse/HDFS-9252
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-9252.00.patch, HDFS-9252.01.patch
>
>
> {{TestFileTruncate}} verifies block size and genstamp by directly accessing
> the local filesystem, e.g.:
> {code}
> assertTrue(cluster.getBlockMetadataFile(dn0,
>newBlock.getBlock()).getName().endsWith(
>newBlock.getBlock().getGenerationStamp() + ".meta"));
> {code}
> Lets abstract the fsdataset-special logic behind FsDatasetTestUtils.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9077) webhdfs client requires SPNEGO to do renew
[ https://issues.apache.org/jira/browse/HDFS-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HeeSoo Kim updated HDFS-9077: - Status: Patch Available (was: In Progress) > webhdfs client requires SPNEGO to do renew > -- > > Key: HDFS-9077 > URL: https://issues.apache.org/jira/browse/HDFS-9077 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim > Attachments: HDFS-9077.patch > > > Simple bug. > webhdfs (the file system) doesn't pass delegation= in its REST call to renew > the same token. This forces a SPNEGO (or other auth) instead of just > renewing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9266) hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 literals
[ https://issues.apache.org/jira/browse/HDFS-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963705#comment-14963705 ] Nemanja Matkovic commented on HDFS-9266: For context on why things are broken up the way they are, see this issue > hadoop-hdfs - Avoid unsafe split and append on fields that might be IPv6 > literals > - > > Key: HDFS-9266 > URL: https://issues.apache.org/jira/browse/HDFS-9266 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Nemanja Matkovic >Assignee: Nemanja Matkovic > Labels: ipv6 > Original Estimate: 48h > Remaining Estimate: 48h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Babu updated HDFS-8914: Attachment: (was: HDFS-8914.1.patch.txt) > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8914) Documentation conflict regarding fail-over of Namenode
[ https://issues.apache.org/jira/browse/HDFS-8914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Babu updated HDFS-8914: Attachment: HDFS-8914.1.patch.txt Review this patch > Documentation conflict regarding fail-over of Namenode > -- > > Key: HDFS-8914 > URL: https://issues.apache.org/jira/browse/HDFS-8914 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 2.7.1 > Environment: Documentation page in live >Reporter: Ravindra Babu >Assignee: Lars Francke >Priority: Trivial > Attachments: HDFS-8914.1.patch > > > Please refer to these two links and correct one of them. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > The NameNode machine is a single point of failure for an HDFS cluster. If the > NameNode machine fails, manual intervention is necessary. Currently, > automatic restart and failover of the NameNode software to another machine is > not supported. > http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html > The HDFS High Availability feature addresses the above problems by providing > the option of running two redundant NameNodes in the same cluster in an > Active/Passive configuration with a hot standby. This allows a fast failover > to a new NameNode in the case that a machine crashes, or a graceful > administrator-initiated failover for the purpose of planned maintenance. > Please update hdfsDesign article with same facts to avoid confusion in > Reader's mind.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9252) Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp.
[
https://issues.apache.org/jira/browse/HDFS-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964229#comment-14964229
]
Colin Patrick McCabe commented on HDFS-9252:
bq. The data length and genstemp read from disk are the expected values here,
and the in-memory {{ExternedBlock}}s are the actual value. It is also
consistent with the above data length tests.
OK.
Thanks for fixing the other stuff. +1
> Change TestFileTruncate to use FsDatasetTestUtils to get block file size and
> genstamp.
> --
>
> Key: HDFS-9252
> URL: https://issues.apache.org/jira/browse/HDFS-9252
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-9252.00.patch, HDFS-9252.01.patch
>
>
> {{TestFileTruncate}} verifies block size and genstamp by directly accessing
> the local filesystem, e.g.:
> {code}
> assertTrue(cluster.getBlockMetadataFile(dn0,
>newBlock.getBlock()).getName().endsWith(
>newBlock.getBlock().getGenerationStamp() + ".meta"));
> {code}
> Lets abstract the fsdataset-special logic behind FsDatasetTestUtils.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7087) Ability to list /.reserved
[
https://issues.apache.org/jira/browse/HDFS-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964253#comment-14964253
]
Hadoop QA commented on HDFS-7087:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12766177/HDFS-7087.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8175c4f |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13061/console |
This message was automatically generated.
> Ability to list /.reserved
> --
>
> Key: HDFS-7087
> URL: https://issues.apache.org/jira/browse/HDFS-7087
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.6.0
>Reporter: Andrew Wang
>Assignee: Xiao Chen
> Attachments: HDFS-7087.001.patch, HDFS-7087.002.patch,
> HDFS-7087.draft.patch
>
>
> We have two special paths within /.reserved now, /.reserved/.inodes and
> /.reserved/raw. It seems like we should be able to list /.reserved to see
> them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964270#comment-14964270
]
Hudson commented on HDFS-9250:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #569 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/569/])
HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. (wang: rev
8175c4f6b9fc17ff2e0fc568d798f9b6f2487696)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestLocatedBlock.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3059) ssl-server.xml causes NullPointer
[
https://issues.apache.org/jira/browse/HDFS-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964321#comment-14964321
]
Andrew Wang commented on HDFS-3059:
---
Hi Xiao, code-wise looks functionally good, only stylistic comments:
* I'm not a fan of this conditional behavior buried in {{initialize}}; how do
you feel about moving starting infoServer to a new function, and calling it
before we do startCheckpointThread in main? This makes this behavior difference
more explicit.
* The new DFSConfigKeys constants should end in _KEY like the other variables
in this file.
> ssl-server.xml causes NullPointer
> -
>
> Key: HDFS-3059
> URL: https://issues.apache.org/jira/browse/HDFS-3059
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, security
>Affects Versions: 2.7.1
> Environment: in core-site.xml:
> {code:xml}
>
> hadoop.security.authentication
> kerberos
>
>
> hadoop.security.authorization
> true
>
> {code}
> in hdfs-site.xml:
> {code:xml}
>
> dfs.https.server.keystore.resource
> /etc/hadoop/conf/ssl-server.xml
>
>
> dfs.https.enable
> true
>
>
> ...other security props
>
> {code}
>Reporter: Evert Lammerts
>Assignee: Xiao Chen
>Priority: Minor
> Labels: BB2015-05-TBR
> Attachments: HDFS-3059.02.patch, HDFS-3059.03.patch,
> HDFS-3059.04.patch, HDFS-3059.05.patch, HDFS-3059.06.patch, HDFS-3059.patch,
> HDFS-3059.patch.2
>
>
> If ssl is enabled (dfs.https.enable) but ssl-server.xml is not available, a
> DN will crash during startup while setting up an SSL socket with a
> NullPointerException:
> {noformat}12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector:
> useKerb = false, useCerts = true
> jetty.ssl.password : jetty.ssl.keypassword : 12/03/07 17:08:36 INFO
> mortbay.log: jetty-6.1.26.cloudera.1
> 12/03/07 17:08:36 INFO mortbay.log: Started
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:36 DEBUG security.Krb5AndCertsSslSocketConnector: Creating new
> KrbServerSocket for: 0.0.0.0
> 12/03/07 17:08:36 WARN mortbay.log: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475: java.io.IOException:
> !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 WARN mortbay.log: failed Server@604788d5:
> java.io.IOException: !JsseListener: java.lang.NullPointerException
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> Krb5AndCertsSslSocketConnector@0.0.0.0:50475
> 12/03/07 17:08:36 INFO mortbay.log: Stopped
> selectchannelconnec...@p-worker35.alley.sara.nl:1006
> 12/03/07 17:08:37 INFO datanode.DataNode: Waiting for threadgroup to exit,
> active threads is 0{noformat}
> The same happens if I set an absolute path to an existing
> dfs.https.server.keystore.resource - in this case the file cannot be found
> but not even a WARN is given.
> Since in dfs.https.server.keystore.resource we know we need to have 4
> properties specified (ssl.server.truststore.location,
> ssl.server.keystore.location, ssl.server.keystore.password, and
> ssl.server.keystore.keypassword) we should check if they are set and throw an
> IOException if they are not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964235#comment-14964235
]
Xiao Chen commented on HDFS-9250:
-
Thanks very much [~andrew.wang] for the review!
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7087) Ability to list /.reserved
[ https://issues.apache.org/jira/browse/HDFS-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964261#comment-14964261 ] Andrew Wang commented on HDFS-7087: --- Looks good, thanks for revving Xiao. Patch needs a small rebase for TestDFSShell. Only two little nits: * FSNamesystem.getCTime is marked VisibleForTesting. What we could do is pass the cTime as a parameter to createReservedStatuses which decouples FSN and FSDir a bit more. Also would be good to add something to createReservedStatuses javadoc about how this is called only once during NN initialization. * Add a Precondition check that reservedStatuses is not null before use? Maybe in getReservedStatuses. This also serves as documentation that createReservedStatuses should be called first. > Ability to list /.reserved > -- > > Key: HDFS-7087 > URL: https://issues.apache.org/jira/browse/HDFS-7087 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Andrew Wang >Assignee: Xiao Chen > Attachments: HDFS-7087.001.patch, HDFS-7087.002.patch, > HDFS-7087.draft.patch > > > We have two special paths within /.reserved now, /.reserved/.inodes and > /.reserved/raw. It seems like we should be able to list /.reserved to see > them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9241) HDFS clients can't construct HdfsConfiguration instances
[
https://issues.apache.org/jira/browse/HDFS-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964272#comment-14964272
]
Haohui Mai commented on HDFS-9241:
--
bq. most of the problematic dependencies like Jackson, Guava, Jetty, Jersey,
Netty, Xerces, Protobuf are included in the client anyway
I found that the above statement inaccurate. Many of the above dependency comes
as transitive dependency from hadoop-common. They are not needed from
hadoop-hdfs-client. They can be excluded as what have been done in hadoop
client. Just to give an example, we cleaned up the implementation of log4j in
the module (HDFS-6564). I don't see how this can be done correctly without the
breakdown.
As the refactor continues to break down hadoop-common into server / client
modules, I anticipate that this is a non-issue. The shadowing approach is
complementary. However, having a client that shadows 2000 classes that it
doesn't need that it happens to work at a first glance but it doesn't seems
elegant and responsible in the long term.
> HDFS clients can't construct HdfsConfiguration instances
>
>
> Key: HDFS-9241
> URL: https://issues.apache.org/jira/browse/HDFS-9241
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: Steve Loughran
>Assignee: Mingliang Liu
> Attachments: HDFS-9241.000.patch
>
>
> the changes for the hdfs client classpath make instantiating
> {{HdfsConfiguration}} from the client impossible; it only lives server side.
> This breaks any app which creates one.
> I know people will look at the {{@Private}} tag and say "don't do that then",
> but it's worth considering precisely why I, at least, do this: it's the only
> way to guarantee that the hdfs-default and hdfs-site resources get on the
> classpath, including all the security settings. It's precisely the use case
> which {{HdfsConfigurationLoader.init();}} offers internally to the hdfs code.
> What am I meant to do now?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9117) Config file reader / options classes for libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9117: - Attachment: HDFS-9117.HDFS-8707.001.patch Implemented config files and Options creations. Gluing the options to constructing the client will occur with HDFS-9144. > Config file reader / options classes for libhdfs++ > -- > > Key: HDFS-9117 > URL: https://issues.apache.org/jira/browse/HDFS-9117 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9117.HDFS-8707.001.patch > > > For environmental compatability with HDFS installations, libhdfs++ should be > able to read the configurations from Hadoop XML files and behave in line with > the Java implementation. > Most notably, machine names and ports should be readable from Hadoop XML > configuration files. > Similarly, an internal Options architecture for libhdfs++ should be developed > to efficiently transport the configuration information within the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9117) Config file reader / options classes for libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9117: - Status: Patch Available (was: Open) > Config file reader / options classes for libhdfs++ > -- > > Key: HDFS-9117 > URL: https://issues.apache.org/jira/browse/HDFS-9117 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9117.HDFS-8707.001.patch > > > For environmental compatability with HDFS installations, libhdfs++ should be > able to read the configurations from Hadoop XML files and behave in line with > the Java implementation. > Most notably, machine names and ports should be readable from Hadoop XML > configuration files. > Similarly, an internal Options architecture for libhdfs++ should be developed > to efficiently transport the configuration information within the system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9124) NullPointerException when underreplicated blocks are there
[
https://issues.apache.org/jira/browse/HDFS-9124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964219#comment-14964219
]
Lei (Eddy) Xu commented on HDFS-9124:
-
Thanks a lot for reporting this, [~syedakram]. It looks like {{datanode.data}}
has not been initialized yet before receiving WRITE_BLOCK requests.
Did you see this particular DN restarts at that time?
> NullPointerException when underreplicated blocks are there
> --
>
> Key: HDFS-9124
> URL: https://issues.apache.org/jira/browse/HDFS-9124
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Syed Akram
>Assignee: Syed Akram
>
> 2015-09-22 09:48:47,830 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: dn1:50010:DataXceiver error
> processing WRITE_BLOCK operation src: /dn1:42973 dst: /dn2:50010
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:186)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:677)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
> at java.lang.Thread.run(Thread.java:744)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9250:
--
Summary: Add Precondition check to LocatedBlock#addCachedLoc (was:
LocatedBlock#addCachedLoc may throw ArrayStoreException when cache is empty)
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9250) LocatedBlock#addCachedLoc may throw ArrayStoreException when cache is empty
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964218#comment-14964218
]
Andrew Wang commented on HDFS-9250:
---
LGTM, thanks Xiao. I'll update the summary and commit this.
> LocatedBlock#addCachedLoc may throw ArrayStoreException when cache is empty
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9250:
--
Issue Type: Improvement (was: Bug)
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9250:
--
Affects Version/s: 2.4.0
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9250:
--
Resolution: Fixed
Fix Version/s: 2.8.0
Target Version/s: 2.8.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2, thanks for the contribution Xiao!
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964233#comment-14964233
]
Andrew Wang commented on HDFS-9263:
---
I retriggered the build on this, some of these seem unrelated. Running locally
I did see some errors in the changed tests, so will likely need a rev besides
this.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9077) webhdfs client requires SPNEGO to do renew
[ https://issues.apache.org/jira/browse/HDFS-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HeeSoo Kim updated HDFS-9077: - Status: In Progress (was: Patch Available) > webhdfs client requires SPNEGO to do renew > -- > > Key: HDFS-9077 > URL: https://issues.apache.org/jira/browse/HDFS-9077 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim > Attachments: HDFS-9077.001.patch, HDFS-9077.patch > > > Simple bug. > webhdfs (the file system) doesn't pass delegation= in its REST call to renew > the same token. This forces a SPNEGO (or other auth) instead of just > renewing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964241#comment-14964241
]
Andrew Wang commented on HDFS-9263:
---
Other sort of related comment is that we have all these temp dirs defined in
hadoop-project-dist/pom.xml:
{noformat}
${project.build.directory}/test
${project.build.directory}/test/data
${project.build.directory}/log
${project.build.directory}/test-classes/webapps
${project.build.directory}/test-classes
${project.build.directory}/test-classes
{noformat}
Some of these are dupes, it'd be nice to get rid of whichever ones are less
popular.
> tests are using /test/build/data; breaking Jenkins
> --
>
> Key: HDFS-9263
> URL: https://issues.apache.org/jira/browse/HDFS-9263
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
> Environment: Jenkins
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HDFS-9263-001.patch
>
>
> Some of the HDFS tests are using the path {{test/build/data}} to store files,
> so leaking files which fail the new post-build RAT test checks on Jenkins
> (and dirtying all development systems with paths which {{mvn clean}} will
> miss.
> fix
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9079) Erasure coding: preallocate multiple generation stamps and serialize updates from data streamers
[
https://issues.apache.org/jira/browse/HDFS-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-9079:
Attachment: HDFS-9079.04.patch
The only locally reproducible failure is {{testDatanodeFailureRandomLength}}.
Adding a fix to clear {{currentPackets}} when allocating new blocks.
> Erasure coding: preallocate multiple generation stamps and serialize updates
> from data streamers
>
>
> Key: HDFS-9079
> URL: https://issues.apache.org/jira/browse/HDFS-9079
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: HDFS-7285
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9079-HDFS-7285.00.patch, HDFS-9079.01.patch,
> HDFS-9079.02.patch, HDFS-9079.03.patch, HDFS-9079.04.patch
>
>
> A non-striped DataStreamer goes through the following steps in error handling:
> {code}
> 1) Finds error => 2) Asks NN for new GS => 3) Gets new GS from NN => 4)
> Applies new GS to DN (createBlockOutputStream) => 5) Ack from DN => 6)
> Updates block on NN
> {code}
> To simplify the above we can preallocate GS when NN creates a new striped
> block group ({{FSN#createNewBlock}}). For each new striped block group we can
> reserve {{NUM_PARITY_BLOCKS}} GS's. Then steps 1~3 in the above sequence can
> be saved. If more than {{NUM_PARITY_BLOCKS}} errors have happened we
> shouldn't try to further recover anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9251) Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly creating Files in tests code.
[
https://issues.apache.org/jira/browse/HDFS-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964246#comment-14964246
]
Colin Patrick McCabe commented on HDFS-9251:
+1 pending Jenkins. Thanks, [~eddyxu].
> Refactor TestWriteToReplica and TestFsDatasetImpl to avoid explicitly
> creating Files in tests code.
> ---
>
> Key: HDFS-9251
> URL: https://issues.apache.org/jira/browse/HDFS-9251
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-9251.00.patch, HDFS-9251.01.patch,
> HDFS-9251.02.patch
>
>
> In {{TestWriteToReplica}} and {{TestFsDatasetImpl}}, tests directly creates
> block and metadata files:
> {code}
> replicaInfo.getBlockFile().createNewFile();
> replicaInfo.getMetaFile().createNewFile();
> {code}
> It leaks the implementation details of {{FsDatasetImpl}}. This JIRA proposes
> to use {{FsDatasetImplTestUtils}} (HDFS-9188) to create replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9077) webhdfs client requires SPNEGO to do renew
[ https://issues.apache.org/jira/browse/HDFS-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HeeSoo Kim updated HDFS-9077: - Attachment: HDFS-9077.001.patch > webhdfs client requires SPNEGO to do renew > -- > > Key: HDFS-9077 > URL: https://issues.apache.org/jira/browse/HDFS-9077 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: HeeSoo Kim > Attachments: HDFS-9077.001.patch, HDFS-9077.patch > > > Simple bug. > webhdfs (the file system) doesn't pass delegation= in its REST call to renew > the same token. This forces a SPNEGO (or other auth) instead of just > renewing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9259) Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario
[
https://issues.apache.org/jira/browse/HDFS-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964250#comment-14964250
]
Colin Patrick McCabe commented on HDFS-9259:
That seems reasonable to me. It sounds like we could have some new client
configs for this... with {{client}} somewhere in the config key name.
> Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario
> --
>
> Key: HDFS-9259
> URL: https://issues.apache.org/jira/browse/HDFS-9259
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Mingliang Liu
>
> We recently found that cross-DC hdfs write could be really slow. Further
> investigation identified that is due to SendBufferSize and ReceiveBufferSize
> used for hdfs write. The test ran "hadoop -fs -copyFromLocal" of a 256MB file
> across DC with different SendBufferSize and ReceiveBufferSize values. The
> results showed that c much faster than b; b is faster than a.
> a. SendBufferSize=128k, ReceiveBufferSize=128k (hdfs default setting).
> b. SendBufferSize=128K, ReceiveBufferSize=not set(TCP auto tuning).
> c. SendBufferSize=not set, ReceiveBufferSize=not set(TCP auto tuning for both)
> HDFS-8829 has enabled scenario b. We would like to enable scenario c by
> making SendBufferSize configurable at DFSClient side. Cc: [~cmccabe] [~He
> Tianyi] [~kanaka] [~vinayrpet].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
[
https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964296#comment-14964296
]
Hadoop QA commented on HDFS-9263:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 9m 40s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 36 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 16s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 6s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 6m 56s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 30m 18s | Tests failed in hadoop-hdfs. |
| | | 64m 10s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots |
| | hadoop.fs.TestUrlStreamHandler |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.server.namenode.TestFileLimit |
| | hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot |
| | hadoop.hdfs.server.namenode.TestEditLogAutoroll |
| | hadoop.TestRefreshCallQueue |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.cli.TestCryptoAdminCLI |
| | hadoop.fs.viewfs.TestViewFsWithAcls |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestHostsFiles |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.fs.contract.hdfs.TestHDFSContractDelete |
| | hadoop.hdfs.server.namenode.TestFileContextAcl |
| | hadoop.fs.TestFcHdfsSetUMask |
| | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.hdfs.server.namenode.TestFSDirectory |
| | hadoop.fs.contract.hdfs.TestHDFSContractOpen |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.fs.contract.hdfs.TestHDFSContractMkdir |
| | hadoop.fs.contract.hdfs.TestHDFSContractAppend |
| | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.server.namenode.TestMalformedURLs |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.server.namenode.TestRecoverStripedBlocks |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.datanode.TestCachingStrategy |
| | hadoop.hdfs.server.namenode.TestAddBlockRetry |
| | hadoop.fs.TestSymlinkHdfsFileSystem |
| | hadoop.fs.viewfs.TestViewFsDefaultValue |
| | hadoop.fs.TestSymlinkHdfsFileContext |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestFSInputChecker |
| | hadoop.hdfs.web.TestWebHDFSXAttr |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol |
| | hadoop.cli.TestAclCLI |
| | hadoop.hdfs.server.namenode.ha.TestHAMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap |
| | hadoop.hdfs.server.namenode.TestFsLimits |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.server.datanode.TestHSync |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
| | hadoop.hdfs.TestBlockMissingException |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.fs.contract.hdfs.TestHDFSContractSeek |
| | hadoop.hdfs.web.TestWebHDFSAcl |
| | hadoop.hdfs.server.namenode.TestFileContextXAttr |
| | hadoop.hdfs.server.namenode.TestAclConfigFlag |
| |
[jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964327#comment-14964327
]
Ming Ma commented on HDFS-8647:
---
Thanks [~brahmareddy]! It looks great. Sorry, there is one question I didn't
mentioned last time.
It seems the replica-to-be-deleted will be removed from the candidates in
{{BlockPlacementPolicyDefault#chooseReplicasToDelete}}:
{{candidates.remove(cur);}}. It will be removed again later in
{{BlockManager#processChosenExcessReplica}}: {{nonExcess.remove(chosen);}}.
Even though this doesn't cause any issue; it is better to keep the semantics of
{{BlockPlacementPolicyDefault#chooseReplicasToDelete}} to not modify the
candidates. What do you think? If you agree, after you update the patch and the
patch passes jenkins, it is good to go.
In addition, I have used the local repo to evaluate how hard it is to
cherry-pick to branch-2. It turns out I had to spend some time manually
resolving the issues. I have taken care of that and you don't need to provide a
version for branch-2. The patch passed HDFS unit tests in branch-2.
> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -
>
> Key: HDFS-8647
> URL: https://issues.apache.org/jira/browse/HDFS-8647
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch,
> HDFS-8647-003.patch, HDFS-8647-004.patch, HDFS-8647-004.patch,
> HDFS-8647-005.patch, HDFS-8647-006.patch, HDFS-8647-007.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy
> such as upgrade domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as
> useDelHint, blockHasEnoughRacks. That means when we have new block placement
> policy, we need to modify BlockManager to account for the new policy. Ideally
> BlockManager should ask BlockPlacementPolicy object instead. That will allow
> us to provide new BlockPlacementPolicy without changing BlockManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9098) Erasure coding: emulate race conditions among striped streamers in write pipeline
[
https://issues.apache.org/jira/browse/HDFS-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964151#comment-14964151
]
Hadoop QA commented on HDFS-9098:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 21m 0s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 9s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 46s | The applied patch generated 8
new checkstyle issues (total was 114, now 121). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 4m 40s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 0m 29s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 54m 11s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-client |
| Failed build | hadoop-hdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767477/HDFS-9098.wip.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 5068a25 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-client.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13056/console |
This message was automatically generated.
> Erasure coding: emulate race conditions among striped streamers in write
> pipeline
> -
>
> Key: HDFS-9098
> URL: https://issues.apache.org/jira/browse/HDFS-9098
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-9098.wip.patch
>
>
> Apparently the interleaving of events among {{StripedDataStreamer}}'s is very
> tricky to handle. [~walter.k.su] and [~jingzhao] have discussed several race
> conditions under HDFS-9040.
> Let's use FaultInjector to emulate different combinations of interleaved
> events.
> In particular, we should consider inject delays in the following places:
> # {{Streamer#endBlock}}
> # {{Streamer#locateFollowingBlock}}
> # {{Streamer#updateBlockForPipeline}}
> # {{Streamer#updatePipeline}}
> # {{OutputStream#writeChunk}}
> # {{OutputStream#close}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++
[
https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964214#comment-14964214
]
Hadoop QA commented on HDFS-9117:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 55s | Pre-patch HDFS-8707 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:red}-1{color} | javac | 1m 29s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12767493/HDFS-9117.HDFS-8707.001.patch
|
| Optional Tests | javac unit javadoc |
| git revision | HDFS-8707 / ea310d7 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13058/console |
This message was automatically generated.
> Config file reader / options classes for libhdfs++
> --
>
> Key: HDFS-9117
> URL: https://issues.apache.org/jira/browse/HDFS-9117
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: HDFS-8707
>Reporter: Bob Hansen
>Assignee: Bob Hansen
> Attachments: HDFS-9117.HDFS-8707.001.patch
>
>
> For environmental compatability with HDFS installations, libhdfs++ should be
> able to read the configurations from Hadoop XML files and behave in line with
> the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML
> configuration files.
> Similarly, an internal Options architecture for libhdfs++ should be developed
> to efficiently transport the configuration information within the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9250) Add Precondition check to LocatedBlock#addCachedLoc
[
https://issues.apache.org/jira/browse/HDFS-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964237#comment-14964237
]
Hudson commented on HDFS-9250:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8666 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8666/])
HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. (wang: rev
8175c4f6b9fc17ff2e0fc568d798f9b6f2487696)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestLocatedBlock.java
*
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlock.java
> Add Precondition check to LocatedBlock#addCachedLoc
> ---
>
> Key: HDFS-9250
> URL: https://issues.apache.org/jira/browse/HDFS-9250
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS
>Affects Versions: 2.4.0
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0
>
> Attachments: HDFS-9250.001.patch, HDFS-9250.002.patch
>
>
> We may see the following exception:
> {noformat}
> java.lang.ArrayStoreException
> at java.util.ArrayList.toArray(ArrayList.java:389)
> at
> org.apache.hadoop.hdfs.protocol.LocatedBlock.addCachedLoc(LocatedBlock.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.CacheManager.setCachedLocations(CacheManager.java:907)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
> {noformat}
> The cause is that in LocatedBlock.java, when {{addCachedLoc}}:
> - Passed in parameter {{loc}}, which is type {{DatanodeDescriptor}}, is added
> to {{cachedList}}
> - {{cachedList}} was assigned to {{EMPTY_LOCS}}, which is type
> {{DatanodeInfoWithStorage}}.
> Both {{DatanodeDescriptor}} and {{DatanodeInfoWithStorage}} are subclasses of
> {{DatanodeInfo}} but do not inherit from each other, resulting in the
> ArrayStoreException.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9070) Allow fsck display pending replica location information for being-written blocks
[ https://issues.apache.org/jira/browse/HDFS-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] GAO Rui updated HDFS-9070: -- Status: Patch Available (was: Open) > Allow fsck display pending replica location information for being-written > blocks > > > Key: HDFS-9070 > URL: https://issues.apache.org/jira/browse/HDFS-9070 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: GAO Rui >Assignee: GAO Rui > Attachments: HDFS-9070--HDFS-7285.00.patch, > HDFS-9070-HDFS-7285.00.patch, HDFS-9070-HDFS-7285.01.patch, > HDFS-9070-HDFS-7285.02.patch, HDFS-9070-trunk.03.patch, > HDFS-9070-trunk.04.patch, HDFS-9070-trunk.05.patch, HDFS-9070-trunk.06.patch, > HDFS-9070-trunk.07.patch > > > When a EC file is being written, it can be helpful to allow fsck display > datanode information of the being-written EC file block group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9208) Disabling atime may fail clients like distCp
[
https://issues.apache.org/jira/browse/HDFS-9208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-9208:
Hadoop Flags: Reviewed
+1 for patch v2. Thank you, Kihwal.
> Disabling atime may fail clients like distCp
>
>
> Key: HDFS-9208
> URL: https://issues.apache.org/jira/browse/HDFS-9208
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-9208.patch, HDFS-9208.v2.patch
>
>
> When atime is disabled, {{setTimes()}} throws an exception if the passed-in
> atime is not -1. But since atime is not -1, distCp fails when it tries to
> set the mtime and atime.
> There are several options:
> 1) make distCp check for 0 atime and call {{setTimes()}} with -1. I am not
> very enthusiastic about it.
> 2) make NN also accept 0 atime in addition to -1, when the atime support is
> disabled.
> 3) support setting mtime & atime regardless of the atime support. The main
> reason why atime is disabled is to avoid edit logging/syncing during
> {{getBlockLocations()}} read calls. Explicit setting can be allowed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
