[jira] [Updated] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-10224: - Attachment: HDFS-10224-HDFS-9924.009.patch I posted patch v009 identical to v008, just in order to trigger Jenkins job correctly. > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-HDFS-9924.009.patch, HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. In addition, rename is implemented in > this JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255870#comment-15255870 ] ASF GitHub Bot commented on HDFS-10224: --- Github user xiaobingo closed the pull request at: https://github.com/apache/hadoop/pull/93 > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. In addition, rename is implemented in > this JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10316) revisit corrupt replicas count
[
https://issues.apache.org/jira/browse/HDFS-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255850#comment-15255850
]
Lin Yiqun commented on HDFS-10316:
--
Hi, [~walter.k.su], it's a good catch! I looked the code and it's right that
{{countNodes(blk).corruptReplicas()}} only checking for
{{NORMAL}},{{READ_ONLY}} two types in method
{{BlockManager#checkReplicaOnStorage}}, while it does not check for these two
cases in {{BlockManager#findAndMarkBlockAsCorrupt}}. Attach a patch from me,
what do you think of this?
> revisit corrupt replicas count
> --
>
> Key: HDFS-10316
> URL: https://issues.apache.org/jira/browse/HDFS-10316
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Walter Su
> Attachments: HDFS-10316.001.patch
>
>
> A DN has 4 types of storages:
> 1. NORMAL
> 2. READ_ONLY
> 3. FAILED
> 4. (missing/pruned)
> blocksMap.numNodes(blk) counts 1,2,3
> blocksMap.getStorages(blk) counts 1,2,3
> countNodes(blk).corruptReplicas() counts 1,2
> corruptReplicas counts 1,2,3,4. Because findAndMarkBlockAsCorrupt(..)
> supports adding blk to the map even if the storage is not found.
> The inconsistency causes bugs like HDFS-9958.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-10316) revisit corrupt replicas count
[ https://issues.apache.org/jira/browse/HDFS-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lin Yiqun updated HDFS-10316: - Attachment: HDFS-10316.001.patch > revisit corrupt replicas count > -- > > Key: HDFS-10316 > URL: https://issues.apache.org/jira/browse/HDFS-10316 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Walter Su > Attachments: HDFS-10316.001.patch > > > A DN has 4 types of storages: > 1. NORMAL > 2. READ_ONLY > 3. FAILED > 4. (missing/pruned) > blocksMap.numNodes(blk) counts 1,2,3 > blocksMap.getStorages(blk) counts 1,2,3 > countNodes(blk).corruptReplicas() counts 1,2 > corruptReplicas counts 1,2,3,4. Because findAndMarkBlockAsCorrupt(..) > supports adding blk to the map even if the storage is not found. > The inconsistency causes bugs like HDFS-9958. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255847#comment-15255847 ] Xiaobing Zhou commented on HDFS-10224: -- Why does the latest job still run patch v007 (https://github.com/apache/hadoop/pull/93)? > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. In addition, rename is implemented in > this JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255787#comment-15255787 ] Walter Su commented on HDFS-10301: -- bq. BR ids are monotonically increasing. The id values are random intially, if it starts with a large value it could overflow after a long run? If DN restarts, the value randomized again. We should be careful in case NN rejects all following BRs. If BR is splitted into multipe RPCs, there's no interleaving natually because DN get the acked before it sends next RPC. Interleaving only exists if BR is not splitted. I agree bug need to be fixed from inside, It's just eliminating interleaving for good maybe not a bad idea, as it simplifies the problem, and is also a simple workaround for this jira. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Colin Patrick McCabe >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.01.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255772#comment-15255772 ] Walter Su commented on HDFS-10301: -- Thank you for your explanation. I learned a lot. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Colin Patrick McCabe >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.01.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255735#comment-15255735 ] Tsz Wo Nicholas Sze commented on HDFS-10224: TestAsyncDFSRename timed out in the previous build. Please take a look. Thanks. > Implement asynchronous rename for DistributedFileSystem > --- > > Key: HDFS-10224 > URL: https://issues.apache.org/jira/browse/HDFS-10224 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs, hdfs-client >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-10224-HDFS-9924.000.patch, > HDFS-10224-HDFS-9924.001.patch, HDFS-10224-HDFS-9924.002.patch, > HDFS-10224-HDFS-9924.003.patch, HDFS-10224-HDFS-9924.004.patch, > HDFS-10224-HDFS-9924.005.patch, HDFS-10224-HDFS-9924.006.patch, > HDFS-10224-HDFS-9924.007.patch, HDFS-10224-HDFS-9924.008.patch, > HDFS-10224-and-HADOOP-12909.000.patch > > > This is proposed to implement an asynchronous DistributedFileSystem based on > AsyncFileSystem APIs in HADOOP-12910. In addition, rename is implemented in > this JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9543) DiskBalancer : Add Data mover
[ https://issues.apache.org/jira/browse/HDFS-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255717#comment-15255717 ] Anu Engineer commented on HDFS-9543: on minor comments : I will fix both the issues in the next patch. > DiskBalancer : Add Data mover > -- > > Key: HDFS-9543 > URL: https://issues.apache.org/jira/browse/HDFS-9543 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-9543-HDFS-1312.001.patch > > > This patch adds the actual mover logic to the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9543) DiskBalancer : Add Data mover
[
https://issues.apache.org/jira/browse/HDFS-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255715#comment-15255715
]
Anu Engineer commented on HDFS-9543:
[~arpitagarwal] Thanks for the code review comments. I will post a revised
patch soon.
bq. I expect this situation will be seen only if the storage type of a disk
has changed since the planner generated the plan?
We allow disk balancer plans to be edited if needed. This prevents users from
making a mistake. We can avoid it but produces no harm. Please let me know if
you need this removed.
bq. That's a heavyweight lock to hold for the move. I don't think we should
synchronize on the Dataset.
Completely agree. I did this to play fair with DirectoryScanner. This lock is
being held by directoryScanner to prevent changes to blockMap. I wanted to make
sure that diskBalancer and DirectoryScanner play nicely. Please look at
{{DirectoryScanner#scan line 586}}. Please do let me know if this is not needed
in DiskBalancer.
bq. think computeDelay should substract the time taken for the move else the
computed delay will be too conservative. What do you think?
good catch. Thank you, I will fix this in next update.
bq. Precondition.checkState is redundant, subsequent lines will NPE if the
either object is null.
Agree and Precondition is going to throw somethign similar. But it does make
the invariant clear to the next person who is reading code instead of wondering
if I forgot to check a condition.
bq. I did not get this TODO. What does it mean to move blocks across block pools
I will make this comment clearer in the next update. Balancer supports a flag
called -blockpools -- HDFS-8890 seems to have added this. Just leaving a note
that we don't do this yet.
> DiskBalancer : Add Data mover
> --
>
> Key: HDFS-9543
> URL: https://issues.apache.org/jira/browse/HDFS-9543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HDFS-9543-HDFS-1312.001.patch
>
>
> This patch adds the actual mover logic to the datanode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9543) DiskBalancer : Add Data mover
[ https://issues.apache.org/jira/browse/HDFS-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255656#comment-15255656 ] Arpit Agarwal commented on HDFS-9543: - Thanks for adding the mover implementation [~anu]. My comments below, all in DiskBalancer.java: # Line 892: I expect this situation will be seen only if the storage type of a disk has changed since the planner generated the plan? # Lie 943: That's a heavyweight lock to hold for the move. I don't think we should synchronize on the Dataset. # Line 756: I think computeDelay should substract the time taken for the move else the computed delay will be too conservative. What do you think? # Lines 822/823: Precondition.checkState is redundant, subsequent lines will NPE if the either object is null. # Line 889: I did not get this TODO. What does it mean to move blocks across block pools? Minor: # Line 958: Don't need the logging guards since you are using SLF4J and the parameters are cheap to generate. # Should we rename getBlockTolerance to getBlockTolerancePercentage to make it clear its a percentage and not a fraction? I will take a look at the tests later. > DiskBalancer : Add Data mover > -- > > Key: HDFS-9543 > URL: https://issues.apache.org/jira/browse/HDFS-9543 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HDFS-9543-HDFS-1312.001.patch > > > This patch adds the actual mover logic to the datanode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8508) TestJournalNode fails occasionally with bind exception
[
https://issues.apache.org/jira/browse/HDFS-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255635#comment-15255635
]
Hadoop QA commented on HDFS-8508:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 41s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 34s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 53m 34s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| JDK v1.7.0_95 Failed junit tests | hadoop.hdfs.TestDFSShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800418/HDFS-8508.1.patch |
| JIRA Issue | HDFS-8508 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4ab1cae21d24 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-2054) BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()
[ https://issues.apache.org/jira/browse/HDFS-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255588#comment-15255588 ] Guram Savinov commented on HDFS-2054: - I have this IOException in unit-tests with miniDFS cluster. Spark job writes to HDFS a file which is about 100MB. Could you explain the problem for me: miniDFS dataNode closes socket right after it gets last bytes of the file, but block sender tries to transfer full last block which is greater than last data chunk. Am I right? > BlockSender.sendChunk() prints ERROR for connection closures encountered > during transferToFully() > -- > > Key: HDFS-2054 > URL: https://issues.apache.org/jira/browse/HDFS-2054 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 0.22.0, 0.23.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-2054-1.patch, HDFS-2054-2.patch, HDFS-2054.patch, > HDFS-2054.patch, HDFS-2054.patch > > > The addition of ERROR was part of HDFS-1527. In environments where clients > tear down FSInputStream/connection before reaching the end of stream, this > error message often pops up. Since these are not really errors and especially > not the fault of data node, the message should be toned down at least. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8508) TestJournalNode fails occasionally with bind exception
[
https://issues.apache.org/jira/browse/HDFS-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takashi Ohnishi updated HDFS-8508:
--
Status: Patch Available (was: Open)
> TestJournalNode fails occasionally with bind exception
> --
>
> Key: HDFS-8508
> URL: https://issues.apache.org/jira/browse/HDFS-8508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Takashi Ohnishi
> Attachments: HDFS-8508.1.patch
>
>
> TestJournalNode uses the default port for {{dfs.journalnode.http-address}} so
> it fails occasionally when running with {{-Pparallel-tests}}.
> The same issue likely exists with other tests. Tests should generate a random
> port number with retry to reduce the chances of a collision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-8508) TestJournalNode fails occasionally with bind exception
[
https://issues.apache.org/jira/browse/HDFS-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takashi Ohnishi reassigned HDFS-8508:
-
Assignee: Takashi Ohnishi
> TestJournalNode fails occasionally with bind exception
> --
>
> Key: HDFS-8508
> URL: https://issues.apache.org/jira/browse/HDFS-8508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
>Assignee: Takashi Ohnishi
> Attachments: HDFS-8508.1.patch
>
>
> TestJournalNode uses the default port for {{dfs.journalnode.http-address}} so
> it fails occasionally when running with {{-Pparallel-tests}}.
> The same issue likely exists with other tests. Tests should generate a random
> port number with retry to reduce the chances of a collision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8508) TestJournalNode fails occasionally with bind exception
[
https://issues.apache.org/jira/browse/HDFS-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takashi Ohnishi updated HDFS-8508:
--
Attachment: HDFS-8508.1.patch
> TestJournalNode fails occasionally with bind exception
> --
>
> Key: HDFS-8508
> URL: https://issues.apache.org/jira/browse/HDFS-8508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
> Attachments: HDFS-8508.1.patch
>
>
> TestJournalNode uses the default port for {{dfs.journalnode.http-address}} so
> it fails occasionally when running with {{-Pparallel-tests}}.
> The same issue likely exists with other tests. Tests should generate a random
> port number with retry to reduce the chances of a collision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8508) TestJournalNode fails occasionally with bind exception
[
https://issues.apache.org/jira/browse/HDFS-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255586#comment-15255586
]
Takashi Ohnishi commented on HDFS-8508:
---
Hi.
I have created a patch for this.
The other tests in org.apache.hadoop.hdfs.qjournal.server do not have the same
issues because they seem to have already used random port number.
> TestJournalNode fails occasionally with bind exception
> --
>
> Key: HDFS-8508
> URL: https://issues.apache.org/jira/browse/HDFS-8508
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.7.0
>Reporter: Arpit Agarwal
> Attachments: HDFS-8508.1.patch
>
>
> TestJournalNode uses the default port for {{dfs.journalnode.http-address}} so
> it fails occasionally when running with {{-Pparallel-tests}}.
> The same issue likely exists with other tests. Tests should generate a random
> port number with retry to reduce the chances of a collision.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8814) BlockSender.sendChunks() exception
[ https://issues.apache.org/jira/browse/HDFS-8814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1529#comment-1529 ] Guram Savinov commented on HDFS-8814: - I have the same IOException in unit-tests with miniDFS cluster > BlockSender.sendChunks() exception > -- > > Key: HDFS-8814 > URL: https://issues.apache.org/jira/browse/HDFS-8814 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0, 2.7.1 > Environment: OS: CentOS Linux release 7.1.1503 (Core) > Kernel: 3.10.0-229.1.2.el7.x86_64 >Reporter: Marius > > Hi > I was running some streaming jobs with avro files from my hadoop cluster. > They performed poorly so i checked the logs of my datanodes and found this: > http://pastebin.com/DXKJJ55z > The cluster is running on CentOS machines: > CentOS Linux release 7.1.1503 (Core) > This is the Kernel: > 3.10.0-229.1.2.el7.x86_64 > No one on the userlist replied and i could not find anything helpful on the > internet despite disk failure which is unlikely to cause this because here > are several machines and its not very likely that all of their disks fail at > the same time. > This error is not reported on the console when running a job and the error > occurs from time to time and then dissapears and comes back again. > The block size of the cluster is the default value. > This is my command: > hadoop jar hadoop-streaming-2.7.1.jar -files mapper.py,reducer.py,avro-1. > 7.7.jar,avro-mapred-1.7.7-hadoop2.jar -D mapreduce.job.reduces=15 -libjars > avro-1.7.7.jar,avro-mapred-1.7.7-hadoop2.jar -input /Y/Y1.avro -output > /htest/output -mapper mapper.py -reducer reducer.py -inputformat > org.apache.avro.mapred.AvroAsTextInputFormat > Marius -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-10224) Implement asynchronous rename for DistributedFileSystem
[
https://issues.apache.org/jira/browse/HDFS-10224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1528#comment-1528
]
Hadoop QA commented on HDFS-10224:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 56s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 24s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 29s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 21s
{color} | {color:red} root: patch generated 1 new + 272 unchanged - 1 fixed =
273 total (was 273) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 29s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 4s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 2s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 14s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 40s
{color} | {color:red} hadoop-hdfs in the patch
