[jira] [Commented] (HDFS-14744) RBF: Non secured routers should not log in error mode when UGI is default.
[
https://issues.apache.org/jira/browse/HDFS-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910184#comment-16910184
]
Hadoop QA commented on HDFS-14744:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 34s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterWithSecureStartup |
| | hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14744 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977840/HDFS-14744.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6f6fb46ae958 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c765584 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27556/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27556/testReport/ |
| Max. process+thread count | 1594 (vs. ulimit of 5500) |
| modules | C:
[jira] [Commented] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910178#comment-16910178
]
Surendra Singh Lilhore commented on HDFS-14746:
---
LGTM, +1
Pending jenkins.
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin su updated HDFS-14746:
Status: Patch Available (was: Open)
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14746) Trivial test code update after HDFS-14687
[
https://issues.apache.org/jira/browse/HDFS-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin su updated HDFS-14746:
Attachment: HDFS-14746.001.patch
> Trivial test code update after HDFS-14687
> -
>
> Key: HDFS-14746
> URL: https://issues.apache.org/jira/browse/HDFS-14746
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ec
>Reporter: Wei-Chiu Chuang
>Assignee: kevin su
>Priority: Trivial
> Attachments: HDFS-14746.001.patch
>
>
> Instead of getting erasure coding policy instance by id, it should use a
> constant value.
> Change
> {code}
> ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
> .get(3);
> {code}
> to
> {code}
> ErasureCodingPolicy ecPolicy =
> SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14396) Failed to load image from FSImageFile when downgrade from 3.x to 2.x
[
https://issues.apache.org/jira/browse/HDFS-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910174#comment-16910174
]
Akira Ajisaka commented on HDFS-14396:
--
LGTM, +1
> Failed to load image from FSImageFile when downgrade from 3.x to 2.x
>
>
> Key: HDFS-14396
> URL: https://issues.apache.org/jira/browse/HDFS-14396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: HDFS-14396.001.patch, HDFS-14396.002.patch
>
>
> After fixing HDFS-13596, try to downgrade from 3.x to 2.x. But namenode can't
> start because exception occurs. The message follows
> {code:java}
> 2019-01-23 17:22:18,730 ERROR org.apache.hadoop.hdfs.server.namenode.FSImage:
> Failed to load image from
> FSImageFile(file=/data1/hadoopdata/hadoop-namenode/current/fsimage_0025310,
> cpktTxId=00
> 25310)
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:243)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:885)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:869)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:742)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:673)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:672)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:839)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:823)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1517)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1583)
> 2019-01-23 17:22:19,023 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Failed to load FSImage file, see error(s) above for more
> info.
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:688)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:290)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:998)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:612)
> {code}
> This issue occurs because 3.x namenode saves image with EC fields during
> upgrade
> Try to fix it
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13596) NN restart fails after RollingUpgrade from 2.x to 3.x
[
https://issues.apache.org/jira/browse/HDFS-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910173#comment-16910173
]
Akira Ajisaka commented on HDFS-13596:
--
LGTM, +1. I'll commit this tomorrow if there are no objections.
> NN restart fails after RollingUpgrade from 2.x to 3.x
> -
>
> Key: HDFS-13596
> URL: https://issues.apache.org/jira/browse/HDFS-13596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Hanisha Koneru
>Assignee: Fei Hui
>Priority: Blocker
> Attachments: HDFS-13596.001.patch, HDFS-13596.002.patch,
> HDFS-13596.003.patch, HDFS-13596.004.patch, HDFS-13596.005.patch,
> HDFS-13596.006.patch, HDFS-13596.007.patch, HDFS-13596.008.patch,
> HDFS-13596.009.patch
>
>
> After rollingUpgrade NN from 2.x and 3.x, if the NN is restarted, it fails
> while replaying edit logs.
> * After NN is started with rollingUpgrade, the layoutVersion written to
> editLogs (before finalizing the upgrade) is the pre-upgrade layout version
> (so as to support downgrade).
> * When writing transactions to log, NN writes as per the current layout
> version. In 3.x, erasureCoding bits are added to the editLog transactions.
> * So any edit log written after the upgrade and before finalizing the
> upgrade will have the old layout version but the new format of transactions.
> * When NN is restarted and the edit logs are replayed, the NN reads the old
> layout version from the editLog file. When parsing the transactions, it
> assumes that the transactions are also from the previous layout and hence
> skips parsing the erasureCoding bits.
> * This cascades into reading the wrong set of bits for other fields and
> leads to NN shutting down.
> Sample error output:
> {code:java}
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:74)
> at org.apache.hadoop.ipc.RetryCache$CacheEntry.(RetryCache.java:86)
> at
> org.apache.hadoop.ipc.RetryCache$CacheEntryWithPayload.(RetryCache.java:163)
> at
> org.apache.hadoop.ipc.RetryCache.addCacheEntryWithPayload(RetryCache.java:322)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.addCacheEntryWithPayload(FSNamesystem.java:960)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:937)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:910)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
> 2018-05-17 19:10:06,522 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: java.lang.IllegalStateException: Cannot skip to less
> than the current value (=16389), where newValue=16388
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.resetLastInodeId(FSDirectory.java:1945)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:298)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:158)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:888)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:745)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:323)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1086)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632)
> at
>
[jira] [Updated] (HDFS-14646) Standby NameNode should not upload fsimage to an inappropriate NameNode.
[
https://issues.apache.org/jira/browse/HDFS-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xudong Cao updated HDFS-14646:
--
Status: Open (was: Patch Available)
> Standby NameNode should not upload fsimage to an inappropriate NameNode.
>
>
> Key: HDFS-14646
> URL: https://issues.apache.org/jira/browse/HDFS-14646
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Major
> Attachments: HDFS-14646.000.patch, HDFS-14646.001.patch,
> HDFS-14646.002.patch
>
>
> *Problem Description:*
> In the multi-NameNode scenario, when a SNN uploads a FsImage, it will put
> the image to all other NNs (whether the peer NN is an ANN or not), and even
> if the peer NN immediately replies an error (such as
> TransferResult.NOT_ACTIVE_NAMENODE_FAILURE, TransferResult
> .OLD_TRANSACTION_ID_FAILURE, etc.), the local SNN will not terminate the put
> process immediately, but will put the FsImage completely to the peer NN, and
> will not read the peer NN's reply until the put is completed.
> Depending on the version of Jetty, this behavior can lead to different
> consequences, I tested it under 2.7.2 and trunk version.
> *1.In Hadoop 2.7.2 (with Jetty 6.1.26)*
> After peer NN called HttpServletResponse.sendError(), the underlying TCP
> connection will still be established, and the data SNN sent will be read by
> Jetty framework itself in the peer NN side, so the SNN will insignificantly
> send the FsImage to the peer NN continuously, causing a waste of time and
> bandwidth. In a relatively large HDFS cluster, the size of FsImage can often
> reach about 30GB, This is indeed a big waste.
> *2.In trunk version (with Jetty 9.3.27)*
> After peer NN called HttpServletResponse.sendError(), the underlying TCP
> connection will be auto closed, and then SNN will directly get an "Error
> writing request body to server" exception, as below, note this test needs a
> relatively big FSImage (e.g. 10MB level):
> {code:java}
> 2019-08-17 03:59:25,413 INFO namenode.TransferFsImage: Sending fileName:
> /tmp/hadoop-root/dfs/name/current/fsimage_3364240, fileSize:
> 9864721. Sent total: 524288 bytes. Size of last segment intended to send:
> 4096 bytes.
> java.io.IOException: Error writing request body to server
> at
> sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3587)
> at
> sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3570)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.copyFileToStream(TransferFsImage.java:396)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.writeFileToPutRequest(TransferFsImage.java:340)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.uploadImage(TransferFsImage.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.uploadImageFromStorage(TransferFsImage.java:249)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$1.call(StandbyCheckpointer.java:277)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer$1.call(StandbyCheckpointer.java:272)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-08-17 03:59:25,422 INFO namenode.TransferFsImage: Sending fileName:
> /tmp/hadoop-root/dfs/name/current/fsimage_3364240, fileSize:
> 9864721. Sent total: 851968 bytes. Size of last segment intended to send:
> 4096 bytes.
> java.io.IOException: Error writing request body to server
> at
> sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3587)
> at
> sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3570)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.copyFileToStream(TransferFsImage.java:396)
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.writeFileToPutRequest(TransferFsImage.java:340)
> {code}
>
> *Solution:*
> A standby NameNode should not upload fsimage to an inappropriate NameNode,
> when he plans to put a FsImage to the peer NN, he need to check whether he
> really need to put it at this time.
> In detail, local SNN should establish an HTTP connection with the peer NN,
> send the put request, and then immediately read the response (this is the key
> point). If the peer NN does not reply an HTTP_OK, it means the local SNN
> should not put image at this
[jira] [Commented] (HDFS-14744) RBF: Non secured routers should not log in error mode when UGI is default.
[
https://issues.apache.org/jira/browse/HDFS-14744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910164#comment-16910164
]
Ayush Saxena commented on HDFS-14744:
-
Seems fair enough to suppress.Guess Jenkins had some problem, Need to re
trigger.
> RBF: Non secured routers should not log in error mode when UGI is default.
> --
>
> Key: HDFS-14744
> URL: https://issues.apache.org/jira/browse/HDFS-14744
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14744.001.patch
>
>
> RouterClientProtocol#getMountPointStatus logs error when groups are not found
> for default web user dr.who. The line should be logged in "error" mode for
> secured cluster, for unsecured clusters, we may want to just specify "debug"
> or else logs are filled up with this non-critical line
> {{ERROR org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer:
> Cannot get the remote user: There is no primary group for UGI dr.who
> (auth:SIMPLE)}}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910163#comment-16910163
]
Hadoop QA commented on HDFS-14476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
8s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}128m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.namenode.TestCheckpoint |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancer |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.server.namenode.TestStripedINodeFile |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:63396beab41 |
| JIRA Issue | HDFS-14476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977905/HDFS-14476.branch-3.2.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c991b824367d 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / e89413d |
| maven | version: Apache Maven 3.3.9 |
|
[jira] [Commented] (HDFS-14741) RBF: RecoverLease should be return false when the file is open in multiple destination
[
https://issues.apache.org/jira/browse/HDFS-14741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910161#comment-16910161
]
Ayush Saxena commented on HDFS-14741:
-
Thanx [~xuzq_zander] for the patch, fix LGTM.
I guess for the test you can use the existing mount entries only, add a file
and delete it in the finnally, there are two mount entries / pointing to ns0-/
and ns1-/ and /same pointing to ns0-/ and /target-ns0-/
{code:java}
247 RouterContext rc = getRouterContext();
248 DistributedFileSystem routerFs =
249 (DistributedFileSystem) rc.getFileSystem();
{code}
Can use {{DistributedFileSystem routerFs = (DistributedFileSystem)
getRouterFileSystem();}}
For FsDataOutputStream we can use try with resources rather than handling this
much.
> RBF: RecoverLease should be return false when the file is open in multiple
> destination
> --
>
> Key: HDFS-14741
> URL: https://issues.apache.org/jira/browse/HDFS-14741
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: xuzq
>Assignee: xuzq
>Priority: Major
> Attachments: HDFS-14741-trunk-001.patch
>
>
> RecoverLease should be return false when the file is open or be writing in
> multiple destinations.
> Liks this:
> Mount point has multiple destination(ns0 and ns1).
> And the file is in ns0 but it is be writing, ns1 doesn't has this file.
> In this case *recoverLease* should return false instead of throw
> FileNotFoundException.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10606) TrashPolicyDefault supports time of auto clean up can configured
[ https://issues.apache.org/jira/browse/HDFS-10606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910157#comment-16910157 ] He Xiaoqiao commented on HDFS-10606: TrashPolicyDefault clean up trash at static time 00:00 UTC, and we have no other ways to tune it to other times. This JIRA aim to offer one configuration to tune auto-clean start time. > TrashPolicyDefault supports time of auto clean up can configured > > > Key: HDFS-10606 > URL: https://issues.apache.org/jira/browse/HDFS-10606 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-10606-branch-2.7.001.patch, HDFS-10606.001.patch, > HDFS-10606.002.patch > > > TrashPolicyDefault clean up Trash based on > [UTC|http://www.worldtimeserver.com/current_time_in_UTC.aspx] currently and > the time of cleaning up is 00:00 UTC. when there are large amount of trash > data should be auto-clean, it will block NN for a long time since Global > Lock, In the most serious situations it may lead some cron job submit > failure. if add configuration about time of cleaning up, it will avoid impact > on this cron jobs at that default time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Xiaoqiao updated HDFS-14109:
---
Resolution: Not A Problem
Status: Resolved (was: Patch Available)
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910155#comment-16910155
]
He Xiaoqiao commented on HDFS-14109:
my first thought is that add `nsid` for namenode audit log to support
federation more friendly and make a distinction between multiply namespaces if
we collect all namenode audit log together, however it seems that this is not
the common requirement, so I will cancel this JIRA and set to `not a problem`.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296996 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 03:11 Start Date: 19/Aug/19 03:11 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522395453 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | 0 | yamllint | 0 | yamllint was not available. | | 0 | shelldocs | 0 | Shelldocs was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 621 | trunk passed | | +1 | compile | 365 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 731 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 155 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 546 | the patch passed | | +1 | compile | 378 | the patch passed | | +1 | javac | 378 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | shellcheck | 0 | There were no new shellcheck issues. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 659 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 163 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 300 | hadoop-hdds in the patch passed. | | -1 | unit | 2080 | hadoop-ozone in the patch failed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 6304 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1304 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient yamllint shellcheck shelldocs | | uname | Linux a38bbfa4497e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b8db5b9 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/testReport/ | | Max. process+thread count | 5411 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/dist U: hadoop-ozone/dist | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1304/6/console | | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296996) Time Spent: 3.5h (was: 3h 20m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3.5h > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira,
[jira] [Resolved] (HDDS-1894) Support listPipelines by filters in scmcli
[
https://issues.apache.org/jira/browse/HDDS-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Cheng resolved HDDS-1894.
Resolution: Fixed
> Support listPipelines by filters in scmcli
> --
>
> Key: HDDS-1894
> URL: https://issues.apache.org/jira/browse/HDDS-1894
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Li Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Today scmcli has a subcmd that allow list all pipelines. This ticket is
> opened to filter the results by switches, e.g., filter by Factor: THREE and
> State: OPEN. This will be useful for trouble shooting in large cluster.
>
> {code}
> bin/ozone scmcli listPipelines
> Pipeline[ Id: a8d1b0c9-e1d4-49ea-8746-3f61dfb5ee3f, Nodes:
> cce44fde-bc8d-4063-97b3-6f557af756e1\{ip: 10.17.112.65, host:
> ia0230.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:ONE, State:OPEN]
> Pipeline[ Id: c9c453d1-d74c-4414-b87f-1d3585d78a7c, Nodes:
> 0b7b0b93-8323-4b82-8cc0-a9a5c10ab827\{ip: 10.17.112.29, host:
> ia0138.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}c756a0e0-5a1b-4d03-ba5b-cafbcabac877\{ip: 10.17.112.27, host:
> ia0134.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}bee45bd7-1ee6-4726-b3d1-81476dc1eb49\{ip: 10.17.112.28, host:
> ia0136.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:THREE, State:OPEN]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-1894) Support listPipelines by filters in scmcli
[
https://issues.apache.org/jira/browse/HDDS-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Cheng reopened HDDS-1894:
Change Fix version
> Support listPipelines by filters in scmcli
> --
>
> Key: HDDS-1894
> URL: https://issues.apache.org/jira/browse/HDDS-1894
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Li Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Today scmcli has a subcmd that allow list all pipelines. This ticket is
> opened to filter the results by switches, e.g., filter by Factor: THREE and
> State: OPEN. This will be useful for trouble shooting in large cluster.
>
> {code}
> bin/ozone scmcli listPipelines
> Pipeline[ Id: a8d1b0c9-e1d4-49ea-8746-3f61dfb5ee3f, Nodes:
> cce44fde-bc8d-4063-97b3-6f557af756e1\{ip: 10.17.112.65, host:
> ia0230.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:ONE, State:OPEN]
> Pipeline[ Id: c9c453d1-d74c-4414-b87f-1d3585d78a7c, Nodes:
> 0b7b0b93-8323-4b82-8cc0-a9a5c10ab827\{ip: 10.17.112.29, host:
> ia0138.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}c756a0e0-5a1b-4d03-ba5b-cafbcabac877\{ip: 10.17.112.27, host:
> ia0134.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}bee45bd7-1ee6-4726-b3d1-81476dc1eb49\{ip: 10.17.112.28, host:
> ia0136.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:THREE, State:OPEN]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1894) Support listPipelines by filters in scmcli
[
https://issues.apache.org/jira/browse/HDDS-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Cheng updated HDDS-1894:
---
Fix Version/s: (was: 0.5.0)
0.4.1
> Support listPipelines by filters in scmcli
> --
>
> Key: HDDS-1894
> URL: https://issues.apache.org/jira/browse/HDDS-1894
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Li Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Today scmcli has a subcmd that allow list all pipelines. This ticket is
> opened to filter the results by switches, e.g., filter by Factor: THREE and
> State: OPEN. This will be useful for trouble shooting in large cluster.
>
> {code}
> bin/ozone scmcli listPipelines
> Pipeline[ Id: a8d1b0c9-e1d4-49ea-8746-3f61dfb5ee3f, Nodes:
> cce44fde-bc8d-4063-97b3-6f557af756e1\{ip: 10.17.112.65, host:
> ia0230.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:ONE, State:OPEN]
> Pipeline[ Id: c9c453d1-d74c-4414-b87f-1d3585d78a7c, Nodes:
> 0b7b0b93-8323-4b82-8cc0-a9a5c10ab827\{ip: 10.17.112.29, host:
> ia0138.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}c756a0e0-5a1b-4d03-ba5b-cafbcabac877\{ip: 10.17.112.27, host:
> ia0134.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}bee45bd7-1ee6-4726-b3d1-81476dc1eb49\{ip: 10.17.112.28, host:
> ia0136.halxg.cloudera.com, networkLocation: /default-rack, certSerialId:
> null}, Type:RATIS, Factor:THREE, State:OPEN]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13101) Yet another fsimage corruption related to snapshot
[ https://issues.apache.org/jira/browse/HDFS-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13101: --- Resolution: Fixed Fix Version/s: 2.9.3 2.8.6 2.10.0 Status: Resolved (was: Patch Available) Pushed to branch-2 branch-2.9. There is a trivial conflict in branch-2.8. I resolved the conflict and pushed to branch-2.8 too. [^HDFS-13101.branch-2.8.patch] for reference. Thanks all!! This is amazing work. > Yet another fsimage corruption related to snapshot > -- > > Key: HDFS-13101 > URL: https://issues.apache.org/jira/browse/HDFS-13101 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Yongjun Zhang >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 2.10.0, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-13101.001.patch, HDFS-13101.002.patch, > HDFS-13101.003.patch, HDFS-13101.004.patch, HDFS-13101.branch-2.001.patch, > HDFS-13101.branch-2.8.patch, HDFS-13101.corruption_repro.patch, > HDFS-13101.corruption_repro_simplified.patch > > > Lately we saw case similar to HDFS-9406, even though HDFS-9406 fix is > present, so it's likely another case not covered by the fix. We are currently > trying to collect good fsimage + editlogs to replay to reproduce it and > investigate. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13101) Yet another fsimage corruption related to snapshot
[ https://issues.apache.org/jira/browse/HDFS-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-13101: --- Attachment: HDFS-13101.branch-2.8.patch > Yet another fsimage corruption related to snapshot > -- > > Key: HDFS-13101 > URL: https://issues.apache.org/jira/browse/HDFS-13101 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Yongjun Zhang >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-13101.001.patch, HDFS-13101.002.patch, > HDFS-13101.003.patch, HDFS-13101.004.patch, HDFS-13101.branch-2.001.patch, > HDFS-13101.branch-2.8.patch, HDFS-13101.corruption_repro.patch, > HDFS-13101.corruption_repro_simplified.patch > > > Lately we saw case similar to HDFS-9406, even though HDFS-9406 fix is > present, so it's likely another case not covered by the fix. We are currently > trying to collect good fsimage + editlogs to replay to reproduce it and > investigate. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910126#comment-16910126
]
Hadoop QA commented on HDFS-14109:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-14109 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14109 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949846/HDFS-14109.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27555/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14109) Improve hdfs auditlog format and support federation friendly
[
https://issues.apache.org/jira/browse/HDFS-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910123#comment-16910123
]
Wei-Chiu Chuang commented on HDFS-14109:
Is this still active? Does this duplicate HDFS-14625 in any ways? HDFS-14625 is
meant to support RBF.
> Improve hdfs auditlog format and support federation friendly
>
>
> Key: HDFS-14109
> URL: https://issues.apache.org/jira/browse/HDFS-14109
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14109.patch
>
>
> The following auditlog format does not well meet requirement for federation
> arch currently. Since some case we need to aggregate all namespace audit log,
> if there are some common path request(e.g. /tmp, /user/ etc. some path may
> not appear in mountTable, but the path is very real), we will have no idea to
> split them that which namespace it request to. So I propose add column
> {{nsid}} to support federation more friendly.
> {quote}2018-11-27 13:20:30,028 INFO FSNamesystem.audit: allowed=true
> ugi=hdfs/hostn...@realm.com (auth:KERBEROS) ip=/10.1.1.2 cmd=getfileinfo
> src=/path dst=null perm=null proto=rpc clientName=null
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-1971:
-
Fix Version/s: 0.4.1
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1, 0.5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910111#comment-16910111
]
Hudson commented on HDFS-14687:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17145 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17145/])
HDFS-14687. Standby Namenode never come out of safemode when EC files (weichiu:
rev b8db5b9a9812023754ed1b3e5b428e161f0add50)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingDataNodeMessages.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingDataNodeMessages.java
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910112#comment-16910112
]
Hudson commented on HDDS-1971:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17145 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17145/])
HDDS-1971. Update document for HDDS-1891: Ozone fs shell command should
(bharat: rev 12c7084be3b03be81cdb688c911798d52dcfc160)
* (edit) hadoop-hdds/docs/content/interface/OzoneFS.md
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1891) Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910113#comment-16910113
]
Hudson commented on HDDS-1891:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17145 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17145/])
HDDS-1971. Update document for HDDS-1891: Ozone fs shell command should
(bharat: rev 12c7084be3b03be81cdb688c911798d52dcfc160)
* (edit) hadoop-hdds/docs/content/interface/OzoneFS.md
> Ozone fs shell command should work with default port when port number is not
> specified
> --
>
> Key: HDDS-1891
> URL: https://issues.apache.org/jira/browse/HDDS-1891
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1, 0.5.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> {code:bash|title=Without port number -> Error}
> $ ozone fs -ls o3fs://bucket.volume.localhost/
> -ls: Ozone file system url should be either one of the two forms:
> o3fs://bucket.volume/key OR o3fs://bucket.volume.om-host.example.com:5678/key
> ...
> {code}
> {code:bash|title=With port number -> Success}
> $ ozone fs -ls o3fs://bucket.volume.localhost:9862/
> Found 1 items
> -rw-rw-rw- 1 hadoop hadoop 1485 2019-08-01 21:14
> o3fs://bucket.volume.localhost:9862/README.txt
> {code}
> We expect the first command to attempt port 9862 by default.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham resolved HDDS-1971.
--
Resolution: Fixed
Fix Version/s: 0.5.0
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?focusedWorklogId=296981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296981
]
ASF GitHub Bot logged work on HDDS-1971:
Author: ASF GitHub Bot
Created on: 19/Aug/19 01:29
Start Date: 19/Aug/19 01:29
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1306:
HDDS-1971. Update document for HDDS-1891: Ozone fs shell command should work
with default port when port number is not specified
URL: https://github.com/apache/hadoop/pull/1306
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296981)
Time Spent: 1h 50m (was: 1h 40m)
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?focusedWorklogId=296980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296980
]
ASF GitHub Bot logged work on HDDS-1971:
Author: ASF GitHub Bot
Created on: 19/Aug/19 01:29
Start Date: 19/Aug/19 01:29
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #1306: HDDS-1971.
Update document for HDDS-1891: Ozone fs shell command should work with default
port when port number is not specified
URL: https://github.com/apache/hadoop/pull/1306#issuecomment-522379992
Thank You @smengcl for the contribution.
I will commit this to the trunk.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296980)
Time Spent: 1h 40m (was: 1.5h)
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14746) Trivial test code update after HDFS-14687
Wei-Chiu Chuang created HDFS-14746:
--
Summary: Trivial test code update after HDFS-14687
Key: HDFS-14746
URL: https://issues.apache.org/jira/browse/HDFS-14746
Project: Hadoop HDFS
Issue Type: Improvement
Components: ec
Reporter: Wei-Chiu Chuang
Instead of getting erasure coding policy instance by id, it should use a
constant value.
Change
{code}
ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getPolicies()
.get(3);
{code}
to
{code}
ErasureCodingPolicy ecPolicy =
SystemErasureCodingPolicies.getByID(XOR_2_1_POLICY_ID);
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296977 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 01:25 Start Date: 19/Aug/19 01:25 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#discussion_r315014124 ## File path: hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + s3g: + image: haproxy:latest + volumes: + - ../..:/opt/hadoop + - ./haproxy-conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg + ports: + - 8081:9878 Review comment: This is when aws cli tests are run from outside docker (it uses 8081 port), but we execute these tests inside docker network. So, I think this should not be a problem. But I will give a try, as I was not able to figure out why it is failing in Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296977) Time Spent: 3h 10m (was: 3h) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296978 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 01:25 Start Date: 19/Aug/19 01:25 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#issuecomment-522379433 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296978) Time Spent: 3h 20m (was: 3h 10m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296976 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 19/Aug/19 01:24 Start Date: 19/Aug/19 01:24 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#discussion_r315014124 ## File path: hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + s3g: + image: haproxy:latest + volumes: + - ../..:/opt/hadoop + - ./haproxy-conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg + ports: + - 8081:9878 Review comment: This is when aws cli tests are run from outside docker, but we execute these tests inside docker network. So, I think this should not be a problem. But I will give a try, as I was not able to figure out why it is failing in Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296976) Time Spent: 3h (was: 2h 50m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14687:
---
Resolution: Fixed
Fix Version/s: 3.1.3
3.2.1
3.3.0
3.0.4
Status: Resolved (was: Patch Available)
Pushed the patch 004 to trunk, branch-3.2 branch-3.1 and branch-3.0.
Thanks [~surendrasingh].
Actually, I just found a nit in the test after committing the patch. Will file
a jira.
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Fix For: 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910105#comment-16910105
]
Wei-Chiu Chuang commented on HDFS-14687:
+1
Thanks. On my machine this test took 1.5 minutes. Longer than your number but
much better than before.
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910103#comment-16910103
]
Wei-Chiu Chuang commented on HDFS-14476:
Pushed to trunk, branch-2, branch-2.9, branch-2.8.
There are conflicts in branch-3.2 and below. Updated the trunk patch and submit
for precommit build.
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3
>
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, HDFS-14476.branch-3.2.001.patch,
> datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14476:
---
Attachment: HDFS-14476.branch-3.2.001.patch
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3
>
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, HDFS-14476.branch-3.2.001.patch,
> datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14476:
---
Fix Version/s: 2.9.3
2.8.6
3.3.0
2.10.0
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Fix For: 2.10.0, 3.3.0, 2.8.6, 2.9.3
>
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910100#comment-16910100
]
Hadoop QA commented on HDFS-14476:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-14476 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14476 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977902/HDFS-14476.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27553/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910099#comment-16910099
]
Hudson commented on HDFS-14476:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17144 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17144/])
HDFS-14476. lock too long when fix inconsistent blocks between disk and
(weichiu: rev b58a35f374a9a750fddc2fc92e7f7a7ae8a4d3a4)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910098#comment-16910098
]
Wei-Chiu Chuang commented on HDFS-14476:
+1
[~seanlook] looks like you're the first time contributor to Hadoop. Welcome.
Note that there is a checkstyle warning. I find this Hadoop code formatter
works for me most of time:
https://github.com/cloudera/blog-eclipse/blob/master/hadoop-format.xml For your
reference in the future.
I've taken care of the checkstyle warning, and committed the code for you. The
updated patch is [^HDFS-14476.002.patch] for your reference.
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14476) lock too long when fix inconsistent blocks between disk and in-memory
[
https://issues.apache.org/jira/browse/HDFS-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14476:
---
Attachment: HDFS-14476.002.patch
> lock too long when fix inconsistent blocks between disk and in-memory
> -
>
> Key: HDFS-14476
> URL: https://issues.apache.org/jira/browse/HDFS-14476
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.6.0, 2.7.0, 3.0.3
>Reporter: Sean Chow
>Assignee: Sean Chow
>Priority: Major
> Attachments: HDFS-14476-branch-2.01.patch, HDFS-14476.00.patch,
> HDFS-14476.002.patch, HDFS-14476.01.patch, datanode-with-patch-14476.png
>
>
> When directoryScanner have the results of differences between disk and
> in-memory blocks. it will try to run {{checkAndUpdate}} to fix it. However
> {{FsDatasetImpl.checkAndUpdate}} is a synchronized call
> As I have about 6millions blocks for every datanodes and every 6hours' scan
> will have about 25000 abnormal blocks to fix. That leads to a long lock
> holding FsDatasetImpl object.
> let's assume every block need 10ms to fix(because of latency of SAS disk),
> that will cost 250 seconds to finish. That means all reads and writes will be
> blocked for 3mins for that datanode.
>
> {code:java}
> 2019-05-06 08:06:51,704 INFO
> org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool
> BP-1644920766-10.223.143.220-1450099987967 Total blocks: 6850197, missing
> metadata files:23574, missing block files:23574, missing blocks in
> memory:47625, mismatched blocks:0
> ...
> 2019-05-06 08:16:41,625 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
> Took 588402ms to process 1 commands from NN
> {code}
> Take long time to process command from nn because threads are blocked. And
> namenode will see long lastContact time for this datanode.
> Maybe this affect all hdfs versions.
> *how to fix:*
> just like process invalidate command from namenode with 1000 batch size, fix
> these abnormal block should be handled with batch too and sleep 2 seconds
> between the batch to allow normal reading/writing blocks.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14567) If kms-acls is failed to load, and it will never be reload
[
https://issues.apache.org/jira/browse/HDFS-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910096#comment-16910096
]
Wei-Chiu Chuang commented on HDFS-14567:
Reviewed the test code.
I am okay with the test in general.
* It would be great to not use Thread.sleep() to control thread order and use
FakeTimer instead. If you really don't want to use FakeTimer because it is
potentially a bigger change, please use a larger sleep time. Say 1 second.
Because the Jenkins is very busy and being too precise does no good to you.
* You don't need to instantiate MiniKMS at all. You just test the behavior of
KMSAcl. Remove it reduces the test run time significantly from 7+ seconds to
less than 1 second.
* Please don't swallow exceptions (catch Exception and do nothing). By swallow
exceptions you don't know if the test passes or fails. Actually it looks to me
the test if flaky after stopping swapping exceptions.
> If kms-acls is failed to load, and it will never be reload
> ---
>
> Key: HDFS-14567
> URL: https://issues.apache.org/jira/browse/HDFS-14567
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: HDFS-14567.patch
>
>
> Scenario : through one automation tool , we are generating kms-acls , though
> the generation of kms-acls is not completed , the system will detect a
> modification of kms-alcs and it will try to load
> Before getting the configuration we are modifiying last reload time , code
> shown below
> {code:java}
> private Configuration loadACLsFromFile() {
> LOG.debug("Loading ACLs file");
> lastReload = System.currentTimeMillis();
> Configuration conf = KMSConfiguration.getACLsConf();
> // triggering the resource loading.
> conf.get(Type.CREATE.getAclConfigKey());
> return conf;
> }{code}
> if the kms-acls file written within next 100ms , the changes will not be
> loaded as this condition "newer = f.lastModified() - time > 100" never meets
> because we have modified last reload time before getting the configuration
> {code:java}
> public static boolean isACLsFileNewer(long time) {
> boolean newer = false;
> String confDir = System.getProperty(KMS_CONFIG_DIR);
> if (confDir != null) {
> Path confPath = new Path(confDir);
> if (!confPath.isUriPathAbsolute()) {
> throw new RuntimeException("System property '" + KMS_CONFIG_DIR +
> "' must be an absolute path: " + confDir);
> }
> File f = new File(confDir, KMS_ACLS_XML);
> LOG.trace("Checking file {}, modification time is {}, last reload time is"
> + " {}", f.getPath(), f.lastModified(), time);
> // at least 100ms newer than time, we do this to ensure the file
> // has been properly closed/flushed
> newer = f.lastModified() - time > 100;
> }
> return newer;
> } {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14648) DeadNodeDetector state machine model
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910091#comment-16910091 ] Lisheng Sun commented on HDFS-14648: Hi [~linyiqun]. Sorry, this JIRA has been off for so long. I have uploaded v003 patch. Could you mind help review it? Thank you. > DeadNodeDetector state machine model > > > Key: HDFS-14648 > URL: https://issues.apache.org/jira/browse/HDFS-14648 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch, > HDFS-14648.003.patch > > > This Jira constructs DeadNodeDetector state machine model. The function it > implements as follow: > # After DFSInputstream detects some DataNode die, it put in DeadNodeDetector > and share this information to others in the same DFSClient. The ohter > DFSInputstreams will not read this DataNode. > # DeadNodeDetector also have DFSInputstream reference relationships to each > DataNode. When DFSInputstream close, DeadNodeDetector also remove this > reference. If some DeadNode of DeadNodeDetector is not read by > DFSInputstream, it also is removed from DeadNodeDetector. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1972) Provide example ha proxy with multiple s3 servers back end.
[ https://issues.apache.org/jira/browse/HDDS-1972?focusedWorklogId=296965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296965 ] ASF GitHub Bot logged work on HDDS-1972: Author: ASF GitHub Bot Created on: 18/Aug/19 22:08 Start Date: 18/Aug/19 22:08 Worklog Time Spent: 10m Work Description: adoroszlai commented on pull request #1304: HDDS-1972. Provide example ha proxy with multiple s3 servers back end. URL: https://github.com/apache/hadoop/pull/1304#discussion_r315004479 ## File path: hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml ## @@ -0,0 +1,83 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" +services: + s3g: + image: haproxy:latest + volumes: + - ../..:/opt/hadoop + - ./haproxy-conf/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg + ports: + - 8081:9878 Review comment: @bharatviswa504, it's just a guess, but can you try the following change? ```suggestion - 9878:9878 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296965) Time Spent: 2h 50m (was: 2h 40m) > Provide example ha proxy with multiple s3 servers back end. > --- > > Key: HDDS-1972 > URL: https://issues.apache.org/jira/browse/HDDS-1972 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > In this Jira, we shall provide docker-compose files where we start 3 s3 > gateway servers, and ha-proxy is used to load balance these S3 Gateway > Servers. > > In this Jira, all are proxy configurations are hardcoded, we can make > improvements to scale and automatically configure with environment variables > as a future improvement. This is just a starter example. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14745) Backport HDFS persistent read cache to branch-3.1
[ https://issues.apache.org/jira/browse/HDFS-14745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-14745: -- Attachment: HDFS-14745.000.patch > Backport HDFS persistent read cache to branch-3.1 > - > > Key: HDFS-14745 > URL: https://issues.apache.org/jira/browse/HDFS-14745 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Feilong He >Priority: Major > Attachments: HDFS-14745.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14745) Backport HDFS persistent read cache to branch-3.1
Feilong He created HDFS-14745: - Summary: Backport HDFS persistent read cache to branch-3.1 Key: HDFS-14745 URL: https://issues.apache.org/jira/browse/HDFS-14745 Project: Hadoop HDFS Issue Type: Improvement Reporter: Feilong He -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10606) TrashPolicyDefault supports time of auto clean up can configured
[ https://issues.apache.org/jira/browse/HDFS-10606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910024#comment-16910024 ] Wei-Chiu Chuang commented on HDFS-10606: I am not sure I understand the problem statement. It's probably just a matter of English translation though. bq. In the most serious situations it may lead some cron job submit failure. if add configuration about time of cleaning up, it will avoid impact on this cron jobs at that default time. I understand NN may choke on a large trash directory cleanup, but it seems the patch just extend the trash emptier thread sleep time. It's not that clear to me how this is going to remedy the problem. Additionally, is HDFS-13529 or HDFS-14586related in any ways? > TrashPolicyDefault supports time of auto clean up can configured > > > Key: HDFS-10606 > URL: https://issues.apache.org/jira/browse/HDFS-10606 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.0 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-10606-branch-2.7.001.patch, HDFS-10606.001.patch, > HDFS-10606.002.patch > > > TrashPolicyDefault clean up Trash based on > [UTC|http://www.worldtimeserver.com/current_time_in_UTC.aspx] currently and > the time of cleaning up is 00:00 UTC. when there are large amount of trash > data should be auto-clean, it will block NN for a long time since Global > Lock, In the most serious situations it may lead some cron job submit > failure. if add configuration about time of cleaning up, it will avoid impact > on this cron jobs at that default time. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1974) Implement OM CancelDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910017#comment-16910017 ] Hudson commented on HDDS-1974: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17143 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17143/]) HDDS-1974. Implement OM CancelDelegationToken request to use Cache and (github: rev b83eae7bdb9ec908cfe5ab87c8862cc9125c8aed) * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketCreateRequest.java * (edit) hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/security/OMGetDelegationTokenRequest.java * (delete) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/security/OMDelegationTokenResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/security/OMCancelDelegationTokenResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/security/OMCancelDelegationTokenRequest.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/security/OMGetDelegationTokenResponse.java > Implement OM CancelDelegationToken request to use Cache and DoubleBuffer > > > Key: HDDS-1974 > URL: https://issues.apache.org/jira/browse/HDDS-1974 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Implement OM CancelDelegationToken request to use OM Cache, double buffer. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1903) Use dynamic ports for SCM in TestSCMClientProtocolServer and TestSCMSecurityProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910018#comment-16910018 ] Hudson commented on HDDS-1903: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17143 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17143/]) HDDS-1903 : Use dynamic ports for SCM in TestSCMClientProtocolServer … (bharat: rev e32f52c75ff28c3f8a67ff627acf784c8e4b05e7) * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMSecurityProtocolServer.java * (edit) hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMClientProtocolServer.java > Use dynamic ports for SCM in TestSCMClientProtocolServer and > TestSCMSecurityProtocolServer > -- > > Key: HDDS-1903 > URL: https://issues.apache.org/jira/browse/HDDS-1903 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Nanda kumar >Assignee: Aravindan Vijayan >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We should use dynamic port for SCM in the following test-cases > * TestSCMClientProtocolServer > * TestSCMSecurityProtocolServer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1971) Update document for HDDS-1891: Ozone fs shell command should work with default port when port number is not specified
[
https://issues.apache.org/jira/browse/HDDS-1971?focusedWorklogId=296944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296944
]
ASF GitHub Bot logged work on HDDS-1971:
Author: ASF GitHub Bot
Created on: 18/Aug/19 17:09
Start Date: 18/Aug/19 17:09
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on issue #1306: HDDS-1971.
Update document for HDDS-1891: Ozone fs shell command should work with default
port when port number is not specified
URL: https://github.com/apache/hadoop/pull/1306#issuecomment-522338726
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 296944)
Time Spent: 1.5h (was: 1h 20m)
> Update document for HDDS-1891: Ozone fs shell command should work with
> default port when port number is not specified
> -
>
> Key: HDDS-1971
> URL: https://issues.apache.org/jira/browse/HDDS-1971
> Project: Hadoop Distributed Data Store
> Issue Type: Task
> Components: documentation
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This should've been part of HDDS-1891.
> Now that fs shell command works without specifying a default OM port number.
> We should update the doc on
> https://hadoop.apache.org/ozone/docs/0.4.0-alpha/ozonefs.html:
> {code}
> ... Moreover, the filesystem URI can take a fully qualified form with the OM
> host and port as a part of the path following the volume name.
> {code}
> CC [~bharatviswa]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1903) Use dynamic ports for SCM in TestSCMClientProtocolServer and TestSCMSecurityProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-1903?focusedWorklogId=296941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296941 ] ASF GitHub Bot logged work on HDDS-1903: Author: ASF GitHub Bot Created on: 18/Aug/19 17:08 Start Date: 18/Aug/19 17:08 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1303: HDDS-1903 : Use dynamic ports for SCM in TestSCMClientProtocolServer … URL: https://github.com/apache/hadoop/pull/1303#issuecomment-522338646 Thank You @avijayanhwx for the fix and @nandakumar131 for the review. I will commit this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296941) Time Spent: 2.5h (was: 2h 20m) > Use dynamic ports for SCM in TestSCMClientProtocolServer and > TestSCMSecurityProtocolServer > -- > > Key: HDDS-1903 > URL: https://issues.apache.org/jira/browse/HDDS-1903 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Nanda kumar >Assignee: Aravindan Vijayan >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > We should use dynamic port for SCM in the following test-cases > * TestSCMClientProtocolServer > * TestSCMSecurityProtocolServer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1903) Use dynamic ports for SCM in TestSCMClientProtocolServer and TestSCMSecurityProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-1903?focusedWorklogId=296943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296943 ] ASF GitHub Bot logged work on HDDS-1903: Author: ASF GitHub Bot Created on: 18/Aug/19 17:08 Start Date: 18/Aug/19 17:08 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1303: HDDS-1903 : Use dynamic ports for SCM in TestSCMClientProtocolServer … URL: https://github.com/apache/hadoop/pull/1303 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296943) Time Spent: 2h 50m (was: 2h 40m) > Use dynamic ports for SCM in TestSCMClientProtocolServer and > TestSCMSecurityProtocolServer > -- > > Key: HDDS-1903 > URL: https://issues.apache.org/jira/browse/HDDS-1903 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Nanda kumar >Assignee: Aravindan Vijayan >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We should use dynamic port for SCM in the following test-cases > * TestSCMClientProtocolServer > * TestSCMSecurityProtocolServer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1903) Use dynamic ports for SCM in TestSCMClientProtocolServer and TestSCMSecurityProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-1903?focusedWorklogId=296942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296942 ] ASF GitHub Bot logged work on HDDS-1903: Author: ASF GitHub Bot Created on: 18/Aug/19 17:08 Start Date: 18/Aug/19 17:08 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1303: HDDS-1903 : Use dynamic ports for SCM in TestSCMClientProtocolServer … URL: https://github.com/apache/hadoop/pull/1303#issuecomment-522338646 Thank You @avijayanhwx for the fix. @nandakumar131 and @adoroszlai for the review. I will commit this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296942) Time Spent: 2h 40m (was: 2.5h) > Use dynamic ports for SCM in TestSCMClientProtocolServer and > TestSCMSecurityProtocolServer > -- > > Key: HDDS-1903 > URL: https://issues.apache.org/jira/browse/HDDS-1903 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Nanda kumar >Assignee: Aravindan Vijayan >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > We should use dynamic port for SCM in the following test-cases > * TestSCMClientProtocolServer > * TestSCMSecurityProtocolServer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1903) Use dynamic ports for SCM in TestSCMClientProtocolServer and TestSCMSecurityProtocolServer
[ https://issues.apache.org/jira/browse/HDDS-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1903: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) > Use dynamic ports for SCM in TestSCMClientProtocolServer and > TestSCMSecurityProtocolServer > -- > > Key: HDDS-1903 > URL: https://issues.apache.org/jira/browse/HDDS-1903 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: test >Reporter: Nanda kumar >Assignee: Aravindan Vijayan >Priority: Major > Labels: newbie, pull-request-available > Fix For: 0.5.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We should use dynamic port for SCM in the following test-cases > * TestSCMClientProtocolServer > * TestSCMSecurityProtocolServer -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1974) Implement OM CancelDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1974: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) > Implement OM CancelDelegationToken request to use Cache and DoubleBuffer > > > Key: HDDS-1974 > URL: https://issues.apache.org/jira/browse/HDDS-1974 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Implement OM CancelDelegationToken request to use OM Cache, double buffer. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1974) Implement OM CancelDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1974?focusedWorklogId=296940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296940 ] ASF GitHub Bot logged work on HDDS-1974: Author: ASF GitHub Bot Created on: 18/Aug/19 17:06 Start Date: 18/Aug/19 17:06 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #1308: HDDS-1974. Implement OM CancelDelegationToken request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1308 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296940) Time Spent: 1h 50m (was: 1h 40m) > Implement OM CancelDelegationToken request to use Cache and DoubleBuffer > > > Key: HDDS-1974 > URL: https://issues.apache.org/jira/browse/HDDS-1974 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Implement OM CancelDelegationToken request to use OM Cache, double buffer. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1974) Implement OM CancelDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1974?focusedWorklogId=296939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296939 ] ASF GitHub Bot logged work on HDDS-1974: Author: ASF GitHub Bot Created on: 18/Aug/19 17:03 Start Date: 18/Aug/19 17:03 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #1308: HDDS-1974. Implement OM CancelDelegationToken request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1308#issuecomment-522338325 Test failures are not related to this patch. Thank You @arp7 for the review. I will commit this to the trunk. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296939) Time Spent: 1h 40m (was: 1.5h) > Implement OM CancelDelegationToken request to use Cache and DoubleBuffer > > > Key: HDDS-1974 > URL: https://issues.apache.org/jira/browse/HDDS-1974 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Implement OM CancelDelegationToken request to use OM Cache, double buffer. > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14648) DeadNodeDetector state machine model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909996#comment-16909996
]
Hadoop QA commented on HDFS-14648:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
127 unchanged - 1 fixed = 129 total (was 128) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 94m
29s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14648 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977890/HDFS-14648.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux bc85e463097d 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3bba808 |
| maven | version: Apache
[jira] [Commented] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909972#comment-16909972 ] Ewan Higgs commented on HDFS-13118: --- [~jojochuang], I turned this into a github with a rebase MR: https://github.com/apache/hadoop/pull/1313 Will take a look at the findbugs warnings. > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13118) SnapshotDiffReport should provide the INode type
[ https://issues.apache.org/jira/browse/HDFS-13118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewan Higgs updated HDFS-13118: -- Status: Open (was: Patch Available) > SnapshotDiffReport should provide the INode type > > > Key: HDFS-13118 > URL: https://issues.apache.org/jira/browse/HDFS-13118 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Ewan Higgs >Assignee: Ewan Higgs >Priority: Major > Attachments: HDFS-13118.001.patch, HDFS-13118.002.patch, > HDFS-13118.003.patch, HDFS-13118.004.patch, HDFS-13118.005.patch > > > Currently the snapshot diff report will list which inodes were added, > removed, renamed, etc. But to see what the INode actually is, we need to > actually access the underlying snapshot - and this is cumbersome to do > programmatically when the snapshot diff already has the information. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14648) DeadNodeDetector state machine model
[ https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-14648: --- Attachment: HDFS-14648.003.patch > DeadNodeDetector state machine model > > > Key: HDFS-14648 > URL: https://issues.apache.org/jira/browse/HDFS-14648 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch, > HDFS-14648.003.patch > > > This Jira constructs DeadNodeDetector state machine model. The function it > implements as follow: > # After DFSInputstream detects some DataNode die, it put in DeadNodeDetector > and share this information to others in the same DFSClient. The ohter > DFSInputstreams will not read this DataNode. > # DeadNodeDetector also have DFSInputstream reference relationships to each > DataNode. When DFSInputstream close, DeadNodeDetector also remove this > reference. If some DeadNode of DeadNodeDetector is not read by > DFSInputstream, it also is removed from DeadNodeDetector. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909945#comment-16909945
]
Hadoop QA commented on HDFS-14687:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.7 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14687 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977884/HDFS-14687.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7d915e48197d 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3bba808 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27551/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27551/testReport/ |
| Max. process+thread count | 2937 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27551/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was
[jira] [Work logged] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir
[ https://issues.apache.org/jira/browse/HDDS-1946?focusedWorklogId=296911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296911 ] ASF GitHub Bot logged work on HDDS-1946: Author: ASF GitHub Bot Created on: 18/Aug/19 10:37 Start Date: 18/Aug/19 10:37 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1311: HDDS-1946. CertificateClient should not persist keys/certs to ozone.m… URL: https://github.com/apache/hadoop/pull/1311#issuecomment-522310100 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 72 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 11 | Maven dependency ordering for branch | | +1 | mvninstall | 599 | trunk passed | | +1 | compile | 371 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 929 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | trunk passed | | 0 | spotbugs | 432 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 632 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 562 | the patch passed | | +1 | compile | 373 | the patch passed | | +1 | javac | 373 | the patch passed | | +1 | checkstyle | 74 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 753 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 188 | the patch passed | | +1 | findbugs | 795 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 372 | hadoop-hdds in the patch passed. | | -1 | unit | 2365 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 8560 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.scm.node.TestQueryNode | | | hadoop.ozone.om.TestSecureOzoneManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog | | | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1311/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1311 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 462101c50b63 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 3bba808 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1311/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1311/3/testReport/ | | Max. process+thread count | 5093 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service U: hadoop-hdds | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1311/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296911) Time Spent: 1.5h (was: 1h 20m) > CertificateClient should not persist keys/certs to ozone.metadata.dir > - > > Key: HDDS-1946 > URL: https://issues.apache.org/jira/browse/HDDS-1946 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Vivek Ratnavel Subramanian
[jira] [Commented] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909926#comment-16909926
]
Surendra Singh Lilhore commented on HDFS-14687:
---
Thanks [~jojochuang],
Attached new patch, reduce number of datanode, earlier it was using 9 DN's (6+3
policy), now it is using 3 DN's (2+1 policy). In my machine it is taking 3
seconds.
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14687) Standby Namenode never come out of safemode when EC files are being written.
[
https://issues.apache.org/jira/browse/HDFS-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14687:
--
Attachment: HDFS-14687.004.patch
> Standby Namenode never come out of safemode when EC files are being written.
>
>
> Key: HDFS-14687
> URL: https://issues.apache.org/jira/browse/HDFS-14687
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
> Attachments: HDFS-14687.001.patch, HDFS-14687.002.patch,
> HDFS-14687.003.patch, HDFS-14687.004.patch
>
>
> When huge number of EC files are being written and SBN is restarted then it
> will never come out of safe mode and required blocks count getting increase.
> {noformat}
> The reported blocks 16658401 needs additional 1702 blocks to reach the
> threshold 0.9 of total blocks 16660120.
> The reported blocks 16658659 needs additional 2935 blocks to reach the
> threshold 0.9 of total blocks 16661611.
> The reported blocks 16659947 needs additional 3868 blocks to reach the
> threshold 0.9 of total blocks 16663832.
> The reported blocks 1335 needs additional 5116 blocks to reach the
> threshold 0.9 of total blocks 16671468.
> The reported blocks 16669311 needs additional 6384 blocks to reach the
> threshold 0.9 of total blocks 16675712.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1946) CertificateClient should not persist keys/certs to ozone.metadata.dir
[ https://issues.apache.org/jira/browse/HDDS-1946?focusedWorklogId=296894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296894 ] ASF GitHub Bot logged work on HDDS-1946: Author: ASF GitHub Bot Created on: 18/Aug/19 08:20 Start Date: 18/Aug/19 08:20 Worklog Time Spent: 10m Work Description: vivekratnavel commented on issue #1311: HDDS-1946. CertificateClient should not persist keys/certs to ozone.m… URL: https://github.com/apache/hadoop/pull/1311#issuecomment-522301588 @bharatviswa504 Thanks for the review! I have updated the patch to include component name in the key location as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296894) Time Spent: 1h 20m (was: 1h 10m) > CertificateClient should not persist keys/certs to ozone.metadata.dir > - > > Key: HDDS-1946 > URL: https://issues.apache.org/jira/browse/HDDS-1946 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Vivek Ratnavel Subramanian >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > For example, when OM and SCM are deployed on the same host with > ozone.metadata.dir defined. SCM can start successfully but OM can not because > the key/cert from OM will collide with SCM. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1974) Implement OM CancelDelegationToken request to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1974?focusedWorklogId=296892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-296892 ] ASF GitHub Bot logged work on HDDS-1974: Author: ASF GitHub Bot Created on: 18/Aug/19 07:14 Start Date: 18/Aug/19 07:14 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1308: HDDS-1974. Implement OM CancelDelegationToken request to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/1308#issuecomment-522297728 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 142 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 33 | Maven dependency ordering for branch | | +1 | mvninstall | 745 | trunk passed | | +1 | compile | 418 | trunk passed | | +1 | checkstyle | 93 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 1078 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 178 | trunk passed | | 0 | spotbugs | 454 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 670 | trunk passed | | -0 | patch | 494 | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 562 | the patch passed | | +1 | compile | 373 | the patch passed | | +1 | javac | 373 | the patch passed | | +1 | checkstyle | 75 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 747 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | the patch passed | | +1 | findbugs | 657 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 342 | hadoop-hdds in the patch passed. | | -1 | unit | 2250 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 8733 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog | | | hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1308/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1308 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux c2d079d529d4 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d873ddd | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1308/5/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1308/5/testReport/ | | Max. process+thread count | 5034 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1308/5/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 296892) Time Spent: 1.5h (was: 1h 20m) > Implement OM CancelDelegationToken request to use Cache and DoubleBuffer > > > Key: HDDS-1974 >
[jira] [Commented] (HDFS-14646) Standby NameNode should not upload fsimage to an inappropriate NameNode.
[
https://issues.apache.org/jira/browse/HDFS-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909904#comment-16909904
]
Hadoop QA commented on HDFS-14646:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 80 unchanged - 1 fixed = 80 total (was 81) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}212m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.server.namenode.TestTransferFsImage |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14646 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977878/HDFS-14646.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 11faca00b04a 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d873ddd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-12831) HDFS throws FileNotFoundException on getFileBlockLocations(path-to-directory)
[
https://issues.apache.org/jira/browse/HDFS-12831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909903#comment-16909903
]
hemanthboyina commented on HDFS-12831:
--
it will be better if we throw exception as PathIsDirectoryException
Need to change it in INodeFIle.java
> HDFS throws FileNotFoundException on getFileBlockLocations(path-to-directory)
> -
>
> Key: HDFS-12831
> URL: https://issues.apache.org/jira/browse/HDFS-12831
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Hanisha Koneru
>Priority: Major
>
> The HDFS implementation of {{getFileBlockLocations(path, offset, len)}}
> throws an exception if the path references a directory.
> The base implementation (and all other filesystems) just return an empty
> array, something implemented in {{getFileBlockLocations(filestatsus, offset,
> len)}}; something written up in filesystem.md as the correct behaviour.
> # has been shown to break things: SPARK-14959
> # there's no contract tests for these APIs; shows up in HADOOP-15044.
> # even if this is considered a wontfix, it should raise something like
> {{PathIsDirectoryException}} rather than FNFE
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13019) dfs put with -f to dir with existing file in dest should return 0, not -1
[ https://issues.apache.org/jira/browse/HDFS-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909902#comment-16909902 ] hemanthboyina commented on HDFS-13019: -- [~bharatviswa] are you working on this ? > dfs put with -f to dir with existing file in dest should return 0, not -1 > - > > Key: HDFS-13019 > URL: https://issues.apache.org/jira/browse/HDFS-13019 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.3 >Reporter: BRYAN T VOLD >Assignee: Bharat Viswanadham >Priority: Major > > When doing an hdfs dfs -put and there are existing > files, the return code will be -1, which is expected. > When you do an hdfs dfs -put -f (force), the error code > still comes back as -1, which is unexpected. > If you use hdfs dfs -copyFromLocal using the same directories as above, the > -copyFromLocal stills gives the error which is expected and when you pass -f > to this version of the command, the error code is 0, which I think is the > correct behavior and I think the hdfs dfs -put should match this. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14720) DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.
[
https://issues.apache.org/jira/browse/HDFS-14720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909900#comment-16909900
]
Hadoop QA commented on HDFS-14720:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 21s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977879/HDFS-14720.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 731ef4ee7a0f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d873ddd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27550/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27550/testReport/ |
| Max. process+thread count | 4607 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11911) SnapshotDiff should maintain the order of file/dir creation and deletion
[
https://issues.apache.org/jira/browse/HDFS-11911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909901#comment-16909901
]
hemanthboyina commented on HDFS-11911:
--
[~manojg] are you working on this ?
> SnapshotDiff should maintain the order of file/dir creation and deletion
>
>
> Key: HDFS-11911
> URL: https://issues.apache.org/jira/browse/HDFS-11911
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
>
> {{DirectoryWithSnapshotFeature}} maintains a separate list for CREATED and
> DELETED children but the ordering of these creation and deletion events are
> not maintained. Assume a case like below, where the time is growing
> downwards...
> {noformat}
> |
> + CREATE File-1
> |
> + Snap S1 created
> |
> + DELETE File-1
> |
> + Snap S2 created
> |
> + CREATE File-1
> |
> + Snap S3 created
> |
> |
> V
> {noformat}
> The snapshot diff report which takes in the DirectoryWithSnapshotFeature diff
> entries and just prints all the creation first and then the deletions,
> thereby giving the perception that file-1 got created first and then got
> deleted. But after S3, file-1 is still available.
> {noformat}
> The difference between snapshot S1 and snapshot S3 under the directory /:
> M .
> + ./file-1
> - ./file-1
> {noformat}
> Can we have DirectoryWithSnapshotFeature maintain the diff entries ordered by
> time or sequence?
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
