[jira] [Assigned] (HDFS-14012) Add diag info in RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HDFS-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Chitlangia reassigned HDFS-14012:
Assignee: Dinesh Chitlangia
> Add diag info in RetryInvocationHandler
> ---
>
> Key: HDFS-14012
> URL: https://issues.apache.org/jira/browse/HDFS-14012
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Yongjun Zhang
>Assignee: Dinesh Chitlangia
>Priority: Major
>
> RetryInvocationHandler does the following logging:
> {code:java}
> } else {
> LOG.warn("A failover has occurred since the start of this method" + "
> invocation attempt.");
> }{code}
> Would be helpful to report the method name, and call stack in this message.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14012) Add diag info in RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HDFS-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658064#comment-16658064
]
Dinesh Chitlangia commented on HDFS-14012:
--
[~yzhangal] In which version are you seeing this?
In trunk, I see the following:
{panel:title=hadoop/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java}
{code:java}
else {
LOG.warn("A failover has occurred since the start of call #" + callId
+ " " + proxyInfo.getString(method.getName()));
}{code}
{panel}
> Add diag info in RetryInvocationHandler
> ---
>
> Key: HDFS-14012
> URL: https://issues.apache.org/jira/browse/HDFS-14012
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Yongjun Zhang
>Priority: Major
>
> RetryInvocationHandler does the following logging:
> {code:java}
> } else {
> LOG.warn("A failover has occurred since the start of this method" + "
> invocation attempt.");
> }{code}
> Would be helpful to report the method name, and call stack in this message.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14012) Add diag info in RetryInvocationHandler
[
https://issues.apache.org/jira/browse/HDFS-14012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-14012:
-
Environment: (was: RetryInvocationHandler does the following logging:
{code:java}
} else {
LOG.warn("A failover has occurred since the start of this method"
+ " invocation attempt.");
}{code}
Would be helpful to report the method name, and call stack in this message.
Thanks.)
Description:
RetryInvocationHandler does the following logging:
{code:java}
} else {
LOG.warn("A failover has occurred since the start of this method" + "
invocation attempt.");
}{code}
Would be helpful to report the method name, and call stack in this message.
Thanks.
> Add diag info in RetryInvocationHandler
> ---
>
> Key: HDFS-14012
> URL: https://issues.apache.org/jira/browse/HDFS-14012
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Yongjun Zhang
>Priority: Major
>
> RetryInvocationHandler does the following logging:
> {code:java}
> } else {
> LOG.warn("A failover has occurred since the start of this method" + "
> invocation attempt.");
> }{code}
> Would be helpful to report the method name, and call stack in this message.
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14012) Add diag info in RetryInvocationHandler
Yongjun Zhang created HDFS-14012:
Summary: Add diag info in RetryInvocationHandler
Key: HDFS-14012
URL: https://issues.apache.org/jira/browse/HDFS-14012
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Environment: RetryInvocationHandler does the following logging:
{code:java}
} else {
LOG.warn("A failover has occurred since the start of this method"
+ " invocation attempt.");
}{code}
Would be helpful to report the method name, and call stack in this message.
Thanks.
Reporter: Yongjun Zhang
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-695) Introduce a new SCM Command to teardown a Pipeline
[
https://issues.apache.org/jira/browse/HDDS-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658058#comment-16658058
]
Yiqun Lin commented on HDDS-695:
Only one comment from me:
*PipelineSelector#demolishPipeline*
Can we print a message when not finding the specified pipeline?
{code}
+ /**
+ * Demolishes a given pipeline.
+ */
+ public void demolishPipeline(PipelineID pipelineID) throws IOException {
+final Pipeline pipeline = pipelineMap.get(pipelineID);
+if (pipeline == null) {
+ // pipeline not found;
+ return;
+}
+LOG.info("Demolishing pipeline. pipelineID: {}", pipeline.getId());
+finalizePipeline(pipeline);
+...
+ }
{code}
> Introduce a new SCM Command to teardown a Pipeline
> --
>
> Key: HDDS-695
> URL: https://issues.apache.org/jira/browse/HDDS-695
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Attachments: HDDS-695-ozone-0.3.000.patch
>
>
> We need to have a tear-down pipeline command in SCM so that an administrator
> can close/destroy a pipeline in the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13087) Snapshotted encryption zone information should be immutable
[ https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng reassigned HDFS-13087: - Assignee: LiXin Ge (was: Siyao Meng) > Snapshotted encryption zone information should be immutable > --- > > Key: HDFS-13087 > URL: https://issues.apache.org/jira/browse/HDFS-13087 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: LiXin Ge >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.3 > > Attachments: HDFS-13087.001.patch, HDFS-13087.002.patch, > HDFS-13087.003.patch, HDFS-13087.004.patch, HDFS-13087.005.patch > > > Snapshots are supposed to be immutable and read only, so the EZ settings > which in a snapshot path shouldn't change when the origin encryption zone > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13087) Snapshotted encryption zone information should be immutable
[ https://issues.apache.org/jira/browse/HDFS-13087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng reassigned HDFS-13087: - Assignee: Siyao Meng (was: LiXin Ge) > Snapshotted encryption zone information should be immutable > --- > > Key: HDFS-13087 > URL: https://issues.apache.org/jira/browse/HDFS-13087 > Project: Hadoop HDFS > Issue Type: Bug > Components: encryption >Affects Versions: 3.1.0 >Reporter: LiXin Ge >Assignee: Siyao Meng >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.3 > > Attachments: HDFS-13087.001.patch, HDFS-13087.002.patch, > HDFS-13087.003.patch, HDFS-13087.004.patch, HDFS-13087.005.patch > > > Snapshots are supposed to be immutable and read only, so the EZ settings > which in a snapshot path shouldn't change when the origin encryption zone > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12284) RBF: Support for Kerberos authentication
[
https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658045#comment-16658045
]
Hadoop QA commented on HDFS-12284:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13532 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
44s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13532 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m
25s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-12284 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944874/HDFS-12284-HDFS-13532.010.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 47828b672316 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13532 / 96ae4ac |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25324/testReport/ |
| Max. process+thread count | 1056 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25324/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This
[jira] [Updated] (HDFS-12284) RBF: Support for Kerberos authentication
[ https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12284: --- Attachment: HDFS-12284-HDFS-13532.010.patch > RBF: Support for Kerberos authentication > > > Key: HDFS-12284 > URL: https://issues.apache.org/jira/browse/HDFS-12284 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Zhe Zhang >Assignee: Sherwood Zheng >Priority: Major > Attachments: HDFS-12284-HDFS-13532.004.patch, > HDFS-12284-HDFS-13532.005.patch, HDFS-12284-HDFS-13532.006.patch, > HDFS-12284-HDFS-13532.007.patch, HDFS-12284-HDFS-13532.008.patch, > HDFS-12284-HDFS-13532.009.patch, HDFS-12284-HDFS-13532.010.patch, > HDFS-12284.000.patch, HDFS-12284.001.patch, HDFS-12284.002.patch, > HDFS-12284.003.patch > > > HDFS Router should support Kerberos authentication and issuing / managing > HDFS delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12284) RBF: Support for Kerberos authentication
[
https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657941#comment-16657941
]
Hadoop QA commented on HDFS-12284:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13532 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
45s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} HDFS-13532 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13532 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
23s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
24s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
25s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-12284 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944869/HDFS-12284-HDFS-13532.009.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient xml findbugs checkstyle |
| uname | Linux 6e75a711bea3 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13532 / 96ae4ac |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/25323/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/25323/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| javac |
[jira] [Updated] (HDFS-12284) RBF: Support for Kerberos authentication
[ https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12284: --- Attachment: HDFS-12284-HDFS-13532.009.patch > RBF: Support for Kerberos authentication > > > Key: HDFS-12284 > URL: https://issues.apache.org/jira/browse/HDFS-12284 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Zhe Zhang >Assignee: Sherwood Zheng >Priority: Major > Attachments: HDFS-12284-HDFS-13532.004.patch, > HDFS-12284-HDFS-13532.005.patch, HDFS-12284-HDFS-13532.006.patch, > HDFS-12284-HDFS-13532.007.patch, HDFS-12284-HDFS-13532.008.patch, > HDFS-12284-HDFS-13532.009.patch, HDFS-12284.000.patch, HDFS-12284.001.patch, > HDFS-12284.002.patch, HDFS-12284.003.patch > > > HDFS Router should support Kerberos authentication and issuing / managing > HDFS delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14004) TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk
[ https://issues.apache.org/jira/browse/HDFS-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657923#comment-16657923 ] Íñigo Goiri commented on HDFS-14004: Thanks [~ayushtkn] for [^HDFS-14004-02.patch]. The test seems to pass: https://builds.apache.org/job/PreCommit-HDFS-Build/25322/testReport/org.apache.hadoop.hdfs/TestLeaseRecovery2/testCloseWhileRecoverLease/ I'm guessing you've run it a bunch of times and it works (it should given that we will only have one block now). +1 on [^HDFS-14004-02.patch]. > TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk > --- > > Key: HDFS-14004 > URL: https://issues.apache.org/jira/browse/HDFS-14004 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14004-01.patch, HDFS-14004-02.patch > > > Reference > https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/930/testReport/junit/org.apache.hadoop.hdfs/TestLeaseRecovery2/testCloseWhileRecoverLease/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657922#comment-16657922
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
18s{color} | {color:green} integration-test in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} |
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657885#comment-16657885 ] Shashikant Banerjee commented on HDDS-676: -- patch v6 addresses the checkstyle issues. > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, HDDS-676.006.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-676: - Attachment: HDDS-676.006.patch > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch, HDDS-676.006.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657831#comment-16657831
]
Hadoop QA commented on HDDS-676:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red} hadoop-hdds/container-service in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 21m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 21m 16s{color}
| {color:red} root generated 3 new + 1312 unchanged - 0 fixed = 1315 total (was
1312) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 45s{color} | {color:orange} root: The patch generated 3 new + 3 unchanged -
0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} container-service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 47s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} |
[jira] [Updated] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-676: - Attachment: HDDS-676.005.patch > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-676: - Attachment: (was: HDDS-676.005.patch) > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-697) update and validate the BCSID for PutSmallFile/GetSmallFile command
[ https://issues.apache.org/jira/browse/HDDS-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-697: - Description: Similar , to putBlock/GetBlock, putSmallFile transaction in Ratis needs to update the BCSID in the container db on datanode. getSmallFile should validate the bcsId while reading the block similar to getBlock. (was: Similar , to putBlock/Get) > update and validate the BCSID for PutSmallFile/GetSmallFile command > --- > > Key: HDDS-697 > URL: https://issues.apache.org/jira/browse/HDDS-697 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > Similar , to putBlock/GetBlock, putSmallFile transaction in Ratis needs to > update the BCSID in the container db on datanode. getSmallFile should > validate the bcsId while reading the block similar to getBlock. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-697) update and validate the BCSID for PutSmallFile/GetSmallFile command
[ https://issues.apache.org/jira/browse/HDDS-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-697: - Summary: update and validate the BCSID for PutSmallFile/GetSmallFile command (was: update the BCSID for PutSmallFile command) > update and validate the BCSID for PutSmallFile/GetSmallFile command > --- > > Key: HDDS-697 > URL: https://issues.apache.org/jira/browse/HDDS-697 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > Similar , to putBlock/Get -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-697) update the BCSID for PutSmallFile command
[ https://issues.apache.org/jira/browse/HDDS-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-697: - Description: Similar , to putBlock/Get > update the BCSID for PutSmallFile command > - > > Key: HDDS-697 > URL: https://issues.apache.org/jira/browse/HDDS-697 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > > Similar , to putBlock/Get -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657779#comment-16657779
]
Shashikant Banerjee edited comment on HDDS-676 at 10/20/18 8:03 AM:
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different non closed container replica. BCSID
should not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have this block for an open container
replica, It is difficult to figure out at client whether block got deleted or
the block commit transaction has not yet happened on this node(in case delete
is allowed on open containers). But for closed container replica we may not
need to retry. I think we need to think through this more as the retry
behaviour might be different for open and closed container replica here.
Patch v5 addresses rest of your review comments.
was (Author: shashikant):
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different non closed container replica. BCSID
should not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge
[jira] [Comment Edited] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657779#comment-16657779
]
Shashikant Banerjee edited comment on HDDS-676 at 10/20/18 8:01 AM:
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different non closed container replica. BCSID
should not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have this block for an open container
replica, It is difficult to figure out at client whether block got deleted or
the block commit transaction has not yet happened on this node(in case delete
is allowed on open containers). But for closed container replica we may not
need to retry. I think we need to think through this more as the retry
behaviour might be different for open and closed container replica here.
was (Author: shashikant):
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different open container replica. BCSID should
not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have
[jira] [Commented] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657779#comment-16657779
]
Shashikant Banerjee commented on HDDS-676:
--
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different open container replica. BCSID should
not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have this block for an open container
replica, It is difficult to figure out at client whether block got deleted or
the block commit transaction has not yet happened on this node(in case delete
is allowed on open containers). But for closed container replica we may not
need to retry. I think we need to think through this more as the retry
behaviour might be different for open and closed container replica here.
> Enable Read from open Containers via Standalone Protocol
>
>
> Key: HDDS-676
> URL: https://issues.apache.org/jira/browse/HDDS-676
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-676.001.patch, HDDS-676.002.patch,
> HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch
>
>
> With BlockCommitSequenceId getting updated per block commit on open
> containers in OM as well datanode, Ozone Client reads can through Standalone
> protocol not necessarily requiring Ratis. Client should verify the BCSID of
> the container which has the data block , which should always be greater than
> or equal to the BCSID of the block to be read and the existing block BCSID
> should exactly match that of the block to be read. As a part of this, Client
> can try to read from a replica with a supplied BCSID and failover to the next
> one in case the block does ont exist on one replica.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[
https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657779#comment-16657779
]
Shashikant Banerjee edited comment on HDDS-676 at 10/20/18 7:57 AM:
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different open container replica. BCSID should
not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i don't have this block for an open container
replica, It is difficult to figure out at client whether block got deleted or
the block commit transaction has not yet happened on this node(in case delete
is allowed on open containers). But for closed container replica we may not
need to retry. I think we need to think through this more as the retry
behaviour might be different for open and closed container replica here.
was (Author: shashikant):
{code:java}
A question rather than a comment. If we make this a required field, and if get
to a datanode and want to read a block – say a case where we say give any
version of data you have, how does that work? Should we make this optional?
This also means that if you have written a block using 0.2.1, then we cannot
read that via this protocol. It is an Alpha release, so we don't have to
strictly be backward compactable; just a thought.{code}
I agree . Made the bcsId filed optional with default value 0.
{code:java}
More a question of code style: should we have bcsId inside blockID or no? In
the blockManager Interface.
{code}
I feel blockID is used to uniquely identify a block in the system but the BCSID
of a block may be different in different open container replica. BCSID should
not be necessarily a part of BlockId.
{code:java}
This does not match with the earlier logic in putBlock. I am presuming the
putSmallFile call was committed via Ratis. Please correct me if I am wrong.
{code}
I agree. But as per [~jnp] comments above, this will be handled in a separate
Jira HDDS-697.
{code:java}
XceiverClientRatis#watchForCommit - if this is a test only function perhaps
have in a test helper file?
{code}
The api will be used HDDS-675. For now, its being only used in a test. I have
removed the visibleForTesting tag.
{code:java}
I am asking more to make sure that I understand this correctly. We look up the
max container bcsid, and then later read the bcsid. I am guessing we do this
because we are counting on containerBCSId will be cached and it will not cause
a real disk I/O and therefore checking it makes it easier for us to fail faster?
{code}
Yes, i have now changed the code to compare the cached container bcsid value
first and then later compare the block bcsid.
{code:java}
There is also an issue that is not taken care in this code path. What happens
if a block is deleted? If the datanode says I don't have this block, do we
still try all 3 data nodes? I am presuming we don't have to deal with it since
it is an edge case.
{code}
I think, if the datanode says i
[jira] [Updated] (HDDS-676) Enable Read from open Containers via Standalone Protocol
[ https://issues.apache.org/jira/browse/HDDS-676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDDS-676: - Attachment: HDDS-676.005.patch > Enable Read from open Containers via Standalone Protocol > > > Key: HDDS-676 > URL: https://issues.apache.org/jira/browse/HDDS-676 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-676.001.patch, HDDS-676.002.patch, > HDDS-676.003.patch, HDDS-676.004.patch, HDDS-676.005.patch > > > With BlockCommitSequenceId getting updated per block commit on open > containers in OM as well datanode, Ozone Client reads can through Standalone > protocol not necessarily requiring Ratis. Client should verify the BCSID of > the container which has the data block , which should always be greater than > or equal to the BCSID of the block to be read and the existing block BCSID > should exactly match that of the block to be read. As a part of this, Client > can try to read from a replica with a supplied BCSID and failover to the next > one in case the block does ont exist on one replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14004) TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk
[
https://issues.apache.org/jira/browse/HDFS-14004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657756#comment-16657756
]
Hadoop QA commented on HDFS-14004:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingReconstruction |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.fs.TestHdfsNativeCodeLoader |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HDFS-14004 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12944843/HDFS-14004-02.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 1a4e54bd494b 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f069d38 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25322/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25322/testReport/ |
| Max. process+thread count | 2962 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Assigned] (HDDS-528) add cli command to checkChill mode status and exit chill mode
[ https://issues.apache.org/jira/browse/HDDS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chencan reassigned HDDS-528: Assignee: chencan > add cli command to checkChill mode status and exit chill mode > - > > Key: HDDS-528 > URL: https://issues.apache.org/jira/browse/HDDS-528 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: chencan >Priority: Major > > [HDDS-370] introduces below 2 API: > * isScmInChillMode > * forceScmExitChillMode > This jira is to call them via relevant cli command. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
