[jira] [Updated] (HDFS-13869) RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
[
https://issues.apache.org/jira/browse/HDFS-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-13869:
-
Attachment: HDFS-13869-HDFS-13891.010.patch
> RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
> --
>
> Key: HDFS-13869
> URL: https://issues.apache.org/jira/browse/HDFS-13869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13869-002.diff, HDFS-13869-003.diff,
> HDFS-13869-004.patch, HDFS-13869-005.patch, HDFS-13869-006.patch,
> HDFS-13869-007.patch, HDFS-13869-HDFS-13891.009.patch,
> HDFS-13869-HDFS-13891.010.patch, HDFS-13869.patch,
> HDFS-13891-HDFS-13869-008.patch
>
>
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getUsed(NamenodeBeanMetrics.java:205)
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getCapacityUsed(NamenodeBeanMetrics.java:519)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> ngMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13869) RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
[
https://issues.apache.org/jira/browse/HDFS-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718520#comment-16718520
]
Ranith Sardar commented on HDFS-13869:
--
Hi [~elgoiri], thanks for reviewing the patch.
In my current patch, have added the intercept in test method and fixed other
comments also.Please review it once.
> RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics()
> --
>
> Key: HDFS-13869
> URL: https://issues.apache.org/jira/browse/HDFS-13869
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-13869-002.diff, HDFS-13869-003.diff,
> HDFS-13869-004.patch, HDFS-13869-005.patch, HDFS-13869-006.patch,
> HDFS-13869-007.patch, HDFS-13869-HDFS-13891.009.patch,
> HDFS-13869-HDFS-13891.010.patch, HDFS-13869.patch,
> HDFS-13891-HDFS-13869-008.patch
>
>
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getUsed(NamenodeBeanMetrics.java:205)
> at
> org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getCapacityUsed(NamenodeBeanMetrics.java:519)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43){code}
> ngMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13443) RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries.
[
https://issues.apache.org/jira/browse/HDFS-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718519#comment-16718519
]
Mohammad Arshad commented on HDFS-13443:
{quote}The refresh API will only makes sense in the shared State Store type not
local type store. right?{quote}
Yes, refresh will not make sense for local state store but, AFAIK, all router
state stores are shared state store even local file based state store is shared
state store.
{quote}I feel default value for this property should be true and description
should say, "for all the routers"{quote}
changed the description. for changing default value to true, [~surendrasingh]
can you please raise another JIRA.
> RBF: Update mount table cache immediately after changing (add/update/remove)
> mount table entries.
> -
>
> Key: HDFS-13443
> URL: https://issues.apache.org/jira/browse/HDFS-13443
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Labels: RBF
> Attachments: HDFS-13443-012.patch, HDFS-13443-013.patch,
> HDFS-13443-014.patch, HDFS-13443-015.patch, HDFS-13443-016.patch,
> HDFS-13443-branch-2.001.patch, HDFS-13443-branch-2.002.patch,
> HDFS-13443.001.patch, HDFS-13443.002.patch, HDFS-13443.003.patch,
> HDFS-13443.004.patch, HDFS-13443.005.patch, HDFS-13443.006.patch,
> HDFS-13443.007.patch, HDFS-13443.008.patch, HDFS-13443.009.patch,
> HDFS-13443.010.patch, HDFS-13443.011.patch
>
>
> Currently mount table cache is updated periodically, by default cache is
> updated every minute. After change in mount table, user operations may still
> use old mount table. This is bit wrong.
> To update mount table cache, maybe we can do following
> * *Add refresh API in MountTableManager which will update mount table cache.*
> * *When there is a change in mount table entries, router admin server can
> update its cache and ask other routers to update their cache*. For example if
> there are three routers R1,R2,R3 in a cluster then add mount table entry API,
> at admin server side, will perform following sequence of action
> ## user submit add mount table entry request on R1
> ## R1 adds the mount table entry in state store
> ## R1 call refresh API on R2
> ## R1 calls refresh API on R3
> ## R1 directly freshest its cache
> ## Add mount table entry response send back to user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14136) ZKDelegationTokenSecretManager should use KerberosName#getShortName to get the user name for ZK ACL
[
https://issues.apache.org/jira/browse/HDFS-14136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14136:
--
Description:
!image-2018-12-10-17-54-33-361.png!
ZKDelegationTokenSecretManager use only first part of principal to set the
znode ACL's.
We can use *{{KerberosName#getShortName()}}* method for getting the principal
based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
the ACL.
was:
!image-2018-12-10-17-54-33-361.png!
If no rule {{kerberos.removeRealmFromPrincipal=true }} and {{
kerberos.removeHostFromPrincipal=true }} is defined to remove host and realm
from the principal, then authorization fails as full Principal is passed for
auth but set one is splitted one..
We can use *{{KerberosName#getShortName()}}* method for getting the principal
based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
the ACL.
> ZKDelegationTokenSecretManager should use KerberosName#getShortName to get
> the user name for ZK ACL
> ---
>
> Key: HDFS-14136
> URL: https://issues.apache.org/jira/browse/HDFS-14136
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shubham Dewan
>Priority: Blocker
> Attachments: HDFS-14136.001.patch, image-2018-12-10-17-54-33-361.png
>
>
> !image-2018-12-10-17-54-33-361.png!
> ZKDelegationTokenSecretManager use only first part of principal to set the
> znode ACL's.
> We can use *{{KerberosName#getShortName()}}* method for getting the principal
> based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
> the ACL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14136) ZKDelegationTokenSecretManager should use KerberosName#getShortName to get the user name for ZK ACL
[
https://issues.apache.org/jira/browse/HDFS-14136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14136:
--
Summary: ZKDelegationTokenSecretManager should use
KerberosName#getShortName to get the user name for ZK ACL (was: In
ZKDelegationTokenSecretManager class Principal which is used to SetAcl is
harcoded )
> ZKDelegationTokenSecretManager should use KerberosName#getShortName to get
> the user name for ZK ACL
> ---
>
> Key: HDFS-14136
> URL: https://issues.apache.org/jira/browse/HDFS-14136
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shubham Dewan
>Priority: Blocker
> Attachments: HDFS-14136.001.patch, image-2018-12-10-17-54-33-361.png
>
>
> !image-2018-12-10-17-54-33-361.png!
> If no rule {{ kerberos.removeRealmFromPrincipal=true }} and {{
> kerberos.removeHostFromPrincipal=true }} is defined to remove host and realm
> from the principal, then authorization fails as full Principal is passed for
> auth but set one is splitted one..
> We can use *{{KerberosName#getShortName()}}* method for getting the principal
> based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
> the ACL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14136) ZKDelegationTokenSecretManager should use KerberosName#getShortName to get the user name for ZK ACL
[
https://issues.apache.org/jira/browse/HDFS-14136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14136:
--
Description:
!image-2018-12-10-17-54-33-361.png!
If no rule {{kerberos.removeRealmFromPrincipal=true }} and {{
kerberos.removeHostFromPrincipal=true }} is defined to remove host and realm
from the principal, then authorization fails as full Principal is passed for
auth but set one is splitted one..
We can use *{{KerberosName#getShortName()}}* method for getting the principal
based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
the ACL.
was:
!image-2018-12-10-17-54-33-361.png!
If no rule {{ kerberos.removeRealmFromPrincipal=true }} and {{
kerberos.removeHostFromPrincipal=true }} is defined to remove host and realm
from the principal, then authorization fails as full Principal is passed for
auth but set one is splitted one..
We can use *{{KerberosName#getShortName()}}* method for getting the principal
based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
the ACL.
> ZKDelegationTokenSecretManager should use KerberosName#getShortName to get
> the user name for ZK ACL
> ---
>
> Key: HDFS-14136
> URL: https://issues.apache.org/jira/browse/HDFS-14136
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Shubham Dewan
>Priority: Blocker
> Attachments: HDFS-14136.001.patch, image-2018-12-10-17-54-33-361.png
>
>
> !image-2018-12-10-17-54-33-361.png!
> If no rule {{kerberos.removeRealmFromPrincipal=true }} and {{
> kerberos.removeHostFromPrincipal=true }} is defined to remove host and realm
> from the principal, then authorization fails as full Principal is passed for
> auth but set one is splitted one..
> We can use *{{KerberosName#getShortName()}}* method for getting the principal
> based upon rules configured in *{{hadoop.security.auth_to_local}}*and setting
> the ACL.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Status: Patch Available (was: Open) > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14127-HDFS-12943.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Attachment: (was: HDFS-123.000.patch) > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14127-HDFS-12943.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Attachment: HDFS-14127-HDFS-12943.000.patch > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-123.000.patch, HDFS-14127-HDFS-12943.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Status: Open (was: Patch Available) > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-123.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14140) JournalNodeSyncer authentication is failing in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-14140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14140:
--
Status: Patch Available (was: Open)
> JournalNodeSyncer authentication is failing in secure cluster
> -
>
> Key: HDFS-14140
> URL: https://issues.apache.org/jira/browse/HDFS-14140
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, security
>Affects Versions: 3.1.1, 3.0.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14140.001.patch
>
>
> {noformat}
> 2018-12-11 17:25:48,407 | ERROR |
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer$$Lambda$31/208830412@42bcafb2
> | Download of Edit Log file for Syncing failed. Deleting temp file:
> //edits.sync/edits_0049016-0049023 |
> JournalNodeSyncer.java:464
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Error while authenticating with endpoint:
> https://xx/getJournal?jid=hacluster=49016=-64%3A716441797%3A1544438882625%3Amyhacluster=false
> at org.apache.hadoop.hdfs.server.common.Util.doGetUrl(Util.java:160)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.downloadMissingLogSegment(JournalNodeSyncer.java:460)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.getMissingLogSegments(JournalNodeSyncer.java:360)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:262)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14140) JournalNodeSyncer authentication is failing in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-14140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718508#comment-16718508
]
Surendra Singh Lilhore commented on HDFS-14140:
---
Attached patch, pls review..
> JournalNodeSyncer authentication is failing in secure cluster
> -
>
> Key: HDFS-14140
> URL: https://issues.apache.org/jira/browse/HDFS-14140
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, security
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14140.001.patch
>
>
> {noformat}
> 2018-12-11 17:25:48,407 | ERROR |
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer$$Lambda$31/208830412@42bcafb2
> | Download of Edit Log file for Syncing failed. Deleting temp file:
> //edits.sync/edits_0049016-0049023 |
> JournalNodeSyncer.java:464
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Error while authenticating with endpoint:
> https://xx/getJournal?jid=hacluster=49016=-64%3A716441797%3A1544438882625%3Amyhacluster=false
> at org.apache.hadoop.hdfs.server.common.Util.doGetUrl(Util.java:160)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.downloadMissingLogSegment(JournalNodeSyncer.java:460)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.getMissingLogSegments(JournalNodeSyncer.java:360)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:262)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14140) JournalNodeSyncer authentication is failing in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-14140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-14140:
--
Attachment: HDFS-14140.001.patch
> JournalNodeSyncer authentication is failing in secure cluster
> -
>
> Key: HDFS-14140
> URL: https://issues.apache.org/jira/browse/HDFS-14140
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, security
>Affects Versions: 3.0.0, 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14140.001.patch
>
>
> {noformat}
> 2018-12-11 17:25:48,407 | ERROR |
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer$$Lambda$31/208830412@42bcafb2
> | Download of Edit Log file for Syncing failed. Deleting temp file:
> //edits.sync/edits_0049016-0049023 |
> JournalNodeSyncer.java:464
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> Error while authenticating with endpoint:
> https://xx/getJournal?jid=hacluster=49016=-64%3A716441797%3A1544438882625%3Amyhacluster=false
> at org.apache.hadoop.hdfs.server.common.Util.doGetUrl(Util.java:160)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.downloadMissingLogSegment(JournalNodeSyncer.java:460)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.getMissingLogSegments(JournalNodeSyncer.java:360)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncWithJournalAtIndex(JournalNodeSyncer.java:262)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.syncJournals(JournalNodeSyncer.java:230)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer.lambda$startSyncJournalsDaemon$0(JournalNodeSyncer.java:190)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718506#comment-16718506 ] xiangheng commented on HDFS-14138: -- hi [~ayushtkn],Thanks for your answer.That’s really useful,thank you very much. > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[
https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718498#comment-16718498
]
Hadoop QA commented on HDFS-14138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
51s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
46s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951459/HDFS-14138-HDFS-12943.000.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5863292d7325 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12943 / 2fe49de |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25774/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25774/testReport/ |
| Max. process+thread count | 1671 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718494#comment-16718494
]
Ranith Sardar commented on HDFS-14129:
--
Hi [~surendrasingh], Thank for the review.
I went through all your comments and fixed. Added UT also. Please check it once.
> RBF: Create new policy provider for router
> --
>
> Key: HDFS-14129
> URL: https://issues.apache.org/jira/browse/HDFS-14129
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-13532
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14129-HDFS-13891.001.patch,
> HDFS-14129-HDFS-13891.002.patch
>
>
> Router is using *{{HDFSPolicyProvider}}*. We can't add new protocol in this
> class for router, its better to create in policy provider for Router.
> {code:java}
> // Set service-level authorization security policy
> if (conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false)) {
> this.adminServer.refreshServiceAcl(conf, new HDFSPolicyProvider());
> }
> {code}
> I got this issue when I am verified HDFS-14079 with secure cluster.
> {noformat}
> ./bin/hdfs dfsrouteradmin -ls /
> ls: Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol
> is not known.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol is
> not known.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1520)
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14129) RBF: Create new policy provider for router
[
https://issues.apache.org/jira/browse/HDFS-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ranith Sardar updated HDFS-14129:
-
Attachment: HDFS-14129-HDFS-13891.002.patch
> RBF: Create new policy provider for router
> --
>
> Key: HDFS-14129
> URL: https://issues.apache.org/jira/browse/HDFS-14129
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-13532
>Reporter: Surendra Singh Lilhore
>Assignee: Ranith Sardar
>Priority: Major
> Attachments: HDFS-14129-HDFS-13891.001.patch,
> HDFS-14129-HDFS-13891.002.patch
>
>
> Router is using *{{HDFSPolicyProvider}}*. We can't add new protocol in this
> class for router, its better to create in policy provider for Router.
> {code:java}
> // Set service-level authorization security policy
> if (conf.getBoolean(HADOOP_SECURITY_AUTHORIZATION, false)) {
> this.adminServer.refreshServiceAcl(conf, new HDFSPolicyProvider());
> }
> {code}
> I got this issue when I am verified HDFS-14079 with secure cluster.
> {noformat}
> ./bin/hdfs dfsrouteradmin -ls /
> ls: Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol
> is not known.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException):
> Protocol interface org.apache.hadoop.hdfs.protocolPB.RouterAdminProtocol is
> not known.
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1520)
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk
[
https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718474#comment-16718474
]
Hadoop QA commented on HDFS-14135:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14135 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951450/HDFS-14135-06.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ad1ef28a8b59 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fb55e52 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/25773/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25773/testReport/ |
| Max. process+thread count | 4488 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25773/console |
| Powered by |
[jira] [Commented] (HDFS-13873) ObserverNode should reject read requests when it is too far behind.
[
https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718465#comment-16718465
]
Hadoop QA commented on HDFS-13873:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
58s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
58s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
49s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
16s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 48s{color} | {color:orange} root: The patch generated 1 new + 220 unchanged
- 0 fixed = 221 total (was 220) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
37s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}191m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13873 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951442/HDFS-13873-HDFS-12943.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: (was: HDFS-14138-HDFS-12943.000.patch.patch) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Patch Available (was: Open) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: (was: HDFS-12998.000.patch) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Open (was: Patch Available) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: HDFS-14138-HDFS-12943.000.patch > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: HDFS-14138-HDFS-12943.000.patch.patch > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-14138-HDFS-12943.000.patch.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Fix Version/s: (was: HDFS-12943) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718442#comment-16718442 ] Ayush Saxena commented on HDFS-14138: - [~xiangheng] while uploading patch for branch try using *HDFS-14138-HDFS-12943.000.patch* naming convention.This will go for branch HDFS-12943 else it will go on trunk where HDFS-12943 is not there. Moreover don't set the Fix version.Affect version is enough.Fix version is to be set after the patch gets committed. :) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Fix For: HDFS-12943 > > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Fix Version/s: (was: HDFS-12943) > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-123.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-918) Expose SCMMXBean
[
https://issues.apache.org/jira/browse/HDDS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham reassigned HDDS-918:
---
Assignee: (was: Bharat Viswanadham)
> Expose SCMMXBean
>
>
> Key: HDDS-918
> URL: https://issues.apache.org/jira/browse/HDDS-918
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Priority: Major
>
> Implement MetricsSource interface, so that external metrics can collect the
> SCMMXBean metrics information.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-918) Expose SCMMXBean
Bharat Viswanadham created HDDS-918:
---
Summary: Expose SCMMXBean
Key: HDDS-918
URL: https://issues.apache.org/jira/browse/HDDS-918
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
Implement MetricsSource interface, so that external metrics can collect the
NodeManagerMXBean metrics information.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-917) Expose NodeManagerMXBean
[
https://issues.apache.org/jira/browse/HDDS-917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-917:
Description:
Implement MetricsSource interface, so that external metrics can collect the
NodeManagerMXBean metrics information.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
was:
Implement MetricsSource interface, so that external metrics can collect the
OMMetrics.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
> Expose NodeManagerMXBean
>
>
> Key: HDDS-917
> URL: https://issues.apache.org/jira/browse/HDDS-917
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>
> Implement MetricsSource interface, so that external metrics can collect the
> NodeManagerMXBean metrics information.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-918) Expose SCMMXBean
[
https://issues.apache.org/jira/browse/HDDS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-918:
Description:
Implement MetricsSource interface, so that external metrics can collect the
SCMMXBean metrics information.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
was:
Implement MetricsSource interface, so that external metrics can collect the
NodeManagerMXBean metrics information.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
> Expose SCMMXBean
>
>
> Key: HDDS-918
> URL: https://issues.apache.org/jira/browse/HDDS-918
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
>
> Implement MetricsSource interface, so that external metrics can collect the
> SCMMXBean metrics information.
>
> From *MetricsSource.java:*
> It registers with \{@link MetricsSystem}, which periodically polls it to
> collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-917) Expose NodeManagerMXBean
Bharat Viswanadham created HDDS-917:
---
Summary: Expose NodeManagerMXBean
Key: HDDS-917
URL: https://issues.apache.org/jira/browse/HDDS-917
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Bharat Viswanadham
Assignee: Bharat Viswanadham
Implement MetricsSource interface, so that external metrics can collect the
OMMetrics.
>From *MetricsSource.java:*
It registers with \{@link MetricsSystem}, which periodically polls it to
collect \{@link MetricsRecord} and passes it to \{@link MetricsSink}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718415#comment-16718415 ] xiangheng commented on HDFS-14138: -- Rebase to trunk and update on HDFS-12943 branch,But “does not apply to trunk” still exists. > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Fix For: HDFS-12943 > > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14135) TestWebHdfsTimeouts Fails intermittently in trunk
[ https://issues.apache.org/jira/browse/HDFS-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14135: Attachment: HDFS-14135-06.patch > TestWebHdfsTimeouts Fails intermittently in trunk > - > > Key: HDFS-14135 > URL: https://issues.apache.org/jira/browse/HDFS-14135 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14135-01.patch, HDFS-14135-02.patch, > HDFS-14135-03.patch, HDFS-14135-04.patch, HDFS-14135-05.patch, > HDFS-14135-06.patch > > > Reference to failure > https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/982/testReport/junit/org.apache.hadoop.hdfs.web/TestWebHdfsTimeouts/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14142) Move ipfailover config key out of HdfsClientConfigKeys
[
https://issues.apache.org/jira/browse/HDFS-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718394#comment-16718394
]
Hadoop QA commented on HDFS-14142:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
50s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-client: The
patch generated 1 new + 28 unchanged - 0 fixed = 29 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
35s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14142 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951444/HDFS-14142-HDFS-12943.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 8841118bbb28 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12943 / 2fe49de |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/25771/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/25771/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C:
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Fix Version/s: HDFS-12943 > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Fix For: HDFS-12943 > > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14127) Add a description about the observer read configuration
[ https://issues.apache.org/jira/browse/HDFS-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14127: - Fix Version/s: HDFS-12943 > Add a description about the observer read configuration > --- > > Key: HDFS-14127 > URL: https://issues.apache.org/jira/browse/HDFS-14127 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Fix For: HDFS-12943 > > Attachments: HDFS-123.000.patch > > > The lack of description of observer reader configuration in hdfs-default.xml > ,That can easily lead users to configure observer read mode to a normal HA > mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[
https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718392#comment-16718392
]
Hadoop QA commented on HDFS-14138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HDFS-14138 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951448/HDFS-12998.000.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25772/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Description errors in the comparison logic of transaction ID
>
>
> Key: HDFS-14138
> URL: https://issues.apache.org/jira/browse/HDFS-14138
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: xiangheng
>Priority: Minor
> Attachments: HDFS-12998.000.patch
>
>
> The call processing should be postponed until the client call's state id is
> aligned (<=) with the server state id,not >=.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: HDFS-12998.000.patch > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Patch Available (was: Open) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Open (was: Patch Available) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: (was: HDFS-12348.000.patch) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12998.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14142) Move ipfailover config key out of HdfsClientConfigKeys
[ https://issues.apache.org/jira/browse/HDFS-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718358#comment-16718358 ] Konstantin Shvachko commented on HDFS-14142: +1 makes sense. Pending Jenkins > Move ipfailover config key out of HdfsClientConfigKeys > -- > > Key: HDFS-14142 > URL: https://issues.apache.org/jira/browse/HDFS-14142 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Minor > Attachments: HDFS-14142-HDFS-12943.001.patch > > > Running TestHdfsConfigFields throws error complaining missing key > dfs.client.failover.ipfailover.virtual-address. Since this config key is > specific to only ORFPPwithIP, This Jira moves this config prefix to > ObserverReadProxyProviderWithIPFailover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13873) ObserverNode should reject read requests when it is too far behind.
[
https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718354#comment-16718354
]
Konstantin Shvachko commented on HDFS-13873:
Slightly modified patch.
# Fixed exception handling. Thanks [~xkrogen] for debugging this.
# Used maxIdleTime from the Sever class rather than digging it out from
configuration in NNRPCServer.
# Fixed long lines in GSI contexts.
# {{TestHdfsConfigFields}} is failing because of HDFS-14017. [~vagarychen]
plans to fix it in the next jira. The otehr failure is passing locally.
We had an internal discussion. So my patch fixes the problem of a client
waiting too long for Observer to catch up. We looked at some remaining vectors
of attack by malicious clients.
* As Erik suggested, one can send requests to Observer with somewhat large
state id than it actually seen on ANN, but not large enough for Observer to
reject it out of the bat. This will increase load on Observer since such
malicious calls will stay longer in the queue than necessary. Although it
doesn't look too different for me from regular DDOS attacks by friendly
clients, which we observer on a daily bases, when they just send too many
{{getListings()}} to ANN.
* A variant of this attack can happen on a read-only (no writes) cluster. Then
those fake calls with slightly higher stateId can stay in the queue forever,
since stateId on the Observer is not progressing. Besides that I've never seen
a read-only HDFS cluster, I think there should be a general logic on RPC level
to clean up idle or about to timeout connections. If a client fails after
sending the request or the request was too long in the queue, there is no
reason for the server to execute such request since the client will never
receive the response. (And most probably already retried). Will be looking in
the code more, but if we don't have this now we should introduce it. This
should as well let Observer recover from such unfriendly attacks.
> ObserverNode should reject read requests when it is too far behind.
> ---
>
> Key: HDFS-13873
> URL: https://issues.apache.org/jira/browse/HDFS-13873
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client, namenode
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-13873-HDFS-12943.001.patch,
> HDFS-13873-HDFS-12943.002.patch
>
>
> Add a server-side threshold for ObserverNode to reject read requests when it
> is too far behind.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Patch Available (was: Open) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12348.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14142) Move ipfailover config key out of HdfsClientConfigKeys
[ https://issues.apache.org/jira/browse/HDFS-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14142: -- Attachment: HDFS-14142-HDFS-12943.001.patch > Move ipfailover config key out of HdfsClientConfigKeys > -- > > Key: HDFS-14142 > URL: https://issues.apache.org/jira/browse/HDFS-14142 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Minor > Attachments: HDFS-14142-HDFS-12943.001.patch > > > Running TestHdfsConfigFields throws error complaining missing key > dfs.client.failover.ipfailover.virtual-address. Since this config key is > specific to only ORFPPwithIP, This Jira moves this config prefix to > ObserverReadProxyProviderWithIPFailover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14142) Move ipfailover config key out of HdfsClientConfigKeys
[ https://issues.apache.org/jira/browse/HDFS-14142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14142: -- Status: Patch Available (was: Open) > Move ipfailover config key out of HdfsClientConfigKeys > -- > > Key: HDFS-14142 > URL: https://issues.apache.org/jira/browse/HDFS-14142 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Minor > Attachments: HDFS-14142-HDFS-12943.001.patch > > > Running TestHdfsConfigFields throws error complaining missing key > dfs.client.failover.ipfailover.virtual-address. Since this config key is > specific to only ORFPPwithIP, This Jira moves this config prefix to > ObserverReadProxyProviderWithIPFailover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14142) Move ipfailover config key out of HdfsClientConfigKeys
Chen Liang created HDFS-14142: - Summary: Move ipfailover config key out of HdfsClientConfigKeys Key: HDFS-14142 URL: https://issues.apache.org/jira/browse/HDFS-14142 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Chen Liang Assignee: Chen Liang Running TestHdfsConfigFields throws error complaining missing key dfs.client.failover.ipfailover.virtual-address. Since this config key is specific to only ORFPPwithIP, This Jira moves this config prefix to ObserverReadProxyProviderWithIPFailover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Open (was: Patch Available) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12348.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13873) ObserverNode should reject read requests when it is too far behind.
[ https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-13873: --- Attachment: HDFS-13873-HDFS-12943.002.patch > ObserverNode should reject read requests when it is too far behind. > --- > > Key: HDFS-13873 > URL: https://issues.apache.org/jira/browse/HDFS-13873 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Attachments: HDFS-13873-HDFS-12943.001.patch, > HDFS-13873-HDFS-12943.002.patch > > > Add a server-side threshold for ObserverNode to reject read requests when it > is too far behind. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Patch Available (was: Open) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12348.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: (was: HDFS-12348.000.patch) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718333#comment-16718333 ] Konstantin Shvachko commented on HDFS-14131: As an option, while we are on the feature branch we can do force commits. So if you want to recommit this we can. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14138) Description errors in the comparison logic of transaction ID
[
https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718331#comment-16718331
]
Hadoop QA commented on HDFS-14138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-14138 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951440/HDFS-12348.000.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25769/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Description errors in the comparison logic of transaction ID
>
>
> Key: HDFS-14138
> URL: https://issues.apache.org/jira/browse/HDFS-14138
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: xiangheng
>Priority: Minor
> Attachments: HDFS-12348.000.patch
>
>
> The call processing should be postponed until the client call's state id is
> aligned (<=) with the server state id,not >=.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Attachment: HDFS-12348.000.patch > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12348.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14138) Description errors in the comparison logic of transaction ID
[ https://issues.apache.org/jira/browse/HDFS-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiangheng updated HDFS-14138: - Status: Open (was: Patch Available) > Description errors in the comparison logic of transaction ID > > > Key: HDFS-14138 > URL: https://issues.apache.org/jira/browse/HDFS-14138 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-12943 >Reporter: xiangheng >Priority: Minor > Attachments: HDFS-12348.000.patch > > > The call processing should be postponed until the client call's state id is > aligned (<=) with the server state id,not >=. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14141) Enable GC monitoring thread for OIV tool
[ https://issues.apache.org/jira/browse/HDFS-14141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718273#comment-16718273 ] Wei-Chiu Chuang commented on HDFS-14141: JvmPauseMonitor outputs in log messages, and log messages are not guaranteed to be backward compatible. So I suggest make it the default (i.e. no extra flags) https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html bq. Note that any change to CLI tool output is considered an incompatible change, so between major versions, the CLI output will not change. Note that the CLI tool output is distinct from the log output produced by the CLI tools. Log output is not intended for automated consumption and may change at any time. > Enable GC monitoring thread for OIV tool > > > Key: HDFS-14141 > URL: https://issues.apache.org/jira/browse/HDFS-14141 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Priority: Minor > > It took OIV quite a long time (84 minutes) to dump a 14G FSImage. I suspect > there was a lot of GC going on. I think we could add an option like > '-m,--gc-monitor' to enable GC monitoring thread for OIV tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-916) MultipartUpload: Complete Multipart upload request
[ https://issues.apache.org/jira/browse/HDDS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-916: Issue Type: Sub-task (was: Bug) Parent: HDDS-763 > MultipartUpload: Complete Multipart upload request > -- > > Key: HDDS-916 > URL: https://issues.apache.org/jira/browse/HDDS-916 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This Jira is to support completeMultipartUpload request in ozone. > This request will stitch all the parts and add the entry into the keyTable, > to make it visible in ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14116) Fix a potential class cast error in ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718267#comment-16718267
]
Chao Sun commented on HDFS-14116:
-
To test this, you can set proxy provider class to be ObserverReadProxyProvider
and call {{createNonHAProxy}} to test. I'll add a test in the subsequent patch.
The reason for using {{createNonHAProxy}} is because, even if we fix the class
cast exception, it will still fail later when casting the result from
{{Proxy.newProxyInstance}} to the generic type {{T}}, which should extend
{{ClientProtocol}}, but it is not.
I'm thinking whether we should find a workaround for this (right now
{{ClientProtocol}} is required in {{ObserverReadProxyProvider}} so we can call
{{getHAServiceState}}), or we should just forbid initializing
{{ObserverReadProxyProvider}} with anything other than {{ClientProtocol}}.
> Fix a potential class cast error in ObserverReadProxyProvider
> -
>
> Key: HDFS-14116
> URL: https://issues.apache.org/jira/browse/HDFS-14116
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Chen Liang
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-14116-HDFS-12943.000.patch
>
>
> Currently in {{ObserverReadProxyProvider}} constructor there is this line
> {code}
> ((ClientHAProxyFactory) factory).setAlignmentContext(alignmentContext);
> {code}
> This could potentially cause failure, because it is possible that factory can
> not be casted here. Specifically,
> {{NameNodeProxiesClient.createFailoverProxyProvider}} is where the
> constructor will be called, and there are two paths that could call into this:
> (1).{{NameNodeProxies.createProxy}}
> (2).{{NameNodeProxiesClient.createFailoverProxyProvider}}
> (2) works fine because it always uses {{ClientHAProxyFactory}} but (1) uses
> {{NameNodeHAProxyFactory}} which can not be casted to
> {{ClientHAProxyFactory}}, this happens when, for example, running
> NNThroughputBenmarck. To fix this we can at least:
> 1. introduce setAlignmentContext to HAProxyFactory which is the parent of
> both ClientHAProxyFactory and NameNodeHAProxyFactory OR
> 2. only setAlignmentContext when it is ClientHAProxyFactory by, say, having a
> if check with reflection.
> Depending on whether it make sense to have alignment context for the case (1)
> calling code paths.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-916) MultipartUpload: Complete Multipart upload request
[ https://issues.apache.org/jira/browse/HDDS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-916 started by Bharat Viswanadham. --- > MultipartUpload: Complete Multipart upload request > -- > > Key: HDDS-916 > URL: https://issues.apache.org/jira/browse/HDDS-916 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This Jira is to support completeMultipartUpload request in ozone. > This request will stitch all the parts and add the entry into the keyTable, > to make it visible in ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-916) MultipartUpload: Complete Multipart upload request
Bharat Viswanadham created HDDS-916: --- Summary: MultipartUpload: Complete Multipart upload request Key: HDDS-916 URL: https://issues.apache.org/jira/browse/HDDS-916 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira is to support completeMultipartUpload request in ozone. This request will stitch all the parts and add the entry into the keyTable, to make it visible in ozone. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13617) Allow wrapping NN QOP into token in encrypted message
[
https://issues.apache.org/jira/browse/HDFS-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718258#comment-16718258
]
Hadoop QA commented on HDFS-13617:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 8s{color} | {color:orange} root: The patch generated 1 new + 760 unchanged
- 0 fixed = 761 total (was 760) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
46s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 44s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.token.TestDtUtilShell |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13617 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951423/HDFS-13617.005.patch |
| Optional Tests
[jira] [Created] (HDDS-915) Submit client request to OM Ratis server
Hanisha Koneru created HDDS-915: --- Summary: Submit client request to OM Ratis server Key: HDDS-915 URL: https://issues.apache.org/jira/browse/HDDS-915 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Hanisha Koneru Assignee: Hanisha Koneru After transforming a client request into Ratis request (HDDS-881), the request should be submitted to the ratis server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13965) hadoop.security.kerberos.ticket.cache.path setting is not honored when KMS encryption is enabled.
[
https://issues.apache.org/jira/browse/HDFS-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718204#comment-16718204
]
Wei-Chiu Chuang commented on HDFS-13965:
Thanks for reporting the issue, [~lokeskumarp].
It looks like part of the issue is that KerberosConfiguration class (which is
used by KMS client to authenticate) hard code the ticket cache path:
{code:java}
String ticketCache = System.getenv("KRB5CCNAME"); < this line
if (IBM_JAVA) {
USER_KERBEROS_OPTIONS.put("useDefaultCcache", "true");
} else {
USER_KERBEROS_OPTIONS.put("doNotPrompt", "true");
USER_KERBEROS_OPTIONS.put("useTicketCache", "true");
}
if (ticketCache != null) {
if (IBM_JAVA) {
// The first value searched when "useDefaultCcache" is used.
System.setProperty("KRB5CCNAME", ticketCache);
} else {
USER_KERBEROS_OPTIONS.put("ticketCache", ticketCache);
}
}{code}
So it always uses system variable {{$KRB5CCNAME}}. If we make it configurable,
just like in {{FileSystem#get(URI, Configuration, String)}}, this issue should
go away.
[~knanasi] how do you think? Thanks.
> hadoop.security.kerberos.ticket.cache.path setting is not honored when KMS
> encryption is enabled.
> -
>
> Key: HDFS-13965
> URL: https://issues.apache.org/jira/browse/HDFS-13965
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, kms
>Affects Versions: 2.7.3, 2.7.7
>Reporter: LOKESKUMAR VIJAYAKUMAR
>Assignee: Kitti Nanasi
>Priority: Major
>
> _We use the *+hadoop.security.kerberos.ticket.cache.path+* setting to provide
> a custom kerberos cache path for all hadoop operations to be run as specified
> user. But this setting is not honored when KMS encryption is enabled._
> _The below program to read a file works when KMS encryption is not enabled,
> but it fails when the KMS encryption is enabled._
> _Looks like *hadoop.security.kerberos.ticket.cache.path* setting is not
> honored by *createConnection on KMSClientProvider.java.*_
>
> HadoopTest.java (CLASSPATH needs to be set to compile and run)
>
> import java.io.InputStream;
> import java.net.URI;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.Path;
>
> public class HadoopTest {
> public static int runRead(String[] args) throws Exception{
> if (args.length < 3) {
> System.err.println("HadoopTest hadoop_file_path
> hadoop_user kerberos_cache");
> return 1;
> }
> Path inputPath = new Path(args[0]);
> Configuration conf = new Configuration();
> URI defaultURI = FileSystem.getDefaultUri(conf);
>
> conf.set("hadoop.security.kerberos.ticket.cache.path",args[2]);
> FileSystem fs =
> FileSystem.newInstance(defaultURI,conf,args[1]);
> InputStream is = fs.open(inputPath);
> byte[] buffer = new byte[4096];
> int nr = is.read(buffer);
> while (nr != -1)
> {
> System.out.write(buffer, 0, nr);
> nr = is.read(buffer);
> }
> return 0;
> }
> public static void main( String[] args ) throws Exception {
> int returnCode = HadoopTest.runRead(args);
> System.exit(returnCode);
> }
> }
>
>
>
> [root@lstrost3 testhadoop]# pwd
> /testhadoop
>
> [root@lstrost3 testhadoop]# ls
> HadoopTest.java
>
> [root@lstrost3 testhadoop]# export CLASSPATH=`hadoop classpath --glob`:.
>
> [root@lstrost3 testhadoop]# javac HadoopTest.java
>
> [root@lstrost3 testhadoop]# java HadoopTest
> HadoopTest hadoop_file_path hadoop_user kerberos_cache
>
> [root@lstrost3 testhadoop]# java HadoopTest /loki/loki.file loki
> /tmp/krb5cc_1006
> 18/09/27 23:23:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/09/27 23:23:21 WARN shortcircuit.DomainSocketFactory: The short-circuit
> local reads feature cannot be used because libhadoop cannot be loaded.
> Exception in thread "main" java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: *{color:#FF}No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt){color}*
> at
> {color:#FF}*org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:551)*{color}
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:831)
> at
>
[jira] [Created] (HDFS-14141) Enable GC monitoring thread for OIV tool
Siyao Meng created HDFS-14141: - Summary: Enable GC monitoring thread for OIV tool Key: HDFS-14141 URL: https://issues.apache.org/jira/browse/HDFS-14141 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.0.0 Reporter: Siyao Meng It took OIV quite a long time (84 minutes) to dump a 14G FSImage. I suspect there was a lot of GC going on. I think we could add an option like '-m,--gc-monitor' to enable GC monitoring thread for OIV tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
[ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718160#comment-16718160 ] Todd Lipcon commented on HDFS-14111: Thanks for the links to those other initiatives, Steve. I think, while they'd be useful improvements, it would be nice to also treat this as a libhdfs bug and see if we can make an internal change to fix the issue without requiring callers to move to a new API. It's worth noting that the issue here is with libhdfs rather than with HDFS itself, as far as I can understand from the code -- i.e. using the Java open() API _does not_ trigger an eager read from offset 0, but the libhdfs C equivalent _does_. > hdfsOpenFile on HDFS causes unnecessary IO from file offset 0 > - > > Key: HDFS-14111 > URL: https://issues.apache.org/jira/browse/HDFS-14111 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client, libhdfs >Affects Versions: 3.2.0 >Reporter: Todd Lipcon >Priority: Major > > hdfsOpenFile() calls readDirect() with a 0-length argument in order to check > whether the underlying stream supports bytebuffer reads. With DFSInputStream, > the read(0) isn't short circuited, and results in the DFSClient opening a > block reader. In the case of a remote block, the block reader will actually > issue a read of the whole block, causing the datanode to perform unnecessary > IO and network transfers in order to fill up the client's TCP buffers. This > causes performance degradation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718159#comment-16718159 ] Erik Krogen commented on HDFS-14131: Whoops, sorry [~elgoiri]! Trunk sounds good to me. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-914) Add Grafana support to docker containers
[
https://issues.apache.org/jira/browse/HDDS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718149#comment-16718149
]
Hadoop QA commented on HDDS-914:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 58s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
50s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-914 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951427/HDDS-914.001.patch |
| Optional Tests | asflicense shellcheck javac javadoc unit |
| uname | Linux 9721150298fb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / fb55e52 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| shellcheck | v0.4.6 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1915/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1915/testReport/ |
| Max. process+thread count | 1090 (vs. ulimit of 1) |
| modules | C: . hadoop-ozone/dist U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1915/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add Grafana support to docker containers
>
>
> Key: HDDS-914
> URL: https://issues.apache.org/jira/browse/HDDS-914
> Project: Hadoop Distributed Data Store
> Issue Type: New Feature
> Components: docker
>Reporter: Dinesh Chitlangia
>Assignee: Dinesh Chitlangia
>Priority: Major
> Attachments: HDDS-914.001.patch
>
>
> Since we are using Prometheus as a datasource to capture metrics from OM &
> SCM, it will be useful to have basic Grafana Dashboards made available once
> we start the docker cluster.
> This will be useful for investigation/analysis.
>
> This Jira proposes to have grafana + prometheus support for following docker
> containers under ~/compose:
> ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3
>
> This Jira will also add 2 simple ozone dashboards - Basic Object Creations
> stats, RPC Metrics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718125#comment-16718125 ] Íñigo Goiri commented on HDFS-14131: Let's wait for the merge vote to go in and then we can do it on trunk. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14092) Remove two-step create/append in WebHdfsFileSystem
[
https://issues.apache.org/jira/browse/HDFS-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng resolved HDFS-14092.
---
Resolution: Won't Do
> Remove two-step create/append in WebHdfsFileSystem
> --
>
> Key: HDFS-14092
> URL: https://issues.apache.org/jira/browse/HDFS-14092
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Siyao Meng
>Priority: Major
>
> Per the javadoc on the {{WebHdfsFileSystem.connect()}} method:
> {code}/**
> * Two-step requests redirected to a DN
> *
> * Create/Append:
> * Step 1) Submit a Http request with neither auto-redirect nor data.
> * Step 2) Submit another Http request with the URL from the Location
> header
> * with data.
> *
> * The reason of having two-step create/append is for preventing clients
> to
> * send out the data before the redirect. This issue is addressed by the
> * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
> * Unfortunately, there are software library bugs (e.g. Jetty 6 http
> server
> * and Java 6 http client), which do not correctly implement "Expect:
> * 100-continue". The two-step create/append is a temporary workaround for
> * the software library bugs.
> *
> * Open/Checksum
> * Also implements two-step connects for other operations redirected to
> * a DN such as open and checksum
> */{code}
> We should validate that it's safe to remove the two-step process and do so.
> FYI, [~smeng].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14092) Remove two-step create/append in WebHdfsFileSystem
[
https://issues.apache.org/jira/browse/HDFS-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718127#comment-16718127
]
Siyao Meng commented on HDFS-14092:
---
I'll resolve this jira as "Won't Do" for now. Reasons:
1. Using library to redirect this for us might not actually increase the
performance since the library might still create two connections since it would
access two servers.
2. When this comment is first written in HDFS-6305, the code is simpler and
"Location" is not saved in a global variable, which is now referenced in 5
other places. If we let library auto redirect request to DNs, we need to figure
out another way to get the redirected address.
(2) shouldn't be hard to do, but (1) is the main reason in my mind that makes
this jira not worth fixing at the moment.
CWIIW. Feel free to reopen this jira.
> Remove two-step create/append in WebHdfsFileSystem
> --
>
> Key: HDFS-14092
> URL: https://issues.apache.org/jira/browse/HDFS-14092
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.2.0
>Reporter: Daniel Templeton
>Assignee: Siyao Meng
>Priority: Major
>
> Per the javadoc on the {{WebHdfsFileSystem.connect()}} method:
> {code}/**
> * Two-step requests redirected to a DN
> *
> * Create/Append:
> * Step 1) Submit a Http request with neither auto-redirect nor data.
> * Step 2) Submit another Http request with the URL from the Location
> header
> * with data.
> *
> * The reason of having two-step create/append is for preventing clients
> to
> * send out the data before the redirect. This issue is addressed by the
> * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
> * Unfortunately, there are software library bugs (e.g. Jetty 6 http
> server
> * and Java 6 http client), which do not correctly implement "Expect:
> * 100-continue". The two-step create/append is a temporary workaround for
> * the software library bugs.
> *
> * Open/Checksum
> * Also implements two-step connects for other operations redirected to
> * a DN such as open and checksum
> */{code}
> We should validate that it's safe to remove the two-step process and do so.
> FYI, [~smeng].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718124#comment-16718124 ] Chao Sun commented on HDFS-14131: - Thanks for the comments [~elgoiri]. Since the patch is already committed, could you create a follow-up JIRA for this? I agree an updated diagram for HDFS architecture will be helpful. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13873) ObserverNode should reject read requests when it is too far behind.
[
https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718115#comment-16718115
]
Hadoop QA commented on HDFS-13873:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
5s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
11s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
48s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} HDFS-12943 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 17s{color} | {color:orange} root: The patch generated 1 new + 272 unchanged
- 0 fixed = 273 total (was 272) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}196m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13873 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951405/HDFS-13873-HDFS-12943.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 68c2b5085f68 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool
[jira] [Updated] (HDDS-914) Add Grafana support to docker containers
[ https://issues.apache.org/jira/browse/HDDS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-914: --- Attachment: HDDS-914.001.patch Status: Patch Available (was: In Progress) > Add Grafana support to docker containers > > > Key: HDDS-914 > URL: https://issues.apache.org/jira/browse/HDDS-914 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: docker >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > Attachments: HDDS-914.001.patch > > > Since we are using Prometheus as a datasource to capture metrics from OM & > SCM, it will be useful to have basic Grafana Dashboards made available once > we start the docker cluster. > This will be useful for investigation/analysis. > > This Jira proposes to have grafana + prometheus support for following docker > containers under ~/compose: > ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 > > This Jira will also add 2 simple ozone dashboards - Basic Object Creations > stats, RPC Metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDDS-914) Add Grafana support to docker containers
[ https://issues.apache.org/jira/browse/HDDS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDDS-914 started by Dinesh Chitlangia. -- > Add Grafana support to docker containers > > > Key: HDDS-914 > URL: https://issues.apache.org/jira/browse/HDDS-914 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: docker >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > Since we are using Prometheus as a datasource to capture metrics from OM & > SCM, it will be useful to have basic Grafana Dashboards made available once > we start the docker cluster. > This will be useful for investigation/analysis. > > This Jira proposes to have grafana + prometheus support for following docker > containers under ~/compose: > ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 > > This Jira will also add 2 simple ozone dashboards - Basic Object Creations > stats, RPC Metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718006#comment-16718006
]
Hadoop QA commented on HDFS-14084:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 39s{color}
| {color:red} root generated 1 new + 1490 unchanged - 0 fixed = 1491 total (was
1490) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 4s{color} | {color:orange} root: The patch generated 17 new + 133 unchanged
- 1 fixed = 150 total (was 134) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.server.namenode.sps.TestBlockStorageMovementAttemptedItems |
| | hadoop.hdfs.server.namenode.snapshot.TestXAttrWithSnapshot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14084 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951394/HDFS-14084.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f54efbf0554c
[jira] [Updated] (HDFS-13617) Allow wrapping NN QOP into token in encrypted message
[ https://issues.apache.org/jira/browse/HDFS-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-13617: -- Attachment: HDFS-13617.005.patch > Allow wrapping NN QOP into token in encrypted message > - > > Key: HDFS-13617 > URL: https://issues.apache.org/jira/browse/HDFS-13617 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13617.001.patch, HDFS-13617.002.patch, > HDFS-13617.003.patch, HDFS-13617.004.patch, HDFS-13617.005.patch > > > This Jira allows NN to configurably wrap the QOP it has established with the > client into the token message sent back to the client. The QOP is sent back > in encrypted message, using BlockAccessToken encryption key as the key. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-914) Add Grafana support to docker containers
[ https://issues.apache.org/jira/browse/HDDS-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDDS-914: --- Description: Since we are using Prometheus as a datasource to capture metrics from OM & SCM, it will be useful to have basic Grafana Dashboards made available once we start the docker cluster. This will be useful for investigation/analysis. This Jira proposes to have grafana + prometheus support for following docker containers under ~/compose: ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 This Jira will also add 2 simple ozone dashboards - Basic Object Creations stats, RPC Metrics. was: Since we are using Prometheus as a datasource to capture metrics from OM & SCM, it will be useful to have basic Grafana Dashboards made available once we start the docker cluster. This will be useful for investigation/analysis. This Jira proposes to have grafana + prometheus support for following docker containers under ~/compose: ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 > Add Grafana support to docker containers > > > Key: HDDS-914 > URL: https://issues.apache.org/jira/browse/HDDS-914 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: docker >Reporter: Dinesh Chitlangia >Assignee: Dinesh Chitlangia >Priority: Major > > Since we are using Prometheus as a datasource to capture metrics from OM & > SCM, it will be useful to have basic Grafana Dashboards made available once > we start the docker cluster. > This will be useful for investigation/analysis. > > This Jira proposes to have grafana + prometheus support for following docker > containers under ~/compose: > ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 > > This Jira will also add 2 simple ozone dashboards - Basic Object Creations > stats, RPC Metrics. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13617) Allow wrapping NN QOP into token in encrypted message
[ https://issues.apache.org/jira/browse/HDFS-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718001#comment-16718001 ] Chen Liang commented on HDFS-13617: --- Fix the checkstyle, javac, findbug warnings in v005 patch. > Allow wrapping NN QOP into token in encrypted message > - > > Key: HDFS-13617 > URL: https://issues.apache.org/jira/browse/HDFS-13617 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13617.001.patch, HDFS-13617.002.patch, > HDFS-13617.003.patch, HDFS-13617.004.patch, HDFS-13617.005.patch > > > This Jira allows NN to configurably wrap the QOP it has established with the > client into the token message sent back to the client. The QOP is sent back > in encrypted message, using BlockAccessToken encryption key as the key. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-914) Add Grafana support to docker containers
Dinesh Chitlangia created HDDS-914: -- Summary: Add Grafana support to docker containers Key: HDDS-914 URL: https://issues.apache.org/jira/browse/HDDS-914 Project: Hadoop Distributed Data Store Issue Type: New Feature Components: docker Reporter: Dinesh Chitlangia Assignee: Dinesh Chitlangia Since we are using Prometheus as a datasource to capture metrics from OM & SCM, it will be useful to have basic Grafana Dashboards made available once we start the docker cluster. This will be useful for investigation/analysis. This Jira proposes to have grafana + prometheus support for following docker containers under ~/compose: ozone, ozone-hdfs, ozonefs, ozoneperf, ozones3 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717986#comment-16717986
]
Íñigo Goiri commented on HDFS-14084:
We should do:
{code}
LOG.debug(" RPC Client stats: {} " + rb.toString());
{code}
Instead of:
{code}
LOG.debug(" RPC Client stats: {}", rb);
{code}
Is there a way to check the output of the logging in the unit test?
It looks like GenericTestUtils has some utils like {{LogCapturer}}.
> Need for more stats in DFSClient
>
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch,
> HDFS-14084.003.patch, HDFS-14084.004.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now
> it's becoming more of like a general purpose filesystem. In most of the cases
> there are issues with the Namenode so we have metrics to know the workload or
> stress on Namenode.
> However, there is a need to have more statistics collected for different
> operations/RPCs in DFSClient to know which RPC operations are taking longer
> time or to know what is the frequency of the operation.These statistics can
> be exposed to the users of DFS Client and they can periodically log or do
> some sort of flow control if the response is slow. This will also help to
> isolate HDFS issue in a mixed environment where on a node say we have Spark,
> HBase and Impala running together. We can check the throughput of different
> operation across client and isolate the problem caused because of noisy
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no
> conclusive evidence as to what caused the problem. If we had metrics or stats
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
> HADOOP-15538 HADOOP-15530 ( client side deadlock)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14084) Need for more stats in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717971#comment-16717971
]
Hadoop QA commented on HDFS-14084:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 52s{color} | {color:orange} root: The patch generated 5 new + 133 unchanged
- 1 fixed = 138 total (was 134) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
15s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.TestDecommission |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14084 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951388/HDFS-14084.004.patch |
| Optional Tests | dupname asflicense compile
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717944#comment-16717944 ] Íñigo Goiri commented on HDFS-14131: A couple comments: * Can we add links for the JIRAs referenced in the document (right now, they are plain text). * Should we add pointers to this page from places like HDFSHighAvailabilityWithQJM.md? At some point we may want to update the HDFS architecture diagram to include the observer and so on. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-14131: --- Fix Version/s: HDFS-12943 > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-14131: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Fix For: HDFS-12943 > > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[
https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717937#comment-16717937
]
Erik Krogen commented on HDFS-14131:
Just committed this to branch {{HDFS-12943}}. Thanks [~csun]!
> Create user guide for "Consistent reads from Observer" feature.
> ---
>
> Key: HDFS-14131
> URL: https://issues.apache.org/jira/browse/HDFS-14131
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-14131-HDFS-12943.000.patch,
> HDFS-14131-HDFS-12943.001.patch
>
>
> The documentation should give an overview of the feature, explain
> configuration parameters, startup procedure, give an example of recommended
> deployment.
> It should include the description of Fast Edits Tailing HDFS-13150, as this
> is required for efficient reads from Observer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[
https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717918#comment-16717918
]
Hadoop QA commented on HDFS-14131:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-12943 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
21s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} HDFS-12943 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14131 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951408/HDFS-14131-HDFS-12943.001.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 4053e2ad0e2c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12943 / 47d7260 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25767/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Create user guide for "Consistent reads from Observer" feature.
> ---
>
> Key: HDFS-14131
> URL: https://issues.apache.org/jira/browse/HDFS-14131
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
>Affects Versions: HDFS-12943
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-14131-HDFS-12943.000.patch,
> HDFS-14131-HDFS-12943.001.patch
>
>
> The documentation should give an overview of the feature, explain
> configuration parameters, startup procedure, give an example of recommended
> deployment.
> It should include the description of Fast Edits Tailing HDFS-13150, as this
> is required for efficient reads from Observer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-913) Ozonefs defaultFs example is wrong in the documentation
[
https://issues.apache.org/jira/browse/HDDS-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-913:
--
Description:
[https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html]
According to this doc the defaultfs should be
{{o3fs://localhost:9864/volume/bucket}}{{}}
IMHO it should be o3fs://bucket.volume
Can be checked with compose/ozonefs cluster definition in the distribution
package.
> Ozonefs defaultFs example is wrong in the documentation
> ---
>
> Key: HDDS-913
> URL: https://issues.apache.org/jira/browse/HDDS-913
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Reporter: Elek, Marton
>Priority: Major
> Labels: newbie
>
> [https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html]
> According to this doc the defaultfs should be
> {{o3fs://localhost:9864/volume/bucket}}{{}}
> IMHO it should be o3fs://bucket.volume
> Can be checked with compose/ozonefs cluster definition in the distribution
> package.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-913) Ozonefs defaultFs example is wrong in the documentation
[
https://issues.apache.org/jira/browse/HDDS-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-913:
--
Environment: (was:
[https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html]
According to this doc the defaultfs should be
{{o3fs://localhost:9864/volume/bucket}}{{}}
IMHO it should be o3fs://bucket.volume
Can be checked with compose/ozonefs cluster definition in the distribution
package.)
> Ozonefs defaultFs example is wrong in the documentation
> ---
>
> Key: HDDS-913
> URL: https://issues.apache.org/jira/browse/HDDS-913
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Reporter: Elek, Marton
>Priority: Major
> Labels: newbie
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717833#comment-16717833 ] Chao Sun edited comment on HDFS-14131 at 12/11/18 7:34 PM: --- Thanks everyone! the comments are really useful. I updated the patch based on the comments. [Rendered|https://github.com/sunchao/hadoop/blob/HDFS-14131/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ObserverNameNode.md]. was (Author: csun): Thanks everyone! the comments are really useful. I updated the patch based on the comments. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14137) TestMaintenanceState fails with ArrayIndexOutOfBound Exception
[ https://issues.apache.org/jira/browse/HDFS-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717868#comment-16717868 ] Hudson commented on HDFS-14137: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15592 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15592/]) HDFS-14137. TestMaintenanceState fails with ArrayIndexOutOfBound (gifuma: rev 60af851e59bf29efc5b2e132c8b9aaa11b29e672) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMaintenanceState.java > TestMaintenanceState fails with ArrayIndexOutOfBound Exception > -- > > Key: HDFS-14137 > URL: https://issues.apache.org/jira/browse/HDFS-14137 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14137-01.patch > > > Reference : > https://builds.apache.org/job/PreCommit-HDFS-Build/25727/testReport/org.apache.hadoop.hdfs/TestMaintenanceState/testMultipleNodesMaintenance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717864#comment-16717864 ] Erik Krogen commented on HDFS-14131: +1 LGTM > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Attachment: (was: HDFS-14131-HDFS-12943.001.patch) > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13873) ObserverNode should reject read requests when it is too far behind.
[ https://issues.apache.org/jira/browse/HDFS-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned HDFS-13873: --- Assignee: Konstantin Shvachko (was: Chao Sun) > ObserverNode should reject read requests when it is too far behind. > --- > > Key: HDFS-13873 > URL: https://issues.apache.org/jira/browse/HDFS-13873 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client, namenode >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko >Priority: Major > Attachments: HDFS-13873-HDFS-12943.001.patch > > > Add a server-side threshold for ObserverNode to reject read requests when it > is too far behind. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-14131: Attachment: HDFS-14131-HDFS-12943.001.patch > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14137) TestMaintenanceState fails with ArrayIndexOutOfBound Exception
[ https://issues.apache.org/jira/browse/HDFS-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HDFS-14137: Fix Version/s: 3.3.0 > TestMaintenanceState fails with ArrayIndexOutOfBound Exception > -- > > Key: HDFS-14137 > URL: https://issues.apache.org/jira/browse/HDFS-14137 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14137-01.patch > > > Reference : > https://builds.apache.org/job/PreCommit-HDFS-Build/25727/testReport/org.apache.hadoop.hdfs/TestMaintenanceState/testMultipleNodesMaintenance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14137) TestMaintenanceState fails with ArrayIndexOutOfBound Exception
[ https://issues.apache.org/jira/browse/HDFS-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717834#comment-16717834 ] Giovanni Matteo Fumarola commented on HDFS-14137: - Thanks [~elgoiri] for the review and [~ayushtkn] for the patch. Committed to trunk. > TestMaintenanceState fails with ArrayIndexOutOfBound Exception > -- > > Key: HDFS-14137 > URL: https://issues.apache.org/jira/browse/HDFS-14137 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14137-01.patch > > > Reference : > https://builds.apache.org/job/PreCommit-HDFS-Build/25727/testReport/org.apache.hadoop.hdfs/TestMaintenanceState/testMultipleNodesMaintenance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14131) Create user guide for "Consistent reads from Observer" feature.
[ https://issues.apache.org/jira/browse/HDFS-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717833#comment-16717833 ] Chao Sun commented on HDFS-14131: - Thanks everyone! the comments are really useful. I updated the patch based on the comments. > Create user guide for "Consistent reads from Observer" feature. > --- > > Key: HDFS-14131 > URL: https://issues.apache.org/jira/browse/HDFS-14131 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-14131-HDFS-12943.000.patch, > HDFS-14131-HDFS-12943.001.patch > > > The documentation should give an overview of the feature, explain > configuration parameters, startup procedure, give an example of recommended > deployment. > It should include the description of Fast Edits Tailing HDFS-13150, as this > is required for efficient reads from Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14137) TestMaintenanceState fails with ArrayIndexOutOfBound Exception
[ https://issues.apache.org/jira/browse/HDFS-14137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HDFS-14137: Resolution: Fixed Status: Resolved (was: Patch Available) > TestMaintenanceState fails with ArrayIndexOutOfBound Exception > -- > > Key: HDFS-14137 > URL: https://issues.apache.org/jira/browse/HDFS-14137 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14137-01.patch > > > Reference : > https://builds.apache.org/job/PreCommit-HDFS-Build/25727/testReport/org.apache.hadoop.hdfs/TestMaintenanceState/testMultipleNodesMaintenance/ -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
