[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426287#comment-16426287
]
Erik Krogen edited comment on HDFS-13331 at 4/4/18 10:46 PM:
-
Hey [~zero45], looks great. I just committed this to branch {{HDFS-12943}}.
Thanks for the work! Can you please make sure to file a follow-on JIRA for the
static {{AlignmentContext}} issues we discussed?
was (Author: xkrogen):
Hey Plamen, looks great. I just committed this to branch {{HDFS-12943}}. Thanks
for the work! Can you please make sure to file a follow-on JIRA for the static
{{AlignmentContext}} issues we discussed?
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331-HDFS-12943.004.patch,
> HDFS-13331-HDFS-12943.005.patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16422883#comment-16422883
]
Plamen Jeliazkov edited comment on HDFS-13331 at 4/2/18 6:00 PM:
-
Thanks [~xkrogen] -- I debated whether to do the static field work now or in
follow-up for a while and I think in follow-up will help make my patch more
focused. I think I am able to do it by passing it through {{Call}} object
though.
As for your points, I agree with them except that for (1) I do not intend the
log statement to last long and therefore would opt not to rely on a log
capture. Log capture tests are a little hacky as well. If you would really like
me to remove the {{@Ignored}} however I can just remove it for now and update
the test later.
I should have a patch by EOD today.
was (Author: zero45):
Thanks [~xkrogen] -- I debated whether to do the static field work now or in
follow-up for a while and I think in follow-up will help make my patch more
focused. I think I am able to do it by passing it through {{Call}} object
though.
As for your points, I agree with them except that for (1) I do not intend the
log statement to last long and therefore would opt not to rely on a log
capture. Log capture tests are a little hacky as well. If you would really like
me to remove the {{@Ignored}} however I can just remove it for now and update
the test later.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420913#comment-16420913
]
Plamen Jeliazkov edited comment on HDFS-13331 at 3/30/18 8:42 PM:
--
I agreed we should try to leverage {{Connection}} objects and that ThreadLocal
would be hacky. However while experimenting with the {{Connection}} approach
but I hit an issue in that DFSClients re-use {{Connections}}, but with multiple
DFSClients having their own {{AlignmentContext}} then only the first
instantiated {{Connection}} will have its {{AlignmentContext}} updated.
I am thinking now that it needs to live in the {{Call}} object instead... I
will continue experimenting however and report next week.
Also, no need to apologize. We are on the same page / team here. :) In fact its
probably more fitting to address this as part of this JIRA then the last anyway
as now its easier to focus on the client side logic.
was (Author: zero45):
I agree we should try to leverage {{Connection}} objects and that ThreadLocal
would be hacky. I've taken the {{Connection}} approach but I am currently
hitting an issue in that DFSClients re-use {{Connections}}, but with multiple
DFSClients having their own {{AlignmentContext}} then only the first
instantiated {{Connection}} will have its {{AlignmentContext}} updated.
I am thinking now that it needs to live in the {{Call}} object instead... I
will continue experimenting however and report next week.
Also, no need to apologize. We are on the same page / team here. :) In fact its
probably more fitting to address this as part of this JIRA then the last anyway
as now its easier to focus on the client side logic.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420913#comment-16420913
]
Plamen Jeliazkov edited comment on HDFS-13331 at 3/30/18 8:41 PM:
--
I agree we should try to leverage {{Connection}} objects and that ThreadLocal
would be hacky. I've taken the {{Connection}} approach but I am currently
hitting an issue in that DFSClients re-use {{Connections}}, but with multiple
DFSClients having their own {{AlignmentContext}} then only the first
instantiated {{Connection}} will have its {{AlignmentContext}} updated.
I am thinking now that it needs to live in the {{Call}} object instead... I
will continue experimenting however and report next week.
Also, no need to apologize. We are on the same page / team here. :) In fact its
probably more fitting to address this as part of this JIRA then the last anyway
as now its easier to focus on the client side logic.
was (Author: zero45):
I agree we should try to leverage {{Connection}} objects and that ThreadLocal
would be hacky. I've taken the {{Connection}} approach but I am currently
hitting an issue in that DFSClients re-use {{Connection}}s, but with multiple
DFSClients having their own {{AlignmentContext}} then only the first
instantiated {{Connection}} will have its {{AlignmentContext}} updated.
I am thinking now that it needs to live in the {{Call}} object instead... I
will continue experimenting however and report next week.
Also, no need to apologize. We are on the same page / team here. :) In fact its
probably more fitting to address this as part of this JIRA then the last anyway
as now its easier to focus on the client side logic.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419948#comment-16419948
]
Erik Krogen edited comment on HDFS-13331 at 3/29/18 11:48 PM:
--
One way to achieve it would be to make {{alignmentContext}} a ThreadLocal in
{{Client}}. This matches well with how other state is maintained within
{{Client}}. Then, before calling into the {{ClientProtocol}}, {{DFSClient}} can
set the proper {{alignmentContext}}. It can reset it after the call completes.
We could facilitate this by bundling it into a {{Closable}} so that it could be
done in a try-with-resource statement like:
{code:java}
try (Client.setAlignmentContext(alignmentContext)) {
namenode.doWhateverCall(...)
}
{code}
Where {{setAlignmentContext}} returns a {{Closable}} whose {{close}} method
sets the alignment context back to null (or whatever it was previously). This
requires the least changes outside of DFSClient, but feels kind of hacky.
Given the {{Client}} already maintains a map of {{ConnectionId}} ->
{{Connection}} mapping, it seems we could leverage this to achieve a cleaner
solution. It seems that really the place this reference should be stored is
within {{Connection}}, or maybe {{ConnectionId}} but that seems a little less
clean. The {{Connection}} objects are created via {{Client#getConnection}},
which is called by {{Client#call}} -> {{ProtobufRpcEngine#invoke}}, so we could
have the {{DFSClient}} pass in its {{alignmentContext}} when it creates its
proxy. Any thoughts on this approach? Sorry for not chiming in on the last
JIRA, btw.
was (Author: xkrogen):
One way to achieve it would be to make {{alignmentContext}} a ThreadLocal in
{{Client}}. This matches well with how other state is maintained within
{{Client}}. Then, before calling into the {{ClientProtocol}}, {{DFSClient}} can
set the proper {{alignmentContext}}. It can reset it after the call completes.
We could facilitate this by bundling it into a {{Closable}} so that it could be
done in a try-with-resource statement like:
{code:java}
try (Client.setAlignmentContext(alignmentContext)) {
namenode.doWhateverCall(...)
}
{code}
Where {{setAlignmentContext}} returns a {{Closable}} whose {{close}} method
sets the alignment context back to null (or whatever it was previously). This
requires the least changes outside of DFSClient, but feels kind of hacky.
Given the {{Client}} already maintains a map of {{ConnectionId}} ->
{{Connection}} mapping, it seems we could leverage this to achieve a cleaner
solution. It seems that really the place this reference should be stored is
within {{Connection}}, or maybe {{ConnectionId}} but that seems a little less
clean. The {{Connection}}s are created via {{Client#getConnection}}, which is
called by {{Client#call}} -> {{ProtobufRpcEngine#invoke}}, so we could have the
{{DFSClient}} pass in its {{alignmentContext}} when it creates its proxy. Any
thoughts on this approach? Sorry for not chiming in on the last JIRA, btw.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[
https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419948#comment-16419948
]
Erik Krogen edited comment on HDFS-13331 at 3/29/18 11:47 PM:
--
One way to achieve it would be to make {{alignmentContext}} a ThreadLocal in
{{Client}}. This matches well with how other state is maintained within
{{Client}}. Then, before calling into the {{ClientProtocol}}, {{DFSClient}} can
set the proper {{alignmentContext}}. It can reset it after the call completes.
We could facilitate this by bundling it into a {{Closable}} so that it could be
done in a try-with-resource statement like:
{code:java}
try (Client.setAlignmentContext(alignmentContext)) {
namenode.doWhateverCall(...)
}
{code}
Where {{setAlignmentContext}} returns a {{Closable}} whose {{close}} method
sets the alignment context back to null (or whatever it was previously). This
requires the least changes outside of DFSClient, but feels kind of hacky.
Given the {{Client}} already maintains a map of {{ConnectionId}} ->
{{Connection}} mapping, it seems we could leverage this to achieve a cleaner
solution. It seems that really the place this reference should be stored is
within {{Connection}}, or maybe {{ConnectionId}} but that seems a little less
clean. The {{Connection}}s are created via {{Client#getConnection}}, which is
called by {{Client#call}} -> {{ProtobufRpcEngine#invoke}}, so we could have the
{{DFSClient}} pass in its {{alignmentContext}} when it creates its proxy. Any
thoughts on this approach? Sorry for not chiming in on the last JIRA, btw.
was (Author: xkrogen):
One way to achieve it would be to make {{alignmentContext}} a ThreadLocal in
{{Client}}. This matches well with how other state is maintained within
{{Client}}. Then, before calling into the {{ClientProtocol}}, {{DFSClient}} can
set the proper {{alignmentContext}}. It can (optionally?) reset it after the
call completes. We could facilitate this by bundling it into a {{Closable}} so
that it could be done in a try-with-resource statement like:
{code:java}
try (Client.setAlignmentContext(alignmentContext)) {
namenode.doWhateverCall(...)
}
{code}
This requires the least changes outside of DFSClient, but feels kind of hacky.
Given the {{Client}} already maintains a map of {{ConnectionId}} ->
{{Connection}} mapping, it seems we could leverage this to achieve a cleaner
solution. It seems that really the place this reference should be stored is
within {{Connection}}, or maybe {{ConnectionId}} but that seems a little less
clean. The {{Connection}}s are created via {{Client#getConnection}}, which is
called by {{Client#call}} -> {{ProtobufRpcEngine#invoke}}, so we could have the
{{DFSClient}} pass in its {{alignmentContext}} when it creates its proxy. Any
thoughts on this approach? Sorry for not chiming in on the last JIRA, btw.
> Add lastSeenStateId to RpcRequestHeader.
>
>
> Key: HDFS-13331
> URL: https://issues.apache.org/jira/browse/HDFS-13331
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-12943
>Reporter: Plamen Jeliazkov
>Assignee: Plamen Jeliazkov
>Priority: Major
> Attachments: HDFS-13331-HDFS-12943.002.patch,
> HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch,
> HDFS_13331.trunk.000.patch
>
>
> HDFS-12977 added a stateId into the RpcResponseHeader which is returned by
> NameNode and stored by DFSClient.
> This JIRA is to followup on that work and have the DFSClient send their
> stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can
> then compare with their own and act accordingly.
> This JIRA work focuses on just the part of making DFSClient send their state
> through RpcRequestHeader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
[ https://issues.apache.org/jira/browse/HDFS-13331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419704#comment-16419704 ] Plamen Jeliazkov edited comment on HDFS-13331 at 3/29/18 8:26 PM: -- Hi [~xkrogen], yes, you are correct. I brought up that concern in the last patch as well stating that I was not happy with the use of a static method in Client as a way to handle alignmentContext, but it seems that point was glossed over. I mentioned this: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16372125&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16372125 And here: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16386960&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16386960 My issue was that I had difficulty finding a way to pass in alignmentContext into Client constructor. I am happy to try again though. was (Author: zero45): Yes you are correct. I brought up that concern in the last patch as well stating that I was not happy with the use of a static method in Client as a way to handle alignmentContext, but it seems that point was glossed over. I mentioned this: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16372125&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16372125 And here: https://issues.apache.org/jira/browse/HDFS-12977?focusedCommentId=16386960&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16386960 My issue was that I had difficulty finding a way to pass in alignmentContext into Client constructor. I am happy to try again though. > Add lastSeenStateId to RpcRequestHeader. > > > Key: HDFS-13331 > URL: https://issues.apache.org/jira/browse/HDFS-13331 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-12943 >Reporter: Plamen Jeliazkov >Assignee: Plamen Jeliazkov >Priority: Major > Attachments: HDFS-13331-HDFS-12943.002.patch, > HDFS-13331-HDFS-12943.003..patch, HDFS-13331.trunk.001.patch, > HDFS_13331.trunk.000.patch > > > HDFS-12977 added a stateId into the RpcResponseHeader which is returned by > NameNode and stored by DFSClient. > This JIRA is to followup on that work and have the DFSClient send their > stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can > then compare with their own and act accordingly. > This JIRA work focuses on just the part of making DFSClient send their state > through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
