[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257650#comment-17257650
]
Xiaoqiao He commented on HDFS-15751:
Thanks [~csun], will commit to active branches shortly if no other comments. cc
[~shv],[~ste...@apache.org]
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch,
> HDFS-15751-03.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257429#comment-17257429
]
Chao Sun commented on HDFS-15751:
-
+1 on patch v3. Thanks again [~hexiaoqiao] and [~shv]!
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch,
> HDFS-15751-03.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15748) RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf.
[ https://issues.apache.org/jira/browse/HDFS-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257392#comment-17257392 ] Ayush Saxena commented on HDFS-15748: - Thanx [~LiJinglun] for the update. Changes Looks fair enough, The router side stuffs are quite stable, For the others, if super-user doesn't pitch in or permissions stuff doesn't bother. That things I don't have a solution too, So, we can handle later. So, I am +1 on the present changes, Once in we can then try the rename stuff at router. :) [~elgoiri] any comments? > RBF: Move the router related part from hadoop-federation-balance module to > hadoop-hdfs-rbf. > --- > > Key: HDFS-15748 > URL: https://issues.apache.org/jira/browse/HDFS-15748 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-15748.001.patch, HDFS-15748.002.patch, > HDFS-15748.003.patch, HDFS-15748.004.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257365#comment-17257365
]
Hadoop QA commented on HDFS-15751:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 44s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 45s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 31s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/397/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15751 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13018032/HDFS-15751-03.patch |
| Optional Tests | dupname asflicense mvnsite markdownlint |
| uname | Linux 7859081e4369 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 1cd96e8dd81 |
| Max. process+thread count | 535 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/397/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch,
> HDFS-15751-03.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257350#comment-17257350
]
Xiaoqiao He commented on HDFS-15751:
Thanks [~csun] for your comments. v03 has updated it, Please take another
reviews. Thanks.
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch,
> HDFS-15751-03.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoqiao He updated HDFS-15751:
---
Attachment: HDFS-15751-03.patch
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch,
> HDFS-15751-03.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15751) Add documentation for msync() API to filesystem.md
[
https://issues.apache.org/jira/browse/HDFS-15751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257335#comment-17257335
]
Chao Sun commented on HDFS-15751:
-
Thanks [~hexiaoqiao]! IMO the preconditions section should be specific to what
conditions should be met prior to calling the method, and we should move the
sentence:
{quote}
It is currently only implemented for HDFS and others will just throw
UnsupportedOperationException.
{quote}
to the previous section before preconditions (also following other methods such
as concat).
> Add documentation for msync() API to filesystem.md
> --
>
> Key: HDFS-15751
> URL: https://issues.apache.org/jira/browse/HDFS-15751
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-15751-01.patch, HDFS-15751-02.patch
>
>
> HDFS-15567 introduced new {{FileSystem}} call {{msync()}}. Should add it to
> the API definitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15756) RBF: Cannot get updated delegation token from zookeeper
[
https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17257324#comment-17257324
]
Xiaoqiao He commented on HDFS-15756:
Thanks [~hbprotoss] for your report, I believe it is possible that there are
data gap between different nodes for a short period of time when using
zookeeper cluster to store token for routers. Reference [zookeeper Consistency
Guarantees|https://zookeeper.apache.org/doc/r3.6.2/zookeeperProgrammers.html#ch_zkGuarantees].
This case appear very rarely in my practice.(Sorry we do not involve
Spark2.4). I agree that we should enrich the implementation of
{{AbstractDelegationTokenSecretManager}} using RDBMS/KV or others.
Just out of interest, is there any impact when RM renew token failed when
receive application request? IIUC, it could be ignore because after 20h (by
default) before token expire RM will trigger another renew request, then token
must synced and the renew operation will be successful. (Of course, some corner
case could meet if tune token parameter)
For the solution, I do not think it is good to throw StandbyException to client
from Router. Considering some one request to renew a invalid/expired token,
which is neither at memory nor at zookeeper store, then it will be failover for
many times by default is it necessary? Thanks, welcome more comments.
cc [~fengnanli],[~csun]
> RBF: Cannot get updated delegation token from zookeeper
> ---
>
> Key: HDFS-15756
> URL: https://issues.apache.org/jira/browse/HDFS-15756
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.0.0
>Reporter: hbprotoss
>Priority: Major
>
> Affected version: all version with rbf
> When RBF work with spark 2.4 client mode, there will be a chance that token
> is missing across different nodes in RBF cluster. The root cause is that
> spark renew the token(via resource manager) immediately after got one, as
> zookeeper don't have a strong consistency guarantee after an update in
> cluster, zookeeper client may read a stale value in some followers not synced
> with other nodes.
>
> We apply a patch in spark, but it is still the problem of RBF. Is it possible
> for RBF to replace the delegation token store using some other
> datasource(redis for example)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
