[jira] [Commented] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[
https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818651#comment-16818651
]
Hadoop QA commented on HDFS-14401:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 498 unchanged - 0 fixed = 499 total (was 498) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 59s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager.count;
locked 60% of time Unsynchronized access at PmemVolumeManager.java:60% of time
Unsynchronized access at PmemVolumeManager.java:[line 238] |
| | Incorrect lazy initialization of static field
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager.pmemVolumeManager
in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager.init(String[])
At PmemVolumeManager.java:field
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager.pmemVolumeManager
in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager.init(String[])
At PmemVolumeManager.java:[lines 139-142] |
| | Should
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.PmemVolumeManager$UsedBytesCount
be a _static_ inner class? At PmemVolumeManager.java:inner class? At
[jira] [Commented] (HDFS-14378) Simplify the design of multiple NN and both logic of edit log roll and checkpoint
[
https://issues.apache.org/jira/browse/HDFS-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818637#comment-16818637
]
Hadoop QA commented on HDFS-14378:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-14378 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14378 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964560/HDFS-14378-trunk.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26642/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Simplify the design of multiple NN and both logic of edit log roll and
> checkpoint
> -
>
> Key: HDFS-14378
> URL: https://issues.apache.org/jira/browse/HDFS-14378
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.1.2
>Reporter: star
>Assignee: star
>Priority: Minor
> Labels: patch
> Attachments: HDFS-14378-trunk.001.patch, HDFS-14378-trunk.002.patch
>
>
> HDFS-6440 introduced a mechanism to support more than 2 NNs. It
> implements a first-writer-win policy to avoid duplicated fsimage downloading.
> Variable 'isPrimaryCheckPointer' is used to hold the first-writer state, with
> which SNN will provide fsimage for ANN next time. Then we have three roles in
> NN cluster: ANN, one primary SNN, one or more normal SNN.
> Since HDFS-12248, there may be more than two primary SNN shortly after
> a exception occurred. It takes care with a scenario that SNN will not upload
> fsimage on IOE and Interrupted exceptions. Though it will not cause any
> further functional issues, it is inconsistent.
> Futher more, edit log may be rolled more frequently than necessary with
> multiple Standby name nodes, HDFS-14349. (I'm not so sure about this, will
> verify by unit tests or any one could point it out.)
> Above all, I‘m wondering if we could make it simple with following
> changes:
> * There are only two roles:ANN, SNN
> * ANN will roll its edit log every DFS_HA_LOGROLL_PERIOD_KEY period.
> * ANN will select a SNN to download checkpoint.
> SNN will just do logtail and checkpoint. Then provide a servlet for fsimage
> downloading as normal. SNN will not try to roll edit log or send checkpoint
> request to ANN.
> In a word, ANN will be more active. Suggestions are welcomed.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
venkata ramkumar reassigned HDFS-14117:
---
Assignee: (was: venkata ramkumar)
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch,
> HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14378) Simplify the design of multiple NN and both logic of edit log roll and checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HDFS-14378: Attachment: (was: HDFS-14378-trunk.002.patch) > Simplify the design of multiple NN and both logic of edit log roll and > checkpoint > - > > Key: HDFS-14378 > URL: https://issues.apache.org/jira/browse/HDFS-14378 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.1.2 >Reporter: star >Assignee: star >Priority: Minor > Labels: patch > Attachments: HDFS-14378-trunk.001.patch, HDFS-14378-trunk.002.patch > > > HDFS-6440 introduced a mechanism to support more than 2 NNs. It > implements a first-writer-win policy to avoid duplicated fsimage downloading. > Variable 'isPrimaryCheckPointer' is used to hold the first-writer state, with > which SNN will provide fsimage for ANN next time. Then we have three roles in > NN cluster: ANN, one primary SNN, one or more normal SNN. > Since HDFS-12248, there may be more than two primary SNN shortly after > a exception occurred. It takes care with a scenario that SNN will not upload > fsimage on IOE and Interrupted exceptions. Though it will not cause any > further functional issues, it is inconsistent. > Futher more, edit log may be rolled more frequently than necessary with > multiple Standby name nodes, HDFS-14349. (I'm not so sure about this, will > verify by unit tests or any one could point it out.) > Above all, I‘m wondering if we could make it simple with following > changes: > * There are only two roles:ANN, SNN > * ANN will roll its edit log every DFS_HA_LOGROLL_PERIOD_KEY period. > * ANN will select a SNN to download checkpoint. > SNN will just do logtail and checkpoint. Then provide a servlet for fsimage > downloading as normal. SNN will not try to roll edit log or send checkpoint > request to ANN. > In a word, ANN will be more active. Suggestions are welcomed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14378) Simplify the design of multiple NN and both logic of edit log roll and checkpoint
[ https://issues.apache.org/jira/browse/HDFS-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HDFS-14378: Labels: patch (was: ) Affects Version/s: 3.1.2 Attachment: HDFS-14378-trunk.002.patch Status: Patch Available (was: Open) > Simplify the design of multiple NN and both logic of edit log roll and > checkpoint > - > > Key: HDFS-14378 > URL: https://issues.apache.org/jira/browse/HDFS-14378 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.1.2 >Reporter: star >Assignee: star >Priority: Minor > Labels: patch > Attachments: HDFS-14378-trunk.001.patch, HDFS-14378-trunk.002.patch, > HDFS-14378-trunk.002.patch > > > HDFS-6440 introduced a mechanism to support more than 2 NNs. It > implements a first-writer-win policy to avoid duplicated fsimage downloading. > Variable 'isPrimaryCheckPointer' is used to hold the first-writer state, with > which SNN will provide fsimage for ANN next time. Then we have three roles in > NN cluster: ANN, one primary SNN, one or more normal SNN. > Since HDFS-12248, there may be more than two primary SNN shortly after > a exception occurred. It takes care with a scenario that SNN will not upload > fsimage on IOE and Interrupted exceptions. Though it will not cause any > further functional issues, it is inconsistent. > Futher more, edit log may be rolled more frequently than necessary with > multiple Standby name nodes, HDFS-14349. (I'm not so sure about this, will > verify by unit tests or any one could point it out.) > Above all, I‘m wondering if we could make it simple with following > changes: > * There are only two roles:ANN, SNN > * ANN will roll its edit log every DFS_HA_LOGROLL_PERIOD_KEY period. > * ANN will select a SNN to download checkpoint. > SNN will just do logtail and checkpoint. Then provide a servlet for fsimage > downloading as normal. SNN will not try to roll edit log or send checkpoint > request to ANN. > In a word, ANN will be more active. Suggestions are welcomed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14424) NN failover failed beacuse of lossing paxos directory
[ https://issues.apache.org/jira/browse/HDFS-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HDFS-14424: External issue ID: (was: HDFS-10659) > NN failover failed beacuse of lossing paxos directory > - > > Key: HDFS-14424 > URL: https://issues.apache.org/jira/browse/HDFS-14424 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: star >Assignee: star >Priority: Major > > Recently, our hadoop namenode shutdown when switching active namenode, just > because of missing paxos directory. It is created in the default /tmp path > and deleted by os for no operation in 7 days. We can avoid this by moving > journal directory to a none tmp dir, but it‘s better to make sure namenode > works well by a default config. > The issue throws exception similar to HDFS-10659, also caused by missing > paxos directory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14424) NN failover failed beacuse of lossing paxos directory
[ https://issues.apache.org/jira/browse/HDFS-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] star updated HDFS-14424: External issue ID: HDFS-10659 > NN failover failed beacuse of lossing paxos directory > - > > Key: HDFS-14424 > URL: https://issues.apache.org/jira/browse/HDFS-14424 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: star >Assignee: star >Priority: Major > > Recently, our hadoop namenode shutdown when switching active namenode, just > because of missing paxos directory. It is created in the default /tmp path > and deleted by os for no operation in 7 days. We can avoid this by moving > journal directory to a none tmp dir, but it‘s better to make sure namenode > works well by a default config. > The issue throws exception similar to HDFS-10659, also caused by missing > paxos directory. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818634#comment-16818634
]
star commented on HDFS-10659:
-
failed test unrelated. check style is not newly added for hidden variable 'sd' .
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch,
> HDFS-10659.005.patch, HDFS-10659.006.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13933) [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification problems for "localhost"
[
https://issues.apache.org/jira/browse/HDFS-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818631#comment-16818631
]
Siyao Meng commented on HDFS-13933:
---
Same error with 4 TestKMS unit tests:
{code}
[ERROR] testDelegationTokensOpsHttpsKerberized(org.apache.hadoop.crypto.key.kms.
server.TestKMS) Time elapsed: 1.784 s <<< ERROR!
java.io.IOException: HTTPS hostname wrong: should be
at java.base/sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(Htt
psClient.java:648)
at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsCl
ient.java:581)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnecti
on.connect(AbstractDelegateHttpsURLConnection.java:185)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0
(HttpURLConnection.java:1581)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(
HttpURLConnection.java:1509)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnectio
n.java:527)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getRespon
seCode(HttpsURLConnectionImpl.java:329)
at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExcept
ionUtils.java:140)
...
{code}
Other three failed TestKMS tests (related) are:
{code}
TestKMS.testDelegationTokensOpsHttpsPseudo
TestKMS.testStartStopHttpsKerberos
TestKMS.testStartStopHttpsPseudo
{code}
Compiled with JDK 8 and ran on OpenJDK 11. The root cause should be the same.
> [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification
> problems for "localhost"
> --
>
> Key: HDFS-13933
> URL: https://issues.apache.org/jira/browse/HDFS-13933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Andrew Purtell
>Priority: Minor
>
> Tests with issues:
> * TestHttpFSFWithSWebhdfsFileSystem
> * TestWebHdfsTokens
> * TestSWebHdfsFileContextMainOperations
> Possibly others. Failure looks like
> {noformat}
> java.io.IOException: localhost:50260: HTTPS hostname wrong: should be
>
> {noformat}
> These tests set up a trust store and use HTTPS connections, and with Java 11
> the client validation of the server name in the generated self-signed
> certificate is failing. Exceptions originate in the JRE's HTTP client
> library. How everything hooks together uses static initializers, static
> methods, JUnit MethodRules... There's a lot to unpack, not sure how to fix.
> This is Java 11+28
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1102) Confusing error log when datanode tries to connect to a destroyed pipeline
[
https://issues.apache.org/jira/browse/HDDS-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1102:
---
Labels: newbie pushed-to-craterlake test-badlands (was:
pushed-to-craterlake test-badlands)
> Confusing error log when datanode tries to connect to a destroyed pipeline
> --
>
> Key: HDDS-1102
> URL: https://issues.apache.org/jira/browse/HDDS-1102
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Critical
> Labels: newbie, pushed-to-craterlake, test-badlands
> Attachments: allnode.log, datanode.log
>
>
> steps taken:
>
> # created 5 datanode cluster.
> # shutdown 2 datanodes
> # started the datanodes again.
> One of the datanodes was shut down.
> exception seen :
>
> {noformat}
> 2019-02-14 07:37:26 INFO LeaderElection:230 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at
> org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187)
> at
> org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-14 07:37:26 INFO LeaderElection:46 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s)
> [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7]
> and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null,
> voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8,
> raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3:
> [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858,
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858,
> 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null
> 2019-02-14 07:37:26 INFO LeaderElection:52 - 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8:
> shutdown LeaderElection
> 2019-02-14 07:37:26 INFO RaftServerImpl:161 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at
> term 7 for changeToLeader
> 2019-02-14 07:37:26 INFO RaftServerImpl:258 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader
> elected after 1066ms
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.staging.catchup.gap = 1000 (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time
> = 25ms (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout
> = 10s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.watch.timeout.denomination = 1s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
>
[jira] [Resolved] (HDFS-14077) DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state
[
https://issues.apache.org/jira/browse/HDFS-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harshakiran Reddy resolved HDFS-14077.
--
Resolution: Not A Problem
> DFSAdmin Report datanode Count was not matched when datanode in
> Decommissioned state
> -
>
> Key: HDFS-14077
> URL: https://issues.apache.org/jira/browse/HDFS-14077
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
>
> {noformat}
> In DFSAdmin Reports showing the live datanodes are incorrect when some
> datanodes in Decommissioned State
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-14077) DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state
[
https://issues.apache.org/jira/browse/HDFS-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harshakiran Reddy reopened HDFS-14077:
--
> DFSAdmin Report datanode Count was not matched when datanode in
> Decommissioned state
> -
>
> Key: HDFS-14077
> URL: https://issues.apache.org/jira/browse/HDFS-14077
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
>
> {noformat}
> In DFSAdmin Reports showing the live datanodes are incorrect when some
> datanodes in Decommissioned State
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818594#comment-16818594
]
Hadoop QA commented on HDFS-10659:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 17 unchanged - 1 fixed = 18 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-10659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966008/HDFS-10659.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0df007c33fe2 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar
18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5583e1b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26640/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Comment Edited] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818577#comment-16818577 ] Feilong He edited comment on HDFS-14401 at 4/16/19 3:02 AM: HDFS-14401.003.patch has been uploaded with fixing a bug during initializing cache loader. was (Author: philohe): The patch HDFS14401.003.patch has been uploaded with fixing a bug during initializing cache loader. > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch, > HDFS-14401.002.patch, HDFS-14401.003.patch > > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818577#comment-16818577 ] Feilong He commented on HDFS-14401: --- The patch HDFS14401.003.patch has been uploaded with fixing a bug during initializing cache loader. > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch, > HDFS-14401.002.patch, HDFS-14401.003.patch > > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[ https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-14401: -- Attachment: HDFS-14401.003.patch > Refine the implementation for HDFS cache on SCM > --- > > Key: HDFS-14401 > URL: https://issues.apache.org/jira/browse/HDFS-14401 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch, > HDFS-14401.002.patch, HDFS-14401.003.patch > > > In this Jira, we will refine the implementation for HDFS cache on SCM, such > as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume > selection impl; 3) Clean up MapppableBlockLoader interface; etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1441) Remove usage of getRetryFailureException
Arpit Agarwal created HDDS-1441: --- Summary: Remove usage of getRetryFailureException Key: HDDS-1441 URL: https://issues.apache.org/jira/browse/HDDS-1441 Project: Hadoop Distributed Data Store Issue Type: Improvement Reporter: Arpit Agarwal Per [~szetszwo]'s comment on RATIS-518, we can remove the usage of getRetryFailureException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1409) TestOzoneClientRetriesOnException is flaky
[
https://issues.apache.org/jira/browse/HDDS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818503#comment-16818503
]
chencan commented on HDDS-1409:
---
The uploaded patch has fixed the assertion error.
> TestOzoneClientRetriesOnException is flaky
> --
>
> Key: HDDS-1409
> URL: https://issues.apache.org/jira/browse/HDDS-1409
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Nanda kumar
>Assignee: chencan
>Priority: Major
> Labels: ozone-flaky-test
> Attachments: HDDS-1409.001.patch
>
>
> TestOzoneClientRetriesOnException is flaky, we get the below exception when
> it fails.
> {noformat}
> [ERROR]
> testMaxRetriesByOzoneClient(org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException)
> Time elapsed: 16.227 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnException.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14422) RBF: Router shouldn't allow READ operations in safe mode
[
https://issues.apache.org/jira/browse/HDFS-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818495#comment-16818495
]
Hadoop QA commented on HDFS-14422:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
25s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
3s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14422 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966006/HDFS-14422-HDFS-13891.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 40b6b86e24c4 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / e508ab9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26639/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26639/testReport/ |
| Max. process+thread count | 1057 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Work logged] (HDDS-1400) Convert all OM Key related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1400?focusedWorklogId=228070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228070 ] ASF GitHub Bot logged work on HDDS-1400: Author: ASF GitHub Bot Created on: 16/Apr/19 01:02 Start Date: 16/Apr/19 01:02 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #744: HDDS-1400. Convert all OM Key related operations to HA model. URL: https://github.com/apache/hadoop/pull/744#issuecomment-483471876 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 23 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 51 | Maven dependency ordering for branch | | +1 | mvninstall | 1045 | trunk passed | | +1 | compile | 134 | trunk passed | | -0 | checkstyle | 33 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 123 | trunk passed | | +1 | shadedclient | 819 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 105 | trunk passed | | +1 | javadoc | 84 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for patch | | +1 | mvninstall | 105 | the patch passed | | +1 | compile | 105 | the patch passed | | +1 | cc | 105 | the patch passed | | +1 | javac | 105 | the patch passed | | -0 | checkstyle | 18 | The patch fails to run checkstyle in hadoop-ozone | | +1 | mvnsite | 84 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 707 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 113 | the patch passed | | +1 | javadoc | 61 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 33 | common in the patch passed. | | -1 | unit | 39 | ozone-manager in the patch failed. | | -1 | unit | 795 | integration-test in the patch failed. | | +1 | asflicense | 27 | The patch does not generate ASF License warnings. | | | | 4524 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.ratis.TestOzoneManagerStateMachine | | | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.scm.TestAllocateContainer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/744 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux a56d541e649e 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 5583e1b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/artifact/out//testptch/patchprocess/maven-branch-checkstyle-hadoop-ozone.txt | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/artifact/out//testptch/patchprocess/maven-patch-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/artifact/out/patch-unit-hadoop-ozone_ozone-manager.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/testReport/ | | Max. process+thread count | 4608 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-744/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818480#comment-16818480
]
star commented on HDFS-10659:
-
fix check style.
Seems func JNStorage#findFinalizedEditsFile is never used. Need to remove?
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch,
> HDFS-10659.005.patch, HDFS-10659.006.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
star updated HDFS-10659:
Attachment: HDFS-10659.006.patch
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch,
> HDFS-10659.005.patch, HDFS-10659.006.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1371) Download RocksDB checkpoint from OM Leader to Follower
[ https://issues.apache.org/jira/browse/HDDS-1371?focusedWorklogId=228061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228061 ] ASF GitHub Bot logged work on HDDS-1371: Author: ASF GitHub Bot Created on: 16/Apr/19 00:13 Start Date: 16/Apr/19 00:13 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #703: HDDS-1371. Download RocksDB checkpoint from OM Leader to Follower. URL: https://github.com/apache/hadoop/pull/703#discussion_r275588755 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -597,6 +597,18 @@ ozone.om.http-address. + +ozone.om.http.policy +HTTP_ONLY +OM, MANAGEMENT Review comment: Just want to tag this: As OzoneManagerHttpServer uses BaseHttpServer, and that uses dfs.http.policy to find when it should use http/httpsAddress. With this newly added property, we should pass this policy when OzoneManagerHtttpServer is started, to updateConnectorAddress during httpServer Start. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228061) Time Spent: 2h 10m (was: 2h) > Download RocksDB checkpoint from OM Leader to Follower > -- > > Key: HDDS-1371 > URL: https://issues.apache.org/jira/browse/HDDS-1371 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > If a follower OM is lagging way behind the leader OM or in case of a restart > or bootstrapping, a follower OM might need RocksDB checkpoint from the leader > to catch up with it. This is because the leader might have purged its logs > after taking a snapshot. > This Jira aims to add support to download a RocksDB checkpoint from leader > OM to follower OM through a HTTP servlet. We reuse the DBCheckpoint servlet > used by Recon server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1371) Download RocksDB checkpoint from OM Leader to Follower
[ https://issues.apache.org/jira/browse/HDDS-1371?focusedWorklogId=228062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228062 ] ASF GitHub Bot logged work on HDDS-1371: Author: ASF GitHub Bot Created on: 16/Apr/19 00:13 Start Date: 16/Apr/19 00:13 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #703: HDDS-1371. Download RocksDB checkpoint from OM Leader to Follower. URL: https://github.com/apache/hadoop/pull/703#discussion_r275588755 ## File path: hadoop-hdds/common/src/main/resources/ozone-default.xml ## @@ -597,6 +597,18 @@ ozone.om.http-address. + +ozone.om.http.policy +HTTP_ONLY +OM, MANAGEMENT Review comment: As OzoneManagerHttpServer uses BaseHttpServer, and that uses dfs.http.policy to find when it should use http/httpsAddress. With this newly added property, we should pass this policy when OzoneManagerHtttpServer is started, to updateConnectorAddress during httpServer Start. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228062) Time Spent: 2h 20m (was: 2h 10m) > Download RocksDB checkpoint from OM Leader to Follower > -- > > Key: HDDS-1371 > URL: https://issues.apache.org/jira/browse/HDDS-1371 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > If a follower OM is lagging way behind the leader OM or in case of a restart > or bootstrapping, a follower OM might need RocksDB checkpoint from the leader > to catch up with it. This is because the leader might have purged its logs > after taking a snapshot. > This Jira aims to add support to download a RocksDB checkpoint from leader > OM to follower OM through a HTTP servlet. We reuse the DBCheckpoint servlet > used by Recon server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14422) RBF: Router shouldn't allow READ operations in safe mode
[ https://issues.apache.org/jira/browse/HDFS-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818472#comment-16818472 ] Íñigo Goiri commented on HDFS-14422: [^HDFS-14422-HDFS-13891.001.patch] tries to read first and if it doesn't work report safe mode. > RBF: Router shouldn't allow READ operations in safe mode > > > Key: HDFS-14422 > URL: https://issues.apache.org/jira/browse/HDFS-14422 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14422-HDFS-13891.000.patch, > HDFS-14422-HDFS-13891.001.patch > > > We are currently seeing: > org.apache.hadoop.hdfs.server.federation.store.StateStoreUnavailableException: > Mount Table not initialized > at > org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.verifyMountTable(MountTableResolver.java:521) > at > org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:394) > at > org.apache.hadoop.hdfs.server.federation.resolver.MultipleDestinationMountTableResolver.getDestinationForPath(MultipleDestinationMountTableResolver.java:87) > at > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:1258) > at > org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getFileInfo(RouterClientProtocol.java:747) > at > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getFileInfo(RouterRpcServer.java:749) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:881) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:513) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1011) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1915) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2621) > The Namenode allows READ operations but for the Router not being able to > access the State Store also hits the read operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14422) RBF: Router shouldn't allow READ operations in safe mode
[ https://issues.apache.org/jira/browse/HDFS-14422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14422: --- Attachment: HDFS-14422-HDFS-13891.001.patch > RBF: Router shouldn't allow READ operations in safe mode > > > Key: HDFS-14422 > URL: https://issues.apache.org/jira/browse/HDFS-14422 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14422-HDFS-13891.000.patch, > HDFS-14422-HDFS-13891.001.patch > > > We are currently seeing: > org.apache.hadoop.hdfs.server.federation.store.StateStoreUnavailableException: > Mount Table not initialized > at > org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.verifyMountTable(MountTableResolver.java:521) > at > org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver.getDestinationForPath(MountTableResolver.java:394) > at > org.apache.hadoop.hdfs.server.federation.resolver.MultipleDestinationMountTableResolver.getDestinationForPath(MultipleDestinationMountTableResolver.java:87) > at > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getLocationsForPath(RouterRpcServer.java:1258) > at > org.apache.hadoop.hdfs.server.federation.router.RouterClientProtocol.getFileInfo(RouterClientProtocol.java:747) > at > org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.getFileInfo(RouterRpcServer.java:749) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:881) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:513) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1011) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1915) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2621) > The Namenode allows READ operations but for the Router not being able to > access the State Store also hits the read operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1440) Convert all MPU related operations to HA model
Bharat Viswanadham created HDDS-1440: Summary: Convert all MPU related operations to HA model Key: HDDS-1440 URL: https://issues.apache.org/jira/browse/HDDS-1440 Project: Hadoop Distributed Data Store Issue Type: Sub-task Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Fix For: 0.5.0 In this jira, we shall convert all OM related operations to OM HA model, which is a 2 step. # StartTransaction, where we validate request and check for any errors and return the response. # ApplyTransaction, where original OM request will have a response which needs to be applied to OM DB. This step is just to apply response to Om DB. In this way, all requests which are failed with like volume not found or some conditions which i have not satisfied like when deleting volume should be empty, these all will be executed during startTransaction, and if it fails these requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1400) Convert all OM Key related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1400: - Labels: pull-request-available (was: ) > Convert all OM Key related operations to HA model > - > > Key: HDDS-1400 > URL: https://issues.apache.org/jira/browse/HDDS-1400 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > > In this jira, we shall convert all OM related operations to OM HA model, > which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like volume not found or some > conditions which are not satisfied like when Key not found during rename, > these all will be executed during startTransaction, and if it fails these > requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1400) Convert all OM Key related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1400?focusedWorklogId=228048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228048 ] ASF GitHub Bot logged work on HDDS-1400: Author: ASF GitHub Bot Created on: 15/Apr/19 23:46 Start Date: 15/Apr/19 23:46 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #744: HDDS-1400. Convert all OM Key related operations to HA model. URL: https://github.com/apache/hadoop/pull/744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 228048) Time Spent: 10m Remaining Estimate: 0h > Convert all OM Key related operations to HA model > - > > Key: HDDS-1400 > URL: https://issues.apache.org/jira/browse/HDDS-1400 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > In this jira, we shall convert all OM related operations to OM HA model, > which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like volume not found or some > conditions which are not satisfied like when Key not found during rename, > these all will be executed during startTransaction, and if it fails these > requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1400) Convert all OM Key related operations to HA model
[ https://issues.apache.org/jira/browse/HDDS-1400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1400: - Status: Patch Available (was: Open) > Convert all OM Key related operations to HA model > - > > Key: HDDS-1400 > URL: https://issues.apache.org/jira/browse/HDDS-1400 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.5.0 > > > In this jira, we shall convert all OM related operations to OM HA model, > which is a 2 step. > # StartTransaction, where we validate request and check for any errors and > return the response. > # ApplyTransaction, where original OM request will have a response which > needs to be applied to OM DB. This step is just to apply response to Om DB. > In this way, all requests which are failed with like volume not found or some > conditions which are not satisfied like when Key not found during rename, > these all will be executed during startTransaction, and if it fails these > requests will not be written to raft log also. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818463#comment-16818463 ] Xue Liu commented on HDFS-14406: [~linyiqun] Thanks for the comment! Checkstyles and others has been fixed. Regarding unit test, I think all metrics registered using MetricsRegistry are exposed via JMX, do I still need to add a single JMX test for it? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
[ https://issues.apache.org/jira/browse/HDDS-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar resolved HDDS-1439. -- Resolution: Not A Problem This error comes because of config issue. fs.o3fs.impl should be set to org.apache.hadoop.fs.ozone.BasicOzoneFileSystem for Hadoop 2. > ozone spark job failing with class not found error for hadoop 2 > --- > > Key: HDDS-1439 > URL: https://issues.apache.org/jira/browse/HDDS-1439 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.4.0 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > spark job fails to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818446#comment-16818446
]
Hadoop QA commented on HDFS-14406:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
28s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965994/HDFS-14406.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 973d96d07acf 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5583e1b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26637/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26637/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add per user
[jira] [Work logged] (HDDS-1376) Datanode exits while executing client command when scmId is null
[
https://issues.apache.org/jira/browse/HDDS-1376?focusedWorklogId=228019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-228019
]
ASF GitHub Bot logged work on HDDS-1376:
Author: ASF GitHub Bot
Created on: 15/Apr/19 22:44
Start Date: 15/Apr/19 22:44
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #724:
HDDS-1376. Datanode exits while executing client command when scmId is null
URL: https://github.com/apache/hadoop/pull/724#discussion_r275571854
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
##
@@ -107,7 +107,6 @@ public void create(VolumeSet volumeSet,
VolumeChoosingPolicy
Preconditions.checkNotNull(volumeChoosingPolicy, "VolumeChoosingPolicy " +
"cannot be null");
Preconditions.checkNotNull(volumeSet, "VolumeSet cannot be null");
-Preconditions.checkNotNull(scmId, "scmId cannot be null");
Review comment:
Added them back.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 228019)
Time Spent: 1h 10m (was: 1h)
> Datanode exits while executing client command when scmId is null
>
>
> Key: HDDS-1376
> URL: https://issues.apache.org/jira/browse/HDDS-1376
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Ozone Datanode exits with the following error, this happens because DN hasn't
> received a scmID from the SCM after registration but is processing a client
> command.
> {code}
> 2019-04-03 17:02:10,958 ERROR storage.RaftLogWorker
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1:
> df6b578e-8d35-44f5-9b21-db7184dcc54e-RaftLogWorker failed.
> java.io.IOException: java.lang.NullPointerException: scmId cannot be null
> at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:54)
> at org.apache.ratis.util.IOUtils.toIOException(IOUtils.java:61)
> at org.apache.ratis.util.IOUtils.getFromFuture(IOUtils.java:83)
> at
> org.apache.ratis.server.storage.RaftLogWorker$StateMachineDataPolicy.getFromFuture(RaftLogWorker.java:76)
> at
> org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:354)
> at
> org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:219)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException: scmId cannot be null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.create(KeyValueContainer.java:110)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleCreateContainer(KeyValueHandler.java:243)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:165)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.createContainer(HddsDispatcher.java:350)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:224)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$handleWriteChunk$0(ContainerStateMachine.java:385)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1590)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ... 1 more
> {code}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.004.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: (was: HDFS-14406.004.patch) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818375#comment-16818375
]
Hadoop QA commented on HDFS-14117:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 22m
24s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14117 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965986/HDFS-14117-HDFS-13891.019.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5ffb11b32f98 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / e508ab9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26636/testReport/ |
| Max. process+thread count | 1355 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26636/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is
[jira] [Commented] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818362#comment-16818362 ] Hudson commented on HDDS-1396: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16411 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16411/]) HDDS-1396 : Recon start fails due to changes in Aggregate Schema (bharat: rev 62e38ea911dab7bff8553a167a97858c3cfb58e3) * (edit) hadoop-ozone/ozone-recon/src/main/java/org/apache/hadoop/ozone/recon/persistence/JooqPersistenceModule.java * (edit) hadoop-ozone/ozone-recon/src/test/java/org/apache/hadoop/ozone/recon/persistence/AbstractSqlDatabaseTest.java * (edit) hadoop-ozone/ozone-recon-codegen/pom.xml * (edit) hadoop-ozone/ozone-recon/pom.xml * (edit) hadoop-ozone/pom.xml > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818359#comment-16818359 ] Siddharth Wagle commented on HDDS-1396: --- Thanks [~bharatviswa] for committing this blocker. > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818359#comment-16818359 ] Siddharth Wagle edited comment on HDDS-1396 at 4/15/19 8:21 PM: Thanks [~bharatviswa] for committing this Recon blocker. was (Author: swagle): Thanks [~bharatviswa] for committing this blocker. > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818355#comment-16818355
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
27s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965985/HDFS-14406.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 99a8a1b70cd5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 254efc9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-compile-root.txt
|
| mvnsite |
[jira] [Resolved] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham resolved HDDS-1396. -- Resolution: Fixed Thank You [~avijayan] for the contribution and [~swagle] for the review. I have committed this to trunk. > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?focusedWorklogId=227922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227922 ] ASF GitHub Bot logged work on HDDS-1396: Author: ASF GitHub Bot Created on: 15/Apr/19 19:58 Start Date: 15/Apr/19 19:58 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #700: HDDS-1396 : Recon start fails due to changes in Aggregate Schema definition. URL: https://github.com/apache/hadoop/pull/700 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227922) Time Spent: 1h 50m (was: 1h 40m) > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?focusedWorklogId=227921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227921 ] ASF GitHub Bot logged work on HDDS-1396: Author: ASF GitHub Bot Created on: 15/Apr/19 19:57 Start Date: 15/Apr/19 19:57 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #700: HDDS-1396 : Recon start fails due to changes in Aggregate Schema definition. URL: https://github.com/apache/hadoop/pull/700#issuecomment-483395673 Test failures are not related to this patch. I will commit this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227921) Time Spent: 1h 40m (was: 1.5h) > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Recon Server start fails due to ClassNotFound exception when looking for > org.apache.hadoop.ozone.recon.ReconServer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1396) Recon start fails due to changes in Aggregate Schema definition (HDDS-1189).
[ https://issues.apache.org/jira/browse/HDDS-1396?focusedWorklogId=227910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227910 ] ASF GitHub Bot logged work on HDDS-1396: Author: ASF GitHub Bot Created on: 15/Apr/19 19:43 Start Date: 15/Apr/19 19:43 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #700: HDDS-1396 : Recon start fails due to changes in Aggregate Schema definition. URL: https://github.com/apache/hadoop/pull/700#issuecomment-483390856 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 39 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 48 | Maven dependency ordering for branch | | +1 | mvninstall | 1122 | trunk passed | | +1 | compile | 129 | trunk passed | | +1 | checkstyle | 101 | trunk passed | | +1 | mvnsite | 192 | trunk passed | | +1 | shadedclient | 995 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone | | +1 | findbugs | 77 | trunk passed | | +1 | javadoc | 110 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for patch | | +1 | mvninstall | 357 | the patch passed | | +1 | compile | 119 | the patch passed | | +1 | javac | 119 | the patch passed | | +1 | checkstyle | 25 | the patch passed | | +1 | mvnsite | 164 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 4 | The patch has no ill-formed XML file. | | +1 | shadedclient | 822 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone | | +1 | findbugs | 85 | the patch passed | | +1 | javadoc | 98 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 1629 | hadoop-ozone in the patch failed. | | +1 | unit | 29 | ozone-recon-codegen in the patch passed. | | +1 | unit | 47 | ozone-recon in the patch passed. | | +1 | asflicense | 34 | The patch does not generate ASF License warnings. | | | | 6091 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.scm.TestAllocateContainer | | | hadoop.ozone.client.rpc.TestCommitWatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-700/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/700 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 747e2b996f22 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7fa73fa | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-700/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-700/2/testReport/ | | Max. process+thread count | 5011 (vs. ulimit of 5500) | | modules | C: hadoop-ozone hadoop-ozone/ozone-recon-codegen hadoop-ozone/ozone-recon U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-700/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227910) Time Spent: 1.5h (was: 1h 20m) > Recon start fails due to changes in Aggregate Schema definition (HDDS-1189). > > > Key: HDDS-1396 > URL: https://issues.apache.org/jira/browse/HDDS-1396 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Recon >Reporter: Aravindan Vijayan >Assignee: Aravindan Vijayan >
[jira] [Commented] (HDFS-13972) RBF: Support for Delegation Token (WebHDFS)
[ https://issues.apache.org/jira/browse/HDFS-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818291#comment-16818291 ] Íñigo Goiri commented on HDFS-13972: If nobody else has concerns about the static approach, I'm fine with it. Can we add the check to verify we are using the super user only for this case? The unit test should have some way to check the UGI of the calls. > RBF: Support for Delegation Token (WebHDFS) > --- > > Key: HDFS-13972 > URL: https://issues.apache.org/jira/browse/HDFS-13972 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13972-HDFS-13891.001.patch, > HDFS-13972-HDFS-13891.002.patch, HDFS-13972-HDFS-13891.003.patch, > HDFS-13972-HDFS-13891.004.patch, HDFS-13972-HDFS-13891.005.patch, > HDFS-13972-HDFS-13891.006.patch, HDFS-13972-HDFS-13891.007.patch, > HDFS-13972-HDFS-13891.008.patch, HDFS-13972-HDFS-13891.009.patch, > HDFS-13972-HDFS-13891.010.patch, HDFS-13972-HDFS-13891.011.patch, > HDFS-13972-HDFS-13891.012.patch, TestRouterWebHDFSContractTokens.java > > > HDFS Router should support issuing HDFS delegation tokens through WebHDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14117:
---
Attachment: HDFS-14117-HDFS-13891.019.patch
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Assignee: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117-HDFS-13891.019.patch,
> HDFS-14117.001.patch, HDFS-14117.002.patch, HDFS-14117.003.patch,
> HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> rm: Failed to move to trash: hdfs://router/tmp/mapred.cmd: rename destination
> parent /user/securedn/.Trash/Current/tmp/mapred.cmd not found.
> 4./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm /test/hdfs.cmd
> 18/11/30 11:01:20 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 18/11/30 11:01:22 INFO fs.TrashPolicyDefault: Moved:
> 'hdfs://router/test/hdfs.cmd' to trash at:
> hdfs://router/user/securedn/.Trash/Current/test/hdfs.cmd
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14426) RBF: Add delegation token total count as one of the federation metrics
[ https://issues.apache.org/jira/browse/HDFS-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818286#comment-16818286 ] Íñigo Goiri commented on HDFS-14426: [~fengnanli] can you provide the patch based on HDFS-13891 so it can build succesfully? You only need to add the suffix to the patch. For the unit test, I would add a check for 0 tokens at the very beginning too. > RBF: Add delegation token total count as one of the federation metrics > -- > > Key: HDFS-14426 > URL: https://issues.apache.org/jira/browse/HDFS-14426 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > Attachments: HDFS-14426.001.patch > > > Currently router doesn't report the total number of current valid delegation > tokens it has, but this piece of information is useful for monitoring and > understanding the real time situation of tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14427) RBF: Optimize some testing set up logic in MiniRouterDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818285#comment-16818285 ] Íñigo Goiri commented on HDFS-14427: Thanks [~fengnanli] for bringing this up. I would leave the behavior as is and update the comment. I also think it'd be nice to have the extra option to specify the number of Routers; we can use -1 to leave as it is right now. > RBF: Optimize some testing set up logic in MiniRouterDFSCluster > --- > > Key: HDFS-14427 > URL: https://issues.apache.org/jira/browse/HDFS-14427 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > > [https://github.com/apache/hadoop/blob/HDFS-13891/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/MiniRouterDFSCluster.java#L808] > the comment says one router is created per name service, while in the code > one router is created per namenode in each nameservice. > There are a couple of things that might need to consider optimization: > # make the code as the the comment > # add some ways to specify the number of routers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14427) RBF: Optimize some testing set up logic in MiniRouterDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-14427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14427: --- Summary: RBF: Optimize some testing set up logic in MiniRouterDFSCluster (was: Optimize some testing set up logic in MiniRouterDFSCluster) > RBF: Optimize some testing set up logic in MiniRouterDFSCluster > --- > > Key: HDFS-14427 > URL: https://issues.apache.org/jira/browse/HDFS-14427 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Fengnan Li >Assignee: Fengnan Li >Priority: Major > > [https://github.com/apache/hadoop/blob/HDFS-13891/hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/MiniRouterDFSCluster.java#L808] > the comment says one router is created per name service, while in the code > one router is created per namenode in each nameservice. > There are a couple of things that might need to consider optimization: > # make the code as the the comment > # add some ways to specify the number of routers -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xue Liu updated HDFS-14406: --- Attachment: HDFS-14406.004.patch > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1373) KeyOutputStream, close after write request fails after retries, runs into IllegalArgumentException
[
https://issues.apache.org/jira/browse/HDDS-1373?focusedWorklogId=227867=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227867
]
ASF GitHub Bot logged work on HDDS-1373:
Author: ASF GitHub Bot
Created on: 15/Apr/19 18:09
Start Date: 15/Apr/19 18:09
Worklog Time Spent: 10m
Work Description: jnp commented on pull request #729: HDDS-1373.
KeyOutputStream, close after write request fails after retries, runs into
IllegalArgumentException
URL: https://github.com/apache/hadoop/pull/729#discussion_r275481162
##
File path:
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java
##
@@ -295,60 +295,66 @@ private void handleWrite(byte[] b, int off, long len,
boolean retry)
throws IOException {
int succeededAllocates = 0;
while (len > 0) {
- if (streamEntries.size() <= currentStreamIndex) {
-Preconditions.checkNotNull(omClient);
-// allocate a new block, if a exception happens, log an error and
-// throw exception to the caller directly, and the write fails.
+ try {
+if (streamEntries.size() <= currentStreamIndex) {
+ Preconditions.checkNotNull(omClient);
+ // allocate a new block, if a exception happens, log an error and
+ // throw exception to the caller directly, and the write fails.
+ try {
+allocateNewBlock(currentStreamIndex);
+succeededAllocates += 1;
+ } catch (IOException ioe) {
+LOG.error("Try to allocate more blocks for write failed, already "
++ "allocated " + succeededAllocates
++ " blocks for this write.");
+throw ioe;
+ }
+}
+// in theory, this condition should never violate due the check above
+// still do a sanity check.
+Preconditions.checkArgument(currentStreamIndex < streamEntries.size());
+BlockOutputStreamEntry current = streamEntries.get(currentStreamIndex);
+
+// length(len) will be in int range if the call is happening through
+// write API of blockOutputStream. Length can be in long range if it
comes
+// via Exception path.
+int writeLen = Math.min((int) len, (int) current.getRemaining());
+long currentPos = current.getWrittenDataLength();
try {
- allocateNewBlock(currentStreamIndex);
- succeededAllocates += 1;
+ if (retry) {
+current.writeOnRetry(len);
+ } else {
+current.write(b, off, writeLen);
+offset += writeLen;
+ }
} catch (IOException ioe) {
- LOG.error("Try to allocate more blocks for write failed, already "
- + "allocated " + succeededAllocates + " blocks for this write.");
- throw ioe;
-}
- }
- // in theory, this condition should never violate due the check above
- // still do a sanity check.
- Preconditions.checkArgument(currentStreamIndex < streamEntries.size());
- BlockOutputStreamEntry current = streamEntries.get(currentStreamIndex);
-
- // length(len) will be in int range if the call is happening through
- // write API of blockOutputStream. Length can be in long range if it
comes
- // via Exception path.
- int writeLen = Math.min((int)len, (int) current.getRemaining());
- long currentPos = current.getWrittenDataLength();
- try {
-if (retry) {
- current.writeOnRetry(len);
-} else {
- current.write(b, off, writeLen);
- offset += writeLen;
+ // for the current iteration, totalDataWritten - currentPos gives the
+ // amount of data already written to the buffer
+
+ // In the retryPath, the total data to be written will always be
equal
+ // to or less than the max length of the buffer allocated.
+ // The len specified here is the combined sum of the data length of
+ // the buffers
+ Preconditions.checkState(!retry || len <= streamBufferMaxSize);
+ int dataWritten = (int) (current.getWrittenDataLength() -
currentPos);
+ writeLen = retry ? (int) len : dataWritten;
+ // In retry path, the data written is already accounted in offset.
+ if (!retry) {
+offset += writeLen;
+ }
+ LOG.debug("writeLen {}, total len {}", writeLen, len);
+ handleException(current, currentStreamIndex, ioe);
}
- } catch (IOException ioe) {
-// for the current iteration, totalDataWritten - currentPos gives the
-// amount of data already written to the buffer
-
-// In the retryPath, the total data to be written will always be equal
-// to or less than the max length of the buffer allocated.
-// The len specified here is the combined
[jira] [Work logged] (HDDS-1434) TestDatanodeStateMachine is flaky
[
https://issues.apache.org/jira/browse/HDDS-1434?focusedWorklogId=227866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227866
]
ASF GitHub Bot logged work on HDDS-1434:
Author: ASF GitHub Bot
Created on: 15/Apr/19 18:01
Start Date: 15/Apr/19 18:01
Worklog Time Spent: 10m
Work Description: adoroszlai commented on issue #740: [HDDS-1434]
TestDatanodeStateMachine is flaky
URL: https://github.com/apache/hadoop/pull/740#issuecomment-483356897
@arp7 @nandakumar131 please review
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227866)
Time Spent: 0.5h (was: 20m)
> TestDatanodeStateMachine is flaky
> -
>
> Key: HDDS-1434
> URL: https://issues.apache.org/jira/browse/HDDS-1434
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Doroszlai, Attila
>Priority: Major
> Labels: ozone-flaky-test, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> TestDatanodeStateMachine is flaky.
> It has failed in the following build
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2650/artifact/out/patch-unit-hadoop-hdds.txt]
>
> [https://builds.apache.org/job/hadoop-multibranch/job/PR-661/6/artifact/out/patch-unit-hadoop-hdds_container-service.txt]
>
> [https://builds.apache.org/job/PreCommit-HDDS-Build/2635/artifact/out/patch-unit-hadoop-hdds.txt]
> Stack trace:
> {noformat}
> java.lang.Thread.State: WAITING (on object monitor)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at
> org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:389)
> at
> org.apache.hadoop.ozone.container.common.TestDatanodeStateMachine.testStartStopDatanodeStateMachine(TestDatanodeStateMachine.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
[jira] [Updated] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
[ https://issues.apache.org/jira/browse/HDDS-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1439: --- Target Version/s: 0.5.0 (was: 0.4.0) > ozone spark job failing with class not found error for hadoop 2 > --- > > Key: HDDS-1439 > URL: https://issues.apache.org/jira/browse/HDDS-1439 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.4.0 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > spark job fails to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
[ https://issues.apache.org/jira/browse/HDDS-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818203#comment-16818203 ] Jitendra Nath Pandey commented on HDDS-1439: I think this should not be a release blocker. IMO, ozone working on hadoop-2 is nice to have, but not a must requirement. However, I agree this should be fixed. > ozone spark job failing with class not found error for hadoop 2 > --- > > Key: HDDS-1439 > URL: https://issues.apache.org/jira/browse/HDDS-1439 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.4.0 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > spark job fails to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
[ https://issues.apache.org/jira/browse/HDDS-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1439: - Priority: Major (was: Blocker) > ozone spark job failing with class not found error for hadoop 2 > --- > > Key: HDDS-1439 > URL: https://issues.apache.org/jira/browse/HDDS-1439 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Affects Versions: 0.4.0 >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > spark job fails to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14117) RBF: We can only delete the files or dirs of one subcluster in a cluster with multiple subclusters when trash is enabled
[
https://issues.apache.org/jira/browse/HDFS-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818190#comment-16818190
]
Íñigo Goiri commented on HDFS-14117:
The failed unit tests is caused by not doing the check for the file locations.
The issue is that now we don't know where the file really is (we just now it's
in the federation).
Then we know that the file might be in ns0 or ns1 and we know the destination
is in ns1.
When we trigger the rename, we only trigger it against ns1.
Then we get a FileNotFoundException from ns1 when it should be a "not eligible
location".
The way to handle this would be to infer that if we found the file first and
now we get a FileNotFoundException, it must be because the src/dst is not
allowed.
With the previous approach we knew this failure case in just the
{{getFileInfo}} while now we have to do a {{getFileInfo}} and a {{rename2}} to
realize.
Here there is a trade off between the amount of calls in the success and
failure code paths and readability.
To summarize:
* First approach ([^HDFS-14117-HDFS-13891.017.patch]): read everywhere and
rename in the right place.
** Positive case: getFileInfo in all subclusters, rename2 in the right
subcluster.
** Negative case: getFileInfo in all subclusters. No rename2 needed.
* Second approach ([^HDFS-14117-HDFS-13891.018.patch]); read in order and
rename everywhere.
** Positive case: getFileInfo in the right subcluster, rename2 in the right
subcluster.
** Negative case: getFileInfo in the right subcluster, rename2 in all
subclusters (+ a non-intuitive code path that catches a FileNotFoundException
and infers a NotEligibleRenameException).
Keep in mind rename2 uses the writeLock while getFileInfo only readLock.
I'm fine with either one.
[~ayushtkn] which one you prefer?
> RBF: We can only delete the files or dirs of one subcluster in a cluster with
> multiple subclusters when trash is enabled
>
>
> Key: HDFS-14117
> URL: https://issues.apache.org/jira/browse/HDFS-14117
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: venkata ramkumar
>Assignee: venkata ramkumar
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14117-HDFS-13891.001.patch,
> HDFS-14117-HDFS-13891.002.patch, HDFS-14117-HDFS-13891.003.patch,
> HDFS-14117-HDFS-13891.004.patch, HDFS-14117-HDFS-13891.005.patch,
> HDFS-14117-HDFS-13891.006.patch, HDFS-14117-HDFS-13891.007.patch,
> HDFS-14117-HDFS-13891.008.patch, HDFS-14117-HDFS-13891.009.patch,
> HDFS-14117-HDFS-13891.010.patch, HDFS-14117-HDFS-13891.011.patch,
> HDFS-14117-HDFS-13891.012.patch, HDFS-14117-HDFS-13891.013.patch,
> HDFS-14117-HDFS-13891.014.patch, HDFS-14117-HDFS-13891.015.patch,
> HDFS-14117-HDFS-13891.016.patch, HDFS-14117-HDFS-13891.017.patch,
> HDFS-14117-HDFS-13891.018.patch, HDFS-14117.001.patch, HDFS-14117.002.patch,
> HDFS-14117.003.patch, HDFS-14117.004.patch, HDFS-14117.005.patch
>
>
> When we delete files or dirs in hdfs, it will move the deleted files or dirs
> to trash by default.
> But in the global path we can only mount one trash dir /user. So we mount
> trash dir /user of the subcluster ns1 to the global path /user. Then we can
> delete files or dirs of ns1, but when we delete the files or dirs of another
> subcluser, such as hacluster, it will be failed.
> h1. Mount Table
> ||Global path||Target nameservice||Target path||Order||Read
> only||Owner||Group||Permission||Quota/Usage||Date Modified||Date Created||
> |/test|hacluster2|/test| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:42|2018/11/29 14:37:42|
> |/tmp|hacluster1|/tmp| | |securedn|users|rwxr-xr-x|[NsQuota: -/-, SsQuota:
> -/-]|2018/11/29 14:37:05|2018/11/29 14:37:05|
> |/user|hacluster2,hacluster1|/user|HASH| |securedn|users|rwxr-xr-x|[NsQuota:
> -/-, SsQuota: -/-]|2018/11/29 14:42:37|2018/11/29 14:38:20|
> commands:
> {noformat}
> 1./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /test/.
> 18/11/30 11:00:47 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 8081 2018-11-30 10:56 /test/hdfs.cmd
> 2./opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -ls /tmp/.
> 18/11/30 11:00:40 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> -rw-r--r-- 3 securedn supergroup 6311 2018-11-30 10:57 /tmp/mapred.cmd
> 3../opt/HAcluater_ram1/install/hadoop/router/bin> ./hdfs dfs -rm
> /tmp/mapred.cmd
> 18/11/30 11:01:02 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using
[jira] [Commented] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
[
https://issues.apache.org/jira/browse/HDDS-1439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818060#comment-16818060
]
Ajay Kumar commented on HDDS-1439:
--
{code}2019-04-15 06:54:00 INFO SparkContext:54 - Added JAR
file:/opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar at
spark://34e995c8c4a5:37467/jars/spark-examples_2.11-2.3.0.jar with timestamp
1555311240685
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/crypto/key/KeyProviderTokenIssuer
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at
org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:121)
at
org.apache.spark.deploy.yarn.Client$$anonfun$5.apply(Client.scala:121)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.deploy.yarn.Client.(Client.scala:121)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)
at org.apache.spark.SparkContext.(SparkContext.scala:500)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2486)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:930)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:921)
at scala.Option.getOrElse(Option.scala:121)
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:921)
at
org.apache.spark.examples.DFSReadWriteTest$.main(DFSReadWriteTest.scala:106)
at
org.apache.spark.examples.DFSReadWriteTest.main(DFSReadWriteTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.crypto.key.KeyProviderTokenIssuer
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 49 more
2019-04-15 06:54:01 INFO DiskBlockManager:54 - Shutdown hook called{code}
> ozone spark job failing with class not found error for hadoop 2
> ---
>
> Key: HDDS-1439
> URL:
[jira] [Created] (HDDS-1439) ozone spark job failing with class not found error for hadoop 2
Ajay Kumar created HDDS-1439: Summary: ozone spark job failing with class not found error for hadoop 2 Key: HDDS-1439 URL: https://issues.apache.org/jira/browse/HDDS-1439 Project: Hadoop Distributed Data Store Issue Type: Sub-task Affects Versions: 0.4.0 Reporter: Ajay Kumar Assignee: Ajay Kumar spark job fails to run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818043#comment-16818043
]
Hadoop QA commented on HDFS-10659:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-10659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965937/HDFS-10659.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f4e270542d7f 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c4c16ca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26634/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26634/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Work logged] (HDDS-1192) Support -conf command line argument in GenericCli
[ https://issues.apache.org/jira/browse/HDDS-1192?focusedWorklogId=227733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227733 ] ASF GitHub Bot logged work on HDDS-1192: Author: ASF GitHub Bot Created on: 15/Apr/19 14:22 Start Date: 15/Apr/19 14:22 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #713: HDDS-1192. Support -conf command line argument in GenericCli URL: https://github.com/apache/hadoop/pull/713#issuecomment-483272310 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 23 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 168 | Maven dependency ordering for branch | | +1 | mvninstall | 1167 | trunk passed | | +1 | compile | | trunk passed | | +1 | checkstyle | 144 | trunk passed | | +1 | mvnsite | 246 | trunk passed | | +1 | shadedclient | 1150 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 219 | trunk passed | | +1 | javadoc | 164 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 178 | the patch passed | | +1 | compile | 907 | the patch passed | | +1 | javac | 907 | the patch passed | | +1 | checkstyle | 140 | the patch passed | | +1 | mvnsite | 194 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 739 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 251 | the patch passed | | +1 | javadoc | 159 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 90 | common in the patch passed. | | +1 | unit | 69 | container-service in the patch passed. | | +1 | unit | 30 | tools in the patch passed. | | -1 | unit | 818 | integration-test in the patch failed. | | +1 | unit | 53 | ozone-manager in the patch passed. | | +1 | asflicense | 46 | The patch does not generate ASF License warnings. | | | | 7970 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/713 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ad6d81684bbe 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c4c16ca | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/testReport/ | | Max. process+thread count | 4275 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service hadoop-hdds/tools hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-713/4/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227733) Time Spent: 1h 10m (was: 1h) > Support -conf command line argument in GenericCli > - > > Key: HDDS-1192 > URL: https://issues.apache.org/jira/browse/HDDS-1192 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Kitti Nanasi >Priority: Major > Labels: newbie, pull-request-available > Attachments: HDDS-1192.001.patch, HDDS-1192.002.patch,
[jira] [Commented] (HDFS-14421) HDFS block two replicas exist in one DataNode
[
https://issues.apache.org/jira/browse/HDFS-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818000#comment-16818000
]
He Xiaoqiao commented on HDFS-14421:
[~yuanbo], thanks for your comments, I am still confused that why same block
located more than one replicas in one datanode instance even if change VERSION
files of different disks which belong to one datanode instance. looks forward
to your more digging information.
> HDFS block two replicas exist in one DataNode
> -
>
> Key: HDFS-14421
> URL: https://issues.apache.org/jira/browse/HDFS-14421
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Priority: Major
> Attachments: 326942161.log
>
>
> We're using Hadoop-2.7.0.
> There is a file in the cluster and it's replication factor is 2. Those two
> replicas exist in one Datande. the fsck info is here:
> {color:#707070}BP-499819267-xx.xxx.131.201-1452072365222:blk_1400651575_326942161
> len=484045 repl=2
> [DatanodeInfoWithStorage[xx.xxx.80.205:50010,DS-d321be27-cbd4-4edd-81ad-29b3d021ee82,DISK],
>
> DatanodeInfoWithStorage[xx.xx.80.205:50010,DS-d321be27-cbd4-4edd-81ad-29b3d021ee82,DISK]].{color}
> and this is the exception from xx.xx.80.205
> {color:#707070}org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
> Replica not found for
> BP-499819267-xx.xxx.131.201-1452072365222:blk_1400651575_326942161{color}
> It's confusing that why NameNode doesn't update block map after exception.
> What's the reason of two replicas existing in one Datande?
> Hope to get your comments. Thanks in advance.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1192) Support -conf command line argument in GenericCli
[
https://issues.apache.org/jira/browse/HDDS-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817978#comment-16817978
]
Hadoop QA commented on HDDS-1192:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
2s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 32s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 54s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2663/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1192 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965936/HDDS-1192.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 204a68d96e82 4.4.0-138-generic #164-Ubuntu SMP
[jira] [Assigned] (HDDS-1436) TestCommitWatcher#testReleaseBuffersOnException fails with IllegalStateException
[
https://issues.apache.org/jira/browse/HDDS-1436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDDS-1436:
-
Assignee: Shashikant Banerjee
> TestCommitWatcher#testReleaseBuffersOnException fails with
> IllegalStateException
>
>
> Key: HDDS-1436
> URL: https://issues.apache.org/jira/browse/HDDS-1436
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.5.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: ozone-flaky-test
>
> the test is failing with the following exception
> {code}
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:129)
> at
> org.apache.hadoop.hdds.scm.storage.CommitWatcher.watchForCommit(CommitWatcher.java:191)
> at
> org.apache.hadoop.ozone.client.rpc.TestCommitWatcher.testReleaseBuffersOnException(TestCommitWatcher.java:277)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
> https://ci.anzix.net/job/ozone-nightly/63/testReport/org.apache.hadoop.ozone.client.rpc/TestCommitWatcher/testReleaseBuffersOnException/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[ https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227669 ] ASF GitHub Bot logged work on HDDS-1301: Author: ASF GitHub Bot Created on: 15/Apr/19 13:11 Start Date: 15/Apr/19 13:11 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #718: HDDS-1301. Optimize recursive ozone filesystem apis URL: https://github.com/apache/hadoop/pull/718#discussion_r275340555 ## File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/utils/db/DBStore.java ## @@ -120,6 +120,48 @@ Table source, Table dest) throws IOException; + /** + * Moves a key from the Source Table to the destination Table. + * + * @param key - Key to move. + * @param source - Source Table. + * @param dest - Destination Table. + * @throws IOException on Failure + */ + void moveWithBatch(KEY key, Table source, + Table dest, BatchOperation batch) throws IOException; + + /** + * Moves a key from the Source Table to the destination Table and updates the + * destination to the new value. + * + * @param key - Key to move. + * @param value - new value to write to the destination table. + * @param source - Source Table. + * @param dest - Destination Table. + * @throws IOException on Failure + */ + void moveWithBatch(KEY key, VALUE value, Table source, Review comment: There are no users of this function, can this be deleted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227669) Time Spent: 1h 10m (was: 1h) > Optimize recursive ozone filesystem apis > > > Key: HDDS-1301 > URL: https://issues.apache.org/jira/browse/HDDS-1301 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1301.001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > This Jira aims to optimise recursive apis in ozone file system. These are the > apis which have a recursive flag which requires an operation to be performed > on all the children of the directory. The Jira would add support for > recursive apis in Ozone manager in order to reduce the number of rpc calls to > Ozone Manager. Also currently these operations are not atomic. This Jira > would make all the operations in ozone filesystem atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227666
]
ASF GitHub Bot logged work on HDDS-1301:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:11
Start Date: 15/Apr/19 13:11
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #718: HDDS-1301.
Optimize recursive ozone filesystem apis
URL: https://github.com/apache/hadoop/pull/718#discussion_r275350966
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java
##
@@ -1546,6 +1559,358 @@ public OmKeyInfo lookupFile(OmKeyArgs args) throws
IOException {
ResultCodes.NOT_A_FILE);
}
+ /**
+ * Rename input file or directory.
+ *
+ * @param args Key args
+ * @param toKeyName new name for the file or directory. In case of a file it
+ * is the new absolute path for the file. In case of a
directory
+ * it is the absolute path of the new parent.
+ * @throws IOException if there is a rename from/to root
+ * For a file rename, if file or folder already exists at
+ * the destination
+ * For a directory rename, if file exists or destination
+ * directory is not empty or rename to a subdirectory
+ */
+ public void rename(OmKeyArgs args, String toKeyName) throws IOException {
+Preconditions.checkNotNull(args, "Key args can not be null");
+String volumeName = args.getVolumeName();
+String bucketName = args.getBucketName();
+String keyName = args.getKeyName();
+
+if (args.getKeyName().length() == 0 || toKeyName.length() == 0) {
+ throw new OMException("Can not rename from/to root directory",
+ ResultCodes.ROOT_DIRECTORY_RENAME_NOT_ALLOWED);
+}
+
+try {
+ metadataManager.getLock().acquireBucketLock(volumeName, bucketName);
+ OzoneFileStatus fromKeyStatus = getFileStatus(args);
+ OzoneFileStatus toKeyStatus = null;
+ OmKeyArgs toKeyArgs = new OmKeyArgs.Builder().setVolumeName(volumeName)
+ .setBucketName(bucketName).setKeyName(toKeyName).build();
+ try {
+toKeyStatus = getFileStatus(toKeyArgs);
+ } catch (OMException e) {
+if (e.getResult() != ResultCodes.FILE_NOT_FOUND) {
+ throw e;
+}
+ }
+ if (fromKeyStatus.isFile()) {
+renameFile(volumeName, bucketName, fromKeyStatus, toKeyStatus,
+toKeyName);
+ } else if (fromKeyStatus.isDirectory()) {
+renameDirectory(volumeName, bucketName, toKeyStatus,
+OzoneFSUtils.addTrailingSlashIfNeeded(keyName),
+OzoneFSUtils.addTrailingSlashIfNeeded(toKeyName));
+ }
+} finally {
+ metadataManager.getLock().releaseBucketLock(volumeName, bucketName);
+}
+ }
+
+ private void renameDirectory(String volumeName, String bucketName,
+ OzoneFileStatus toKeyStatus, String fromKeyName, String toKeyName)
+ throws IOException {
+if (fromKeyName.equals(toKeyName)) {
+ return;
+}
+if (toKeyStatus != null) {
+ if (toKeyStatus.isFile()) {
+throw new OMException(
+"Can not rename. File exists at destination " + toKeyName,
+ResultCodes.FILE_ALREADY_EXISTS);
+ } else if (toKeyStatus.isDirectory()) {
+if (!isDirectoryEmpty(volumeName, bucketName, toKeyName)) {
+ throw new OMException(
+ "Can not rename. Destination directory " + toKeyName
+ + " not empty", ResultCodes.DIRECTORY_NOT_EMPTY);
+}
+ }
+}
+String fromParent = OzoneFSUtils.getParent(fromKeyName);
+toKeyName = toKeyStatus != null ?
+toKeyName + fromKeyName.substring(fromParent.length()) : toKeyName;
+renamePrefix(volumeName, bucketName, fromKeyName, toKeyName);
+ }
+
+ private void renamePrefix(String volumeName, String bucketName,
+ String fromPrefix, String toPrefix) throws IOException {
+if (toPrefix.startsWith(fromPrefix)) {
+ throw new OMException("Can not rename to subdirectory",
+ ResultCodes.RENAME_TO_SUB_DIRECTORY_NOT_ALLOWED);
+}
+BatchOperation batch = metadataManager.getStore().initBatchOperation();
+TableIterator>
+iterator = metadataManager.getKeyTable().iterator();
+String fromPrefixInDb =
+metadataManager.getOzoneKey(volumeName, bucketName, fromPrefix);
+iterator.seek(fromPrefixInDb);
+while (iterator.hasNext()) {
+ Table.KeyValue kv = iterator.next();
+ String key = kv.getKey();
+ if (key.startsWith(fromPrefixInDb)) {
+renameKey(volumeName, bucketName, kv.getValue().getKeyName(),
+toPrefix + key.substring(fromPrefixInDb.length()), kv.getValue(),
+batch);
+ }
+}
+
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227668=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227668
]
ASF GitHub Bot logged work on HDDS-1301:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:11
Start Date: 15/Apr/19 13:11
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #718: HDDS-1301.
Optimize recursive ozone filesystem apis
URL: https://github.com/apache/hadoop/pull/718#discussion_r275343510
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -697,49 +346,16 @@ public String getUsername() {
* @throws IOException
*/
private boolean mkdir(Path path) throws IOException {
-Path fPart = path;
-Path prevfPart = null;
-do {
- LOG.trace("validating path:{}", fPart);
- try {
-FileStatus fileStatus = getFileStatus(fPart);
-if (fileStatus.isDirectory()) {
- // If path exists and a directory, exit
- break;
-} else {
- // Found a file here, rollback and delete newly created directories
- LOG.trace("Found a file with same name as directory, path:{}",
fPart);
- if (prevfPart != null) {
-delete(prevfPart, true);
- }
- throw new FileAlreadyExistsException(String.format(
- "Can't make directory for path '%s', it is a file.", fPart));
-}
- } catch (FileNotFoundException fnfe) {
-LOG.trace("creating directory for fpart:{}", fPart);
-String key = pathToKey(fPart);
-String dirKey = addTrailingSlashIfNeeded(key);
-if (!adapter.createDirectory(dirKey)) {
- // Directory creation failed here,
- // rollback and delete newly created directories
- LOG.trace("Directory creation failed, path:{}", fPart);
- if (prevfPart != null) {
-delete(prevfPart, true);
- }
- return false;
-}
- }
- prevfPart = fPart;
- fPart = fPart.getParent();
-} while (fPart != null);
+String key = pathToKey(path);
+adapter.createDirectory(key);
Review comment:
Lets return the value of the apapter.createDirectory here ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227668)
Time Spent: 1h (was: 50m)
> Optimize recursive ozone filesystem apis
>
>
> Key: HDDS-1301
> URL: https://issues.apache.org/jira/browse/HDDS-1301
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1301.001.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This Jira aims to optimise recursive apis in ozone file system. These are the
> apis which have a recursive flag which requires an operation to be performed
> on all the children of the directory. The Jira would add support for
> recursive apis in Ozone manager in order to reduce the number of rpc calls to
> Ozone Manager. Also currently these operations are not atomic. This Jira
> would make all the operations in ozone filesystem atomic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[ https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227670 ] ASF GitHub Bot logged work on HDDS-1301: Author: ASF GitHub Bot Created on: 15/Apr/19 13:11 Start Date: 15/Apr/19 13:11 Worklog Time Spent: 10m Work Description: mukul1987 commented on pull request #718: HDDS-1301. Optimize recursive ozone filesystem apis URL: https://github.com/apache/hadoop/pull/718#discussion_r275344671 ## File path: hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java ## @@ -50,6 +51,13 @@ OzoneFSOutputStream createFile(String key, boolean overWrite, Review comment: There are lots of unused functions in this file which can be removed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 227670) Time Spent: 1h 20m (was: 1h 10m) > Optimize recursive ozone filesystem apis > > > Key: HDDS-1301 > URL: https://issues.apache.org/jira/browse/HDDS-1301 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1301.001.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > This Jira aims to optimise recursive apis in ozone file system. These are the > apis which have a recursive flag which requires an operation to be performed > on all the children of the directory. The Jira would add support for > recursive apis in Ozone manager in order to reduce the number of rpc calls to > Ozone Manager. Also currently these operations are not atomic. This Jira > would make all the operations in ozone filesystem atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227672=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227672
]
ASF GitHub Bot logged work on HDDS-1301:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:11
Start Date: 15/Apr/19 13:11
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #718: HDDS-1301.
Optimize recursive ozone filesystem apis
URL: https://github.com/apache/hadoop/pull/718#discussion_r275343824
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -697,49 +346,16 @@ public String getUsername() {
* @throws IOException
*/
private boolean mkdir(Path path) throws IOException {
-Path fPart = path;
-Path prevfPart = null;
-do {
- LOG.trace("validating path:{}", fPart);
- try {
-FileStatus fileStatus = getFileStatus(fPart);
-if (fileStatus.isDirectory()) {
- // If path exists and a directory, exit
- break;
-} else {
- // Found a file here, rollback and delete newly created directories
- LOG.trace("Found a file with same name as directory, path:{}",
fPart);
- if (prevfPart != null) {
-delete(prevfPart, true);
- }
- throw new FileAlreadyExistsException(String.format(
- "Can't make directory for path '%s', it is a file.", fPart));
-}
- } catch (FileNotFoundException fnfe) {
-LOG.trace("creating directory for fpart:{}", fPart);
-String key = pathToKey(fPart);
-String dirKey = addTrailingSlashIfNeeded(key);
-if (!adapter.createDirectory(dirKey)) {
- // Directory creation failed here,
- // rollback and delete newly created directories
- LOG.trace("Directory creation failed, path:{}", fPart);
- if (prevfPart != null) {
-delete(prevfPart, true);
- }
- return false;
-}
- }
- prevfPart = fPart;
- fPart = fPart.getParent();
-} while (fPart != null);
+String key = pathToKey(path);
+adapter.createDirectory(key);
return true;
}
@Override
public boolean mkdirs(Path f, FsPermission permission) throws IOException {
LOG.trace("mkdir() path:{} ", f);
String key = pathToKey(f);
-if (isEmpty(key)) {
+if (StringUtils.isEmpty(key)) {
Review comment:
Can we use the earlier isEmpty here ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227672)
Time Spent: 1h 40m (was: 1.5h)
> Optimize recursive ozone filesystem apis
>
>
> Key: HDDS-1301
> URL: https://issues.apache.org/jira/browse/HDDS-1301
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1301.001.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This Jira aims to optimise recursive apis in ozone file system. These are the
> apis which have a recursive flag which requires an operation to be performed
> on all the children of the directory. The Jira would add support for
> recursive apis in Ozone manager in order to reduce the number of rpc calls to
> Ozone Manager. Also currently these operations are not atomic. This Jira
> would make all the operations in ozone filesystem atomic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227671
]
ASF GitHub Bot logged work on HDDS-1301:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:11
Start Date: 15/Apr/19 13:11
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #718: HDDS-1301.
Optimize recursive ozone filesystem apis
URL: https://github.com/apache/hadoop/pull/718#discussion_r275343085
##
File path:
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
##
@@ -319,337 +260,45 @@ boolean processKey(String key) throws IOException {
public boolean rename(Path src, Path dst) throws IOException {
incrementCounter(Statistic.INVOCATION_RENAME);
statistics.incrementWriteOps(1);
-if (src.equals(dst)) {
- return true;
-}
-
LOG.trace("rename() from:{} to:{}", src, dst);
-if (src.isRoot()) {
- // Cannot rename root of file system
- LOG.trace("Cannot rename the root of a filesystem");
- return false;
-}
-
-// Cannot rename a directory to its own subdirectory
-Path dstParent = dst.getParent();
-while (dstParent != null && !src.equals(dstParent)) {
- dstParent = dstParent.getParent();
-}
-Preconditions.checkArgument(dstParent == null,
-"Cannot rename a directory to its own subdirectory");
-// Check if the source exists
-FileStatus srcStatus;
-try {
- srcStatus = getFileStatus(src);
-} catch (FileNotFoundException fnfe) {
- // source doesn't exist, return
- return false;
-}
-
-// Check if the destination exists
-FileStatus dstStatus;
-try {
- dstStatus = getFileStatus(dst);
-} catch (FileNotFoundException fnde) {
- dstStatus = null;
-}
-
-if (dstStatus == null) {
- // If dst doesn't exist, check whether dst parent dir exists or not
- // if the parent exists, the source can still be renamed to dst path
- dstStatus = getFileStatus(dst.getParent());
- if (!dstStatus.isDirectory()) {
-throw new IOException(String.format(
-"Failed to rename %s to %s, %s is a file", src, dst,
-dst.getParent()));
- }
-} else {
- // if dst exists and source and destination are same,
- // check both the src and dst are of same type
- if (srcStatus.getPath().equals(dstStatus.getPath())) {
-return !srcStatus.isDirectory();
- } else if (dstStatus.isDirectory()) {
-// If dst is a directory, rename source as subpath of it.
-// for example rename /source to /dst will lead to /dst/source
-dst = new Path(dst, src.getName());
-FileStatus[] statuses;
-try {
- statuses = listStatus(dst);
-} catch (FileNotFoundException fnde) {
- statuses = null;
-}
-
-if (statuses != null && statuses.length > 0) {
- // If dst exists and not a directory not empty
- throw new FileAlreadyExistsException(String.format(
- "Failed to rename %s to %s, file already exists or not empty!",
- src, dst));
-}
- } else {
-// If dst is not a directory
-throw new FileAlreadyExistsException(String.format(
-"Failed to rename %s to %s, file already exists!", src, dst));
- }
-}
-
-if (srcStatus.isDirectory()) {
- if (dst.toString().startsWith(src.toString() + OZONE_URI_DELIMITER)) {
-LOG.trace("Cannot rename a directory to a subdirectory of self");
-return false;
- }
-}
-RenameIterator iterator = new RenameIterator(src, dst);
-return iterator.iterate();
- }
-
- private class DeleteIterator extends OzoneListingIterator {
-private boolean recursive;
-
-DeleteIterator(Path f, boolean recursive)
-throws IOException {
- super(f);
- this.recursive = recursive;
- if (getStatus().isDirectory()
- && !this.recursive
- && listStatus(f).length != 0) {
-throw new PathIsNotEmptyDirectoryException(f.toString());
- }
-}
-
-@Override
-boolean processKey(String key) throws IOException {
- if (key.equals("")) {
-LOG.trace("Skipping deleting root directory");
-return true;
- } else {
-LOG.trace("deleting key:" + key);
-boolean succeed = adapter.deleteObject(key);
-// if recursive delete is requested ignore the return value of
-// deleteObject and issue deletes for other keys.
-return recursive || succeed;
- }
-}
- }
-
- /**
- * Deletes the children of the input dir path by iterating though the
- * DeleteIterator.
- *
- * @param f directory path to be deleted
- * @return true if successfully deletes all required keys, false otherwise
- * @throws
[jira] [Work logged] (HDDS-1301) Optimize recursive ozone filesystem apis
[
https://issues.apache.org/jira/browse/HDDS-1301?focusedWorklogId=227667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-227667
]
ASF GitHub Bot logged work on HDDS-1301:
Author: ASF GitHub Bot
Created on: 15/Apr/19 13:11
Start Date: 15/Apr/19 13:11
Worklog Time Spent: 10m
Work Description: mukul1987 commented on pull request #718: HDDS-1301.
Optimize recursive ozone filesystem apis
URL: https://github.com/apache/hadoop/pull/718#discussion_r275336240
##
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##
@@ -599,6 +613,33 @@ message LookupFileResponse {
optional KeyInfo keyInfo = 1;
}
+message RenameFSEntryRequest {
+required KeyArgs keyArgs = 1;
+required string toKeyName = 2;
+}
+
+message RenameFSEntryResponse {
+}
+
+message DeleteFSEntryRequest {
+required KeyArgs keyArgs = 1;
+required bool recursive = 2;
+}
+
+message DeleteFSEntryResponse {
+}
+
+message ListStatusRequest {
+required KeyArgs keyArgs = 1;
+required bool recursive = 2;
+required string startKey = 3;
+required uint64 numEntries = 4;
+}
+
+message ListStatusResponse {
+repeated hadoop.fs.FileStatusProto statuses = 1;
Review comment:
Lets change this to repeated GetFileStatusResponse or lets introduce a new
OzoneFileStatus proto and use that both in GetFileStatus as well as List Status
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 227667)
Time Spent: 50m (was: 40m)
> Optimize recursive ozone filesystem apis
>
>
> Key: HDDS-1301
> URL: https://issues.apache.org/jira/browse/HDDS-1301
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-1301.001.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This Jira aims to optimise recursive apis in ozone file system. These are the
> apis which have a recursive flag which requires an operation to be performed
> on all the children of the directory. The Jira would add support for
> recursive apis in Ozone manager in order to reduce the number of rpc calls to
> Ozone Manager. Also currently these operations are not atomic. This Jira
> would make all the operations in ozone filesystem atomic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817936#comment-16817936
]
star commented on HDFS-10659:
-
[~hanishakoneru] thanks.
[~jojochuang], patch for trunk uploaded. Mind make a review?
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch,
> HDFS-10659.005.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
star updated HDFS-10659:
Attachment: HDFS-10659.005.patch
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch,
> HDFS-10659.005.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1409) TestOzoneClientRetriesOnException is flaky
[
https://issues.apache.org/jira/browse/HDDS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817917#comment-16817917
]
Hadoop QA commented on HDDS-1409:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 4m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 13s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 4s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.ozone.container.TestContainerReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2662/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1409 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965927/HDDS-1409.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b991e7c32125 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / c4c16ca |
| Default Java |
[jira] [Commented] (HDDS-1192) Support -conf command line argument in GenericCli
[ https://issues.apache.org/jira/browse/HDDS-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817916#comment-16817916 ] Kitti Nanasi commented on HDDS-1192: [~elek], could you review this? > Support -conf command line argument in GenericCli > - > > Key: HDDS-1192 > URL: https://issues.apache.org/jira/browse/HDDS-1192 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Kitti Nanasi >Priority: Major > Labels: newbie, pull-request-available > Attachments: HDDS-1192.001.patch, HDDS-1192.002.patch, > HDDS-1192.003.patch, HDDS-1192.004.patch, HDDS-1192.005.patch > > Time Spent: 1h > Remaining Estimate: 0h > > org.apache.hadoop.hdds.GenericCli is the based class for all the Ozone > related command line application. It supports to define custom configuration > variables (-D or --set) but doesn't support the '--conf ozone-site.xml' > argument to load an external xml file to the configuration. > Configuration and OzoneConfiguration classes load the ozone-site.xml from the > classpath. But it makes very hard to start Ozone components in IDE as we > can't modify the classpath easily. > One option here is to support the --conf everywhere to make it possible to > start ozone cluster in the IDE. > Note: It's a nice to have for 0.4.0. I marked it as 0.5.0 but safe to commit > at anytime to 0.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1192) Support -conf command line argument in GenericCli
[ https://issues.apache.org/jira/browse/HDDS-1192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kitti Nanasi updated HDDS-1192: --- Attachment: HDDS-1192.005.patch > Support -conf command line argument in GenericCli > - > > Key: HDDS-1192 > URL: https://issues.apache.org/jira/browse/HDDS-1192 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Kitti Nanasi >Priority: Major > Labels: newbie, pull-request-available > Attachments: HDDS-1192.001.patch, HDDS-1192.002.patch, > HDDS-1192.003.patch, HDDS-1192.004.patch, HDDS-1192.005.patch > > Time Spent: 1h > Remaining Estimate: 0h > > org.apache.hadoop.hdds.GenericCli is the based class for all the Ozone > related command line application. It supports to define custom configuration > variables (-D or --set) but doesn't support the '--conf ozone-site.xml' > argument to load an external xml file to the configuration. > Configuration and OzoneConfiguration classes load the ozone-site.xml from the > classpath. But it makes very hard to start Ozone components in IDE as we > can't modify the classpath easily. > One option here is to support the --conf everywhere to make it possible to > start ozone cluster in the IDE. > Note: It's a nice to have for 0.4.0. I marked it as 0.5.0 but safe to commit > at anytime to 0.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
star reassigned HDFS-10659:
---
Assignee: star (was: Hanisha Koneru)
> Namenode crashes after Journalnode re-installation in an HA cluster due to
> missing paxos directory
> --
>
> Key: HDFS-10659
> URL: https://issues.apache.org/jira/browse/HDFS-10659
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, journal-node
>Affects Versions: 2.7.0
>Reporter: Amit Anand
>Assignee: star
>Priority: Major
> Attachments: HDFS-10659.000.patch, HDFS-10659.001.patch,
> HDFS-10659.002.patch, HDFS-10659.003.patch, HDFS-10659.004.patch
>
>
> In my environment I am seeing {{Namenodes}} crashing down after majority of
> {{Journalnodes}} are re-installed. We manage multiple clusters and do rolling
> upgrades followed by rolling re-install of each node including master(NN, JN,
> RM, ZK) nodes. When a journal node is re-installed or moved to a new
> disk/host, instead of running {{"initializeSharedEdits"}} command, I copy
> {{VERSION}} file from one of the other {{Journalnode}} and that allows my
> {{NN}} to start writing data to the newly installed {{Journalnode}}.
> To acheive quorum for JN and recover unfinalized segments NN during starupt
> creates .tmp files under {{"/jn/current/paxos"}} directory . In
> current implementation "paxos" directry is only created during
> {{"initializeSharedEdits"}} command and if a JN is re-installed the "paxos"
> directory is not created upon JN startup or by NN while writing .tmp
> files which causes NN to crash with following error message:
> {code}
> 192.168.100.16:8485: /disk/1/dfs/jn/Test-Laptop/current/paxos/64044.tmp (No
> such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at
> org.apache.hadoop.hdfs.util.AtomicFileOutputStream.(AtomicFileOutputStream.java:58)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.persistPaxosData(Journal.java:971)
> at
> org.apache.hadoop.hdfs.qjournal.server.Journal.acceptRecovery(Journal.java:846)
> at
> org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.acceptRecovery(JournalNodeRpcServer.java:205)
> at
> org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.acceptRecovery(QJournalProtocolServerSideTranslatorPB.java:249)
> at
> org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25435)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
> {code}
> The current
> [getPaxosFile|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java#L128-L130]
> method simply returns a path to a file under "paxos" directory without
> verifiying its existence. Since "paxos" directoy holds files that are
> required for NN recovery and acheiving JN quorum my proposed solution is to
> add a check to "getPaxosFile" method and create the {{"paxos"}} directory if
> it is missing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10477) Stop decommission a rack of DataNodes caused NameNode fail over to standby
[
https://issues.apache.org/jira/browse/HDFS-10477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817908#comment-16817908
]
star commented on HDFS-10477:
-
[~jojochuang], [^HDFS-10477.branch-2.8.patch] is for branch 2.8.
Anything else should I do to make the patch committed?
> Stop decommission a rack of DataNodes caused NameNode fail over to standby
> --
>
> Key: HDFS-10477
> URL: https://issues.apache.org/jira/browse/HDFS-10477
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
>
> Attachments: HDFS-10477.002.patch, HDFS-10477.003.patch,
> HDFS-10477.004.patch, HDFS-10477.005.patch, HDFS-10477.006.patch,
> HDFS-10477.007.patch, HDFS-10477.branch-2.8.patch, HDFS-10477.branch-2.patch,
> HDFS-10477.patch
>
>
> In our cluster, when we stop decommissioning a rack which have 46 DataNodes,
> it locked Namesystem for about 7 minutes as below log shows:
> {code}
> 2016-05-26 20:11:41,697 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.27:1004
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 285258 over-replicated blocks on 10.142.27.27:1004 during recommissioning
> 2016-05-26 20:11:51,171 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.118:1004
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 279923 over-replicated blocks on 10.142.27.118:1004 during recommissioning
> 2016-05-26 20:11:59,972 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.113:1004
> 2016-05-26 20:12:09,007 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 294307 over-replicated blocks on 10.142.27.113:1004 during recommissioning
> 2016-05-26 20:12:09,008 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.117:1004
> 2016-05-26 20:12:18,055 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 314381 over-replicated blocks on 10.142.27.117:1004 during recommissioning
> 2016-05-26 20:12:18,056 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.130:1004
> 2016-05-26 20:12:25,938 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 272779 over-replicated blocks on 10.142.27.130:1004 during recommissioning
> 2016-05-26 20:12:25,939 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.121:1004
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 287248 over-replicated blocks on 10.142.27.121:1004 during recommissioning
> 2016-05-26 20:12:34,134 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.33:1004
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 299868 over-replicated blocks on 10.142.27.33:1004 during recommissioning
> 2016-05-26 20:12:43,020 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.137:1004
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 303914 over-replicated blocks on 10.142.27.137:1004 during recommissioning
> 2016-05-26 20:12:52,220 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.51:1004
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 281175 over-replicated blocks on 10.142.27.51:1004 during recommissioning
> 2016-05-26 20:13:00,362 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.12:1004
> 2016-05-26 20:13:08,756 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 274880 over-replicated blocks on 10.142.27.12:1004 during recommissioning
> 2016-05-26 20:13:08,757 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.15:1004
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Invalidated
> 286334 over-replicated blocks on 10.142.27.15:1004 during recommissioning
> 2016-05-26 20:13:17,185 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Stop
> Decommissioning 10.142.27.14:1004
> 2016-05-26
[jira] [Created] (HDFS-14429) Block remain in COMMITTED but not COMPLETE cause by Decommission
Yicong Cai created HDFS-14429:
-
Summary: Block remain in COMMITTED but not COMPLETE cause by
Decommission
Key: HDFS-14429
URL: https://issues.apache.org/jira/browse/HDFS-14429
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.9.2
Reporter: Yicong Cai
In the following scenario, the Block will remain in the COMMITTED but not
COMPLETE state and cannot be closed properly:
# Client writes Block(bk1) to three data nodes (dn1/dn2/dn3).
# bk1 has been completely written to three data nodes, and the data node
succeeds FinalizeBlock, joins IBR and waits to report to NameNode.
# The client commits bk1 after receiving the ACK.
# When the DN has not been reported to the IBR, all three nodes dn1/dn2/dn3
enter Decommissioning.
# The DN reports the IBR, but the block cannot be completed normally.
Then it will lead to the following related exceptions:
{panel:title=Exception}
2019-04-02 13:40:31,882 INFO namenode.FSNamesystem
(FSNamesystem.java:checkBlocksComplete(2790)) - BLOCK*
blk_4313483521_3245321090 is COMMITTED but not COMPLETE(numNodes= 3 >= minimum
= 1) in file xxx
2019-04-02 13:40:31,882 INFO ipc.Server (Server.java:logException(2650)) - IPC
Server handler 499 on 8020, call Call#122552 Retry#0
org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from xxx:47615
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
replicated yet: xxx
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:171)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2579)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:846)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:510)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:503)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:871)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:817)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2606)
{panel}
And will cause the scenario described in HDFS-12747
The root cause is that addStoredBlock does not consider the case where the
replications are in Decommission.
This problem needs to be fixed like HDFS-11499.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14425) Native build fails on macos due to jlong in hdfs.c
[
https://issues.apache.org/jira/browse/HDFS-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817849#comment-16817849
]
Hadoop QA commented on HDFS-14425:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
43s{color} | {color:green} hadoop-hdfs-native-client in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-741/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/741 |
| JIRA Issue | HDFS-14425 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux e06571e8483e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / c4c16ca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-741/1/testReport/ |
| Max. process+thread count | 411 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-741/1/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> Native build fails on macos due to jlong in hdfs.c
> --
>
> Key: HDFS-14425
> URL: https://issues.apache.org/jira/browse/HDFS-14425
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
>
> [WARNING]
> /Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:/Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:4949::
> warningwarning: : incompatible pointer types passing 'tOffset *' (aka 'long
> long *') to parameter of type 'jlong *' (aka
[jira] [Updated] (HDDS-1409) TestOzoneClientRetriesOnException is flaky
[
https://issues.apache.org/jira/browse/HDDS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chencan updated HDDS-1409:
--
Attachment: HDDS-1409.001.patch
> TestOzoneClientRetriesOnException is flaky
> --
>
> Key: HDDS-1409
> URL: https://issues.apache.org/jira/browse/HDDS-1409
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Nanda kumar
>Priority: Major
> Labels: ozone-flaky-test
> Attachments: HDDS-1409.001.patch
>
>
> TestOzoneClientRetriesOnException is flaky, we get the below exception when
> it fails.
> {noformat}
> [ERROR]
> testMaxRetriesByOzoneClient(org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException)
> Time elapsed: 16.227 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnException.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1409) TestOzoneClientRetriesOnException is flaky
[
https://issues.apache.org/jira/browse/HDDS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chencan updated HDDS-1409:
--
Status: Patch Available (was: Open)
> TestOzoneClientRetriesOnException is flaky
> --
>
> Key: HDDS-1409
> URL: https://issues.apache.org/jira/browse/HDDS-1409
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Nanda kumar
>Assignee: chencan
>Priority: Major
> Labels: ozone-flaky-test
> Attachments: HDDS-1409.001.patch
>
>
> TestOzoneClientRetriesOnException is flaky, we get the below exception when
> it fails.
> {noformat}
> [ERROR]
> testMaxRetriesByOzoneClient(org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException)
> Time elapsed: 16.227 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnException.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1409) TestOzoneClientRetriesOnException is flaky
[
https://issues.apache.org/jira/browse/HDDS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chencan reassigned HDDS-1409:
-
Assignee: chencan
> TestOzoneClientRetriesOnException is flaky
> --
>
> Key: HDDS-1409
> URL: https://issues.apache.org/jira/browse/HDDS-1409
> Project: Hadoop Distributed Data Store
> Issue Type: Test
> Components: test
>Reporter: Nanda kumar
>Assignee: chencan
>Priority: Major
> Labels: ozone-flaky-test
> Attachments: HDDS-1409.001.patch
>
>
> TestOzoneClientRetriesOnException is flaky, we get the below exception when
> it fails.
> {noformat}
> [ERROR]
> testMaxRetriesByOzoneClient(org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException)
> Time elapsed: 16.227 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException.testMaxRetriesByOzoneClient(TestOzoneClientRetriesOnException.java:197)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14077) DFSAdmin Report datanode Count was not matched when datanode in Decommissioned state
[
https://issues.apache.org/jira/browse/HDFS-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harshakiran Reddy resolved HDFS-14077.
--
Resolution: Fixed
> DFSAdmin Report datanode Count was not matched when datanode in
> Decommissioned state
> -
>
> Key: HDFS-14077
> URL: https://issues.apache.org/jira/browse/HDFS-14077
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.1.1
>Reporter: Harshakiran Reddy
>Assignee: Ranith Sardar
>Priority: Major
>
> {noformat}
> In DFSAdmin Reports showing the live datanodes are incorrect when some
> datanodes in Decommissioned State
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817679#comment-16817679 ] Kitti Nanasi commented on HDFS-14353: - Thanks [~maobaolong] for reporting this issue and providing a fix! Could you add a unit test? I think TestReconstructStripedFile could be extended with this check. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14425) Native build fails on macos due to jlong in hdfs.c
[
https://issues.apache.org/jira/browse/HDFS-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817668#comment-16817668
]
hunshenshi edited comment on HDFS-14425 at 4/15/19 9:03 AM:
[~jojochuang]
My env：
{code:java}
// gcc version
➜ ~ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
// operating system info
Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018;
root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64{code}
OK , I will offer a patch , Thanks
was (Author: hunhun):
[~jojochuang]
My env：
{code:java}
// gcc version
➜ ~ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
// operating system info
Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018;
root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64{code}
OK , I will offer a patch
> Native build fails on macos due to jlong in hdfs.c
> --
>
> Key: HDFS-14425
> URL: https://issues.apache.org/jira/browse/HDFS-14425
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
>
> [WARNING]
> /Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:/Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:4949::
> warningwarning: : incompatible pointer types passing 'tOffset *' (aka 'long
> long *') to parameter of type 'jlong *' (aka 'long *')
> [-Wincompatible-pointer-types]incompatible pointer types passing 'tOffset *'
> (aka 'long long *') to parameter of type 'jlong *' (aka 'long *')
> [-Wincompatible-pointer-types]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14425) Native build fails on macos due to jlong in hdfs.c
[
https://issues.apache.org/jira/browse/HDFS-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817668#comment-16817668
]
hunshenshi commented on HDFS-14425:
---
[~jojochuang]
My env：
{code:java}
// gcc version
➜ ~ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
// operating system info
Darwin 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018;
root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64{code}
OK , I will offer a patch
> Native build fails on macos due to jlong in hdfs.c
> --
>
> Key: HDFS-14425
> URL: https://issues.apache.org/jira/browse/HDFS-14425
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: hunshenshi
>Priority: Major
>
> [WARNING]
> /Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:/Users/xx/tmp/idea/hadoop-3.2.0-src/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:3033:4949::
> warningwarning: : incompatible pointer types passing 'tOffset *' (aka 'long
> long *') to parameter of type 'jlong *' (aka 'long *')
> [-Wincompatible-pointer-types]incompatible pointer types passing 'tOffset *'
> (aka 'long long *') to parameter of type 'jlong *' (aka 'long *')
> [-Wincompatible-pointer-types]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14421) HDFS block two replicas exist in one DataNode
[
https://issues.apache.org/jira/browse/HDFS-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817614#comment-16817614
]
Yuanbo Liu commented on HDFS-14421:
---
[~hexiaoqiao] I think I might find the reason. There are 12 disk in the
datanode, but somehow the VERSION files are the same, somebody has changed them
wrongly. I will comment later if I make the datanode run normally again.
> HDFS block two replicas exist in one DataNode
> -
>
> Key: HDFS-14421
> URL: https://issues.apache.org/jira/browse/HDFS-14421
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Priority: Major
> Attachments: 326942161.log
>
>
> We're using Hadoop-2.7.0.
> There is a file in the cluster and it's replication factor is 2. Those two
> replicas exist in one Datande. the fsck info is here:
> {color:#707070}BP-499819267-xx.xxx.131.201-1452072365222:blk_1400651575_326942161
> len=484045 repl=2
> [DatanodeInfoWithStorage[xx.xxx.80.205:50010,DS-d321be27-cbd4-4edd-81ad-29b3d021ee82,DISK],
>
> DatanodeInfoWithStorage[xx.xx.80.205:50010,DS-d321be27-cbd4-4edd-81ad-29b3d021ee82,DISK]].{color}
> and this is the exception from xx.xx.80.205
> {color:#707070}org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException:
> Replica not found for
> BP-499819267-xx.xxx.131.201-1452072365222:blk_1400651575_326942161{color}
> It's confusing that why NameNode doesn't update block map after exception.
> What's the reason of two replicas existing in one Datande?
> Hope to get your comments. Thanks in advance.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14428) NPE while DN is shutting down
[
https://issues.apache.org/jira/browse/HDFS-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated HDFS-14428:
-
Description:
At version 3.2.0, we meet exception while test
{code:java}
2019-04-14 23:24:16,041 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Unexpected exception in block pool Block pool
BP-2009994492-10.3.1.11-1555255450040 (Datanode Uuid
89d786bc-9b26-4c47-adcf-46197231e974) service to hadoop11/10.3.1.11:29000
java.lang.NullPointerException
at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:1213)
at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:3114)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:1782)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createBPRegistration(DataNode.java:1567)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.createRegistration(BPOfferService.java:453)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:764)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
{code:java}
2019-04-14 23:24:16,041 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Unexpected exception in block pool Block pool
BP-2009994492-10.3.1.11-1555255450040 (Datanode Uuid
89d786bc-9b26-4c47-adcf-46197231e974) service to hadoop11/10.3.1.11:29000
java.lang.NullPointerException
at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:1213)
at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:3114)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:1782)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createBPRegistration(DataNode.java:1567)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.createRegistration(BPOfferService.java:453)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:764)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819)
at java.lang.Thread.run(Thread.java:745)
{code}
> NPE while DN is shutting down
> -
>
> Key: HDFS-14428
> URL: https://issues.apache.org/jira/browse/HDFS-14428
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: lujie
>Priority: Major
>
> At version 3.2.0, we meet exception while test
> {code:java}
> 2019-04-14 23:24:16,041 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> Unexpected exception in block pool Block pool
> BP-2009994492-10.3.1.11-1555255450040 (Datanode Uuid
> 89d786bc-9b26-4c47-adcf-46197231e974) service to hadoop11/10.3.1.11:29000
> java.lang.NullPointerException
> at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:1213)
> at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:3114)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:1782)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createBPRegistration(DataNode.java:1567)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.createRegistration(BPOfferService.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:764)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14428) NPE while DN is shutting down
lujie created HDFS-14428:
Summary: NPE while DN is shutting down
Key: HDFS-14428
URL: https://issues.apache.org/jira/browse/HDFS-14428
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.2.0
Reporter: lujie
{code:java}
2019-04-14 23:24:16,041 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
Unexpected exception in block pool Block pool
BP-2009994492-10.3.1.11-1555255450040 (Datanode Uuid
89d786bc-9b26-4c47-adcf-46197231e974) service to hadoop11/10.3.1.11:29000
java.lang.NullPointerException
at org.apache.hadoop.ipc.Server$Listener.getAddress(Server.java:1213)
at org.apache.hadoop.ipc.Server.getListenerAddress(Server.java:3114)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.getIpcPort(DataNode.java:1782)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createBPRegistration(DataNode.java:1567)
at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.createRegistration(BPOfferService.java:453)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:764)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:286)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:819)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
