[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848026#comment-16848026 ] lindongdong commented on HDFS-12733: [~hexiaoqiao] Hi, Pls think about the old editlog remains in the local disk if we disable the local edits. When the NN restarts or disaster recovery, does it affect? > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?focusedWorklogId=248386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248386 ] ASF GitHub Bot logged work on HDDS-1231: Author: ASF GitHub Bot Created on: 25/May/19 02:58 Start Date: 25/May/19 02:58 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #851: HDDS-1231. Add ChillMode metrics. URL: https://github.com/apache/hadoop/pull/851#issuecomment-495833486 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 1 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 537 | trunk passed | | +1 | compile | 267 | trunk passed | | +1 | checkstyle | 78 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 931 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 148 | trunk passed | | 0 | spotbugs | 292 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 485 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 473 | the patch passed | | +1 | compile | 270 | the patch passed | | +1 | javac | 270 | the patch passed | | -0 | checkstyle | 40 | hadoop-hdds: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 750 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 147 | the patch passed | | +1 | findbugs | 499 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 165 | hadoop-hdds in the patch failed. | | -1 | unit | 1140 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 11627 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/851 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux aac9b558311c 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 55e0c13 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/artifact/out/diff-checkstyle-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/testReport/ | | Max. process+thread count | 4422 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm U: hadoop-hdds/server-scm | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-851/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248386) Time Spent: 20m (was: 10m) > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 20m
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848012#comment-16848012
]
Hadoop QA commented on HDFS-14403:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 19s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ipc.TestProcessingDetails |
| | hadoop.ipc.TestDecayRpcScheduler |
| | hadoop.ipc.TestRPC |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969719/HDFS-14403.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux da2e9b7ba78a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55e0c13 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/26842/artifact/out/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26842/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test
[jira] [Commented] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[ https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848000#comment-16848000 ] Siyao Meng commented on HDFS-14510: --- Uploaded patch for branch-2.7. Just in case. Though we may not commit it. [^HDFS-14510.branch-2.7.001.patch] 2.7.x is missing a series of refactoring commits: HDFS-7529, HDFS-8900, HDFS-8934, HDFS-9244. > Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information > should be immutable) > - > > Key: HDFS-14510 > URL: https://issues.apache.org/jira/browse/HDFS-14510 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption, snapshots >Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7 >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Fix For: 2.10.0, 2.8.6, 2.9.3 > > Attachments: HDFS-14510.branch-2.7.001.patch, > HDFS-14510.branch-2.8.001.patch > > > HDFS-13087 is a great fix. We should bring it into branch-2 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[ https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14510: -- Attachment: HDFS-14510.branch-2.7.001.patch > Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information > should be immutable) > - > > Key: HDFS-14510 > URL: https://issues.apache.org/jira/browse/HDFS-14510 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption, snapshots >Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7 >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Fix For: 2.10.0, 2.8.6, 2.9.3 > > Attachments: HDFS-14510.branch-2.7.001.patch, > HDFS-14510.branch-2.8.001.patch > > > HDFS-13087 is a great fix. We should bring it into branch-2 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847991#comment-16847991
]
Hadoop QA commented on HDFS-12979:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-12979 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12979 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969726/HDFS-12979.009.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26843/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> StandbyNode should upload FsImage to ObserverNode after checkpointing.
> --
>
> Key: HDFS-12979
> URL: https://issues.apache.org/jira/browse/HDFS-12979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Konstantin Shvachko
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch,
> HDFS-12979.003.patch, HDFS-12979.004.patch, HDFS-12979.005.patch,
> HDFS-12979.006.patch, HDFS-12979.007.patch, HDFS-12979.008.patch,
> HDFS-12979.009.patch
>
>
> ObserverNode does not create checkpoints. So it's fsimage file can get very
> old making bootstrap of ObserverNode too long. A StandbyNode should copy
> latest fsimage to ObserverNode(s) along with ANN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[ https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12979: -- Attachment: HDFS-12979.009.patch > StandbyNode should upload FsImage to ObserverNode after checkpointing. > -- > > Key: HDFS-12979 > URL: https://issues.apache.org/jira/browse/HDFS-12979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Konstantin Shvachko >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch, > HDFS-12979.003.patch, HDFS-12979.004.patch, HDFS-12979.005.patch, > HDFS-12979.006.patch, HDFS-12979.007.patch, HDFS-12979.008.patch, > HDFS-12979.009.patch > > > ObserverNode does not create checkpoints. So it's fsimage file can get very > old making bootstrap of ObserverNode too long. A StandbyNode should copy > latest fsimage to ObserverNode(s) along with ANN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12979) StandbyNode should upload FsImage to ObserverNode after checkpointing.
[
https://issues.apache.org/jira/browse/HDFS-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847984#comment-16847984
]
Chen Liang commented on HDFS-12979:
---
Thanks for the review [~xkrogen]. Post v009 patch.
Actually I was initially tweaking period and txn id to avoid having the new
variable. But the issue is some tests are already making assumptions about
these values for their own purpose, so changing these values can mess up the
test. And for some of the tests I needed to set these values very carefully:
being too small, checkpoint may get triggered too early, failing some checks;
being too large, the small delta can not be accepted, also failing the test. In
the end, it turned out to be even more messy/confusing and less readable. Also,
this makes it complex to add new unit test on this in the future. My thought
here was to let this behave as if having a new configuration key, except the
configuration is not exposed, but only used by tests. There are cases of
introducing variables for tests, such as
{{FSNameSystem#alwaysUseDelegationTokensForTests}}.
All other comments are addressed.
> StandbyNode should upload FsImage to ObserverNode after checkpointing.
> --
>
> Key: HDFS-12979
> URL: https://issues.apache.org/jira/browse/HDFS-12979
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Konstantin Shvachko
>Assignee: Chen Liang
>Priority: Major
> Attachments: HDFS-12979.001.patch, HDFS-12979.002.patch,
> HDFS-12979.003.patch, HDFS-12979.004.patch, HDFS-12979.005.patch,
> HDFS-12979.006.patch, HDFS-12979.007.patch, HDFS-12979.008.patch
>
>
> ObserverNode does not create checkpoints. So it's fsimage file can get very
> old making bootstrap of ObserverNode too long. A StandbyNode should copy
> latest fsimage to ObserverNode(s) along with ANN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14403) Cost-Based RPC FairCallQueue
[
https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847975#comment-16847975
]
Erik Krogen commented on HDFS-14403:
Now that HADOOP-16266 has been committed, we can resume progress here. I've
attached a v007 patch based on top of the HADOOP-16266 changes. One thing I
haven't done yet is change the variable names inside of {{DecayRpcScheduler}}
-- all of the {{callCount}} references need to be changed to {{callCost}}. I
think this will make it a bit harder to review, so I'll wait until folks have
time to do a first pass.
Ping [~linyiqun], [~csun], [~daryn] in case any of you are interested in taking
a look.
> Cost-Based RPC FairCallQueue
>
>
> Key: HDFS-14403
> URL: https://issues.apache.org/jira/browse/HDFS-14403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc, namenode
>Reporter: Erik Krogen
>Assignee: Christopher Gregorian
>Priority: Major
> Labels: qos, rpc
> Attachments: CostBasedFairCallQueueDesign_v0.pdf,
> HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch,
> HDFS-14403.004.patch, HDFS-14403.005.patch, HDFS-14403.006.combined.patch,
> HDFS-14403.006.patch, HDFS-14403.007.patch, HDFS-14403.branch-2.8.patch
>
>
> HADOOP-15016 initially described extensions to the Hadoop FairCallQueue
> encompassing both cost-based analysis of incoming RPCs, as well as support
> for reservations of RPC capacity for system/platform users. This JIRA intends
> to track the former, as HADOOP-15016 was repurposed to more specifically
> focus on the reservation portion of the work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14403) Cost-Based RPC FairCallQueue
[ https://issues.apache.org/jira/browse/HDFS-14403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HDFS-14403: --- Attachment: HDFS-14403.007.patch > Cost-Based RPC FairCallQueue > > > Key: HDFS-14403 > URL: https://issues.apache.org/jira/browse/HDFS-14403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ipc, namenode >Reporter: Erik Krogen >Assignee: Christopher Gregorian >Priority: Major > Labels: qos, rpc > Attachments: CostBasedFairCallQueueDesign_v0.pdf, > HDFS-14403.001.patch, HDFS-14403.002.patch, HDFS-14403.003.patch, > HDFS-14403.004.patch, HDFS-14403.005.patch, HDFS-14403.006.combined.patch, > HDFS-14403.006.patch, HDFS-14403.007.patch, HDFS-14403.branch-2.8.patch > > > HADOOP-15016 initially described extensions to the Hadoop FairCallQueue > encompassing both cost-based analysis of incoming RPCs, as well as support > for reservations of RPC capacity for system/platform users. This JIRA intends > to track the former, as HADOOP-15016 was repurposed to more specifically > focus on the reservation portion of the work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1231: - Target Version/s: 0.4.1 > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1231: - Fix Version/s: (was: 0.4.1) > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1231: - Fix Version/s: 0.4.1 > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1231: - Status: Patch Available (was: In Progress) > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?focusedWorklogId=248363=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248363 ] ASF GitHub Bot logged work on HDDS-1231: Author: ASF GitHub Bot Created on: 24/May/19 23:40 Start Date: 24/May/19 23:40 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #851: HDDS-1231. Add ChillMode metrics. URL: https://github.com/apache/hadoop/pull/851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248363) Time Spent: 10m Remaining Estimate: 0h > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1591) Protocol buffer classes are incompatible between builds
Eric Yang created HDDS-1591:
---
Summary: Protocol buffer classes are incompatible between builds
Key: HDDS-1591
URL: https://issues.apache.org/jira/browse/HDDS-1591
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Eric Yang
The generated classes are incompatible between builds. If developer run mvn
clean package to build the entire Ozone project and did not preserve the
generated jar files in maven cache. When he rebuild certain subproject, he
might end up using snapshot jar files from nightly build. However, the
protobuff message generated locally is not compatible with newly downloaded
protobuff classes from nightly build. This result in strange RPC behavior such
as:
{code:java}
2019-05-22 17:57:14,650 INFO retry.RetryInvocationHandler
(RetryInvocationHandler.java:log(411)) - com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(com.google.protobuf.InvalidProtocolBufferException):
Message missing required fields:
createVolumeRequest.volumeInfo.volumeAcls[0].rights,
createVolumeRequest.volumeInfo.volumeAcls[1].rights,
createVolumeRequest.volumeInfo.volumeAcls[2].rights,
createVolumeRequest.volumeInfo.volumeAcls[3].rights
at
com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
at
com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at
org.apache.hadoop.ipc.RpcWritable$ProtobufWrapper.readFrom(RpcWritable.java:125)
at org.apache.hadoop.ipc.RpcWritable$Buffer.getValue(RpcWritable.java:187)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:514)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over null(localhost:9862). Trying to
failover immediately.{code}
This is a codegen issue that version control of the generated code is not done
correctly, and result in incompatible RPC calls. There are two solutions,
using proto3 to remove mandatory fields, or get better at version control the
codegen process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1231) Add ChillMode metrics
[ https://issues.apache.org/jira/browse/HDDS-1231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1231: - Labels: pull-request-available (was: ) > Add ChillMode metrics > - > > Key: HDDS-1231 > URL: https://issues.apache.org/jira/browse/HDDS-1231 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This Jira is to add few of the chill mode metrics: > # NumberofHealthyPipelinesThreshold > # currentHealthyPipelinesCount > # NumberofPipelinesWithAtleastOneReplicaThreshold > # CurrentPipelinesWithAtleastOneReplicaCount > # ChillModeContainerWithOneReplicaReportedCutoff > # CurrentContainerCutoff > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1565) Rename k8s-dev and k8s-dev-push profiles to docker-build and docker-push
[ https://issues.apache.org/jira/browse/HDDS-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847956#comment-16847956 ] Eric Yang commented on HDDS-1565: - +1 on rename. However I prefer HDDS-1495 approach to separate docker image build from tarball image build. Build docker image as an separate submodule allows user selectively build tarball or docker image, or both. The current dist project setup does not allow selective rebuild. > Rename k8s-dev and k8s-dev-push profiles to docker-build and docker-push > > > Key: HDDS-1565 > URL: https://issues.apache.org/jira/browse/HDDS-1565 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Based on the feedback from [~eyang] I realized that the names of the k8s-dev > and k8s-dev-push profiles are not expressive enough as the created containers > can be used not only for kubernetes but can be used together with any other > container orchestrator. > I propose to rename them to docker-build/docker-push. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=248338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248338 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 24/May/19 22:34 Start Date: 24/May/19 22:34 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #850: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuf… URL: https://github.com/apache/hadoop/pull/850#issuecomment-495805274 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 33 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 15 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 63 | Maven dependency ordering for branch | | +1 | mvninstall | 541 | trunk passed | | +1 | compile | 291 | trunk passed | | +1 | checkstyle | 89 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 890 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 145 | trunk passed | | 0 | spotbugs | 290 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 475 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 26 | Maven dependency ordering for patch | | +1 | mvninstall | 478 | the patch passed | | +1 | compile | 280 | the patch passed | | +1 | cc | 280 | the patch passed | | +1 | javac | 280 | the patch passed | | -0 | checkstyle | 48 | hadoop-ozone: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 678 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 84 | hadoop-ozone generated 4 new + 5 unchanged - 0 fixed = 9 total (was 5) | | -1 | findbugs | 200 | hadoop-hdds generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | -1 | unit | 163 | hadoop-hdds in the patch failed. | | -1 | unit | 1255 | hadoop-ozone in the patch failed. | | +1 | asflicense | 61 | The patch does not generate ASF License warnings. | | | | 6430 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdds | | | Synchronization performed on java.util.concurrent.ConcurrentHashMap in org.apache.hadoop.utils.db.cache.PartialTableCache.evictCache(long) At PartialTableCache.java:org.apache.hadoop.utils.db.cache.PartialTableCache.evictCache(long) At PartialTableCache.java:[line 92] | | | Synchronization performed on java.util.concurrent.ConcurrentHashMap in org.apache.hadoop.utils.db.cache.PartialTableCache.put(CacheKey, CacheValue) At PartialTableCache.java:org.apache.hadoop.utils.db.cache.PartialTableCache.put(CacheKey, CacheValue) At PartialTableCache.java:[line 68] | | Failed junit tests | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.container.common.impl.TestContainerPersistence | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/850 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux a563a699b14a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 55e0c13 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/new-findbugs-hadoop-hdds.html | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-850/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results |
[jira] [Comment Edited] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847927#comment-16847927 ] Eric Yang edited comment on HDDS-1554 at 5/24/19 10:05 PM: --- Patch 001 requires HDDS-1458 patch 13 or newer. was (Author: eyang): This patch requires HDDS-1458 patch 13 or newer. > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847927#comment-16847927 ] Eric Yang commented on HDDS-1554: - This patch requires HDDS-1458 patch 13 or newer. > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1554) Create disk tests for fault injection test
[
https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847925#comment-16847925
]
Hadoop QA commented on HDDS-1554:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red} HDDS-1554 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDDS-1554 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969695/HDDS-1554.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2709/console |
| versions | git=1.9.1 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
> Create disk tests for fault injection test
> --
>
> Key: HDDS-1554
> URL: https://issues.apache.org/jira/browse/HDDS-1554
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: build
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1554.001.patch
>
>
> The current plan for fault injection disk tests are:
> # Scenario 1 - Read/Write test
> ## Run docker-compose to bring up a cluster
> ## Initialize scm and om
> ## Upload data to Ozone cluster
> ## Verify data is correct
> ## Shutdown cluster
> # Scenario 2 - Read/Only test
> ## Repeat Scenario 1
> ## Mount data disk as read only
> ## Try to write data to Ozone cluster
> ## Validate error message is correct
> ## Shutdown cluster
> # Scenario 3 - Corruption test
> ## Repeat Scenario 2
> ## Shutdown cluster
> ## Modify data disk data
> ## Restart cluster
> ## Validate error message for read from corrupted data
> ## Validate error message for write to corrupted volume
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1554: Status: Patch Available (was: Open) > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1554: Attachment: HDDS-1554.001.patch > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1554) Create disk tests for fault injection test
[ https://issues.apache.org/jira/browse/HDDS-1554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang reassigned HDDS-1554: --- Assignee: Eric Yang > Create disk tests for fault injection test > -- > > Key: HDDS-1554 > URL: https://issues.apache.org/jira/browse/HDDS-1554 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1554.001.patch > > > The current plan for fault injection disk tests are: > # Scenario 1 - Read/Write test > ## Run docker-compose to bring up a cluster > ## Initialize scm and om > ## Upload data to Ozone cluster > ## Verify data is correct > ## Shutdown cluster > # Scenario 2 - Read/Only test > ## Repeat Scenario 1 > ## Mount data disk as read only > ## Try to write data to Ozone cluster > ## Validate error message is correct > ## Shutdown cluster > # Scenario 3 - Corruption test > ## Repeat Scenario 2 > ## Shutdown cluster > ## Modify data disk data > ## Restart cluster > ## Validate error message for read from corrupted data > ## Validate error message for write to corrupted volume -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[ https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847915#comment-16847915 ] Siyao Meng commented on HDFS-14510: --- I agree most credit should be given to LiXin Ge. I'm merely manually rebasing the patch on to 2.x. Thanks for committing [~jojochuang]. > Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information > should be immutable) > - > > Key: HDFS-14510 > URL: https://issues.apache.org/jira/browse/HDFS-14510 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption, snapshots >Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7 >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Fix For: 2.10.0, 2.8.6, 2.9.3 > > Attachments: HDFS-14510.branch-2.8.001.patch > > > HDFS-13087 is a great fix. We should bring it into branch-2 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14502) keepResults option in NNThroughputBenchmark should call saveNamespace()
[
https://issues.apache.org/jira/browse/HDFS-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-14502:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
3.2.1
3.3.0
3.0.4
2.10.0
Status: Resolved (was: Patch Available)
I just committed this to trunk and all branches down to branch-2.
> keepResults option in NNThroughputBenchmark should call saveNamespace()
> ---
>
> Key: HDFS-14502
> URL: https://issues.apache.org/jira/browse/HDFS-14502
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, hdfs
>Affects Versions: 2.7.6
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14502.001.patch
>
>
> {{-keepResults}} option is usually used to make it possible to rerun
> NNThroughputBenchmark with existing namespace state {{-useExisting}}. E.g.
> first generate files with {{create}} command and then run {{fileStatus}}
> command on generated files.
> NNThroughputBenchmark should call {{saveNamespace()}} when {{-keepResults}}
> option is specified. Otherwise NN startup takes a while since it needs to
> digest large edits file starting from the empty image.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=248283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248283 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 24/May/19 21:00 Start Date: 24/May/19 21:00 Worklog Time Spent: 10m Work Description: arp7 commented on issue #827: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/827#issuecomment-495786562 Obsoleted by #850 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248283) Time Spent: 2h 40m (was: 2.5h) > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=248284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248284 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 24/May/19 21:00 Start Date: 24/May/19 21:00 Worklog Time Spent: 10m Work Description: arp7 commented on pull request #827: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/827 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248284) Time Spent: 2h 50m (was: 2h 40m) > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen resolved HDFS-14500.
Resolution: Fixed
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14500-branch-2.001.patch, HDFS-14500.000.patch,
> HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847867#comment-16847867
]
Erik Krogen edited comment on HDFS-14500 at 5/24/19 8:57 PM:
-
Just committed this to {{trunk}} and backported it to branch-3.2, branch-3.1,
branch-3.0, and branch-2. I had to make a minor modification for branch-2
because of the use of method references (Java 8 feature) in the patch, so I've
attached the updated patch. Thanks for the review [~vagarychen]!
was (Author: xkrogen):
Just committed this to {{trunk}}. Thanks for the review [~vagarychen]!
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14500-branch-2.001.patch, HDFS-14500.000.patch,
> HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14500:
---
Fix Version/s: 3.1.3
3.2.1
3.0.4
2.10.0
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14500-branch-2.001.patch, HDFS-14500.000.patch,
> HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14500:
---
Attachment: HDFS-14500-branch-2.001.patch
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14500-branch-2.001.patch, HDFS-14500.000.patch,
> HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?focusedWorklogId=248281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248281 ] ASF GitHub Bot logged work on HDDS-1551: Author: ASF GitHub Bot Created on: 24/May/19 20:45 Start Date: 24/May/19 20:45 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #850: HDDS-1551. Implement Bucket Write Requests to use Cache and DoubleBuf… URL: https://github.com/apache/hadoop/pull/850 …fer. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248281) Time Spent: 2.5h (was: 2h 20m) > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847879#comment-16847879
]
Hudson commented on HDFS-14500:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16604 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16604/])
HDFS-14500. NameNode StartupProgress should not allow new steps in an (xkrogen:
rev 55e0c134f002f74cb4a0360b6682a1b6796d1598)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/startupprogress/TestStartupProgress.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/startupprogress/StartupProgress.java
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14500.000.patch, HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1496) Support partial chunk reads and checksum verification
[
https://issues.apache.org/jira/browse/HDDS-1496?focusedWorklogId=248275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248275
]
ASF GitHub Bot logged work on HDDS-1496:
Author: ASF GitHub Bot
Created on: 24/May/19 20:27
Start Date: 24/May/19 20:27
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #804: HDDS-1496.
Support partial chunk reads and checksum verification
URL: https://github.com/apache/hadoop/pull/804#discussion_r287505228
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java
##
@@ -0,0 +1,536 @@
+package org.apache.hadoop.hdds.scm.storage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ReadChunkResponseProto;
+import org.apache.hadoop.hdds.scm.XceiverClientReply;
+import org.apache.hadoop.hdds.scm.XceiverClientSpi;
+import
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.common.Checksum;
+import org.apache.hadoop.ozone.common.ChecksumData;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+
+/**
+ * An {@link InputStream} used by the REST service in combination with the
+ * SCMClient to read the value of a key from a sequence of container chunks.
+ * All bytes of the key value are stored in container chunks. Each chunk may
+ * contain multiple underlying {@link ByteBuffer} instances. This class
+ * encapsulates all state management for iterating through the sequence of
+ * buffers within each chunk.
+ */
+public class ChunkInputStream extends InputStream implements Seekable {
+
+ private final ChunkInfo chunkInfo;
+ private final long length;
+ private final BlockID blockID;
+ private final String traceID;
+ private XceiverClientSpi xceiverClient;
+ private final boolean verifyChecksum;
+ private boolean allocated = false;
+
+ // Buffer to store the chunk data read from the DN container
+ private List buffers;
+
+ // Index of the buffers corresponding to the current postion of the buffers
+ private int bufferIndex;
+
+ // The offset of the current data residing in the buffers w.r.t the start
+ // of chunk data
+ private long bufferOffset;
+
+ // The number of bytes of chunk data residing in the buffers currently
+ private long bufferLength;
+
Review comment:
whitespace:end of line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248275)
Time Spent: 4h 40m (was: 4.5h)
> Support partial chunk reads and checksum verification
> -
>
> Key: HDDS-1496
> URL: https://issues.apache.org/jira/browse/HDDS-1496
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> BlockInputStream#readChunkFromContainer() reads the whole chunk from disk
> even if we need to read only a part of the chunk.
> This Jira aims to improve readChunkFromContainer so that only that part of
> the chunk file is read which is needed by client plus the part of chunk file
> which is required to verify the checksum.
> For example, lets say the client is reading from index 120 to 450 in the
> chunk. And let's say checksum is stored for every 100 bytes in the chunk i.e.
> the first checksum is for bytes from index 0 to 99, the next for bytes from
> index 100 to 199 and so on. To verify bytes from 120 to 450, we would need to
> read from bytes 100 to 499 so that checksum verification can be done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Work logged] (HDDS-1496) Support partial chunk reads and checksum verification
[ https://issues.apache.org/jira/browse/HDDS-1496?focusedWorklogId=248277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248277 ] ASF GitHub Bot logged work on HDDS-1496: Author: ASF GitHub Bot Created on: 24/May/19 20:27 Start Date: 24/May/19 20:27 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #804: HDDS-1496. Support partial chunk reads and checksum verification URL: https://github.com/apache/hadoop/pull/804#issuecomment-495778406 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 74 | Maven dependency ordering for branch | | +1 | mvninstall | 541 | trunk passed | | +1 | compile | 264 | trunk passed | | +1 | checkstyle | 78 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 944 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 158 | trunk passed | | 0 | spotbugs | 312 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 525 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 503 | the patch passed | | +1 | compile | 273 | the patch passed | | +1 | javac | 273 | the patch passed | | -0 | checkstyle | 40 | hadoop-hdds: The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) | | -0 | checkstyle | 41 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | -1 | whitespace | 0 | The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply | | +1 | shadedclient | 738 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 148 | the patch passed | | -1 | findbugs | 205 | hadoop-hdds generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | -1 | findbugs | 297 | hadoop-ozone generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | ||| _ Other Tests _ | | -1 | unit | 162 | hadoop-hdds in the patch failed. | | -1 | unit | 1182 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 6470 | | | Reason | Tests | |---:|:--| | FindBugs | module:hadoop-hdds | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.BlockInputStream.chunkIndex; locked 88% of time Unsynchronized access at BlockInputStream.java:88% of time Unsynchronized access at BlockInputStream.java:[line 379] | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.BlockInputStream.chunkOffsets; locked 88% of time Unsynchronized access at BlockInputStream.java:88% of time Unsynchronized access at BlockInputStream.java:[line 336] | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.BlockInputStream.chunkStreams; locked 92% of time Unsynchronized access at BlockInputStream.java:92% of time Unsynchronized access at BlockInputStream.java:[line 336] | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.ChunkInputStream.allocated; locked 50% of time Unsynchronized access at ChunkInputStream.java:50% of time Unsynchronized access at ChunkInputStream.java:[line 501] | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.ChunkInputStream.bufferLength; locked 50% of time Unsynchronized access at ChunkInputStream.java:50% of time Unsynchronized access at ChunkInputStream.java:[line 491] | | | Inconsistent synchronization of org.apache.hadoop.hdds.scm.storage.ChunkInputStream.bufferOffset; locked 53% of time Unsynchronized access at ChunkInputStream.java:53% of time Unsynchronized access at ChunkInputStream.java:[line 491] | | FindBugs | module:hadoop-ozone | | | Inconsistent synchronization of org.apache.hadoop.ozone.client.io.KeyInputStream.blockIndex; locked 92% of time Unsynchronized access at KeyInputStream.java:92% of time Unsynchronized access at KeyInputStream.java:[line 238] | | | Inconsistent synchronization of org.apache.hadoop.ozone.client.io.KeyInputStream.blockOffsets; locked 66% of time Unsynchronized access at KeyInputStream.java:66% of time Unsynchronized access at KeyInputStream.java:[line 91] | | Failed junit tests | hadoop.ozone.om.TestScmSafeMode | | |
[jira] [Work logged] (HDDS-1496) Support partial chunk reads and checksum verification
[
https://issues.apache.org/jira/browse/HDDS-1496?focusedWorklogId=248276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248276
]
ASF GitHub Bot logged work on HDDS-1496:
Author: ASF GitHub Bot
Created on: 24/May/19 20:27
Start Date: 24/May/19 20:27
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #804: HDDS-1496.
Support partial chunk reads and checksum verification
URL: https://github.com/apache/hadoop/pull/804#discussion_r287505208
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java
##
@@ -0,0 +1,536 @@
+package org.apache.hadoop.hdds.scm.storage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ReadChunkResponseProto;
+import org.apache.hadoop.hdds.scm.XceiverClientReply;
+import org.apache.hadoop.hdds.scm.XceiverClientSpi;
+import
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.common.Checksum;
+import org.apache.hadoop.ozone.common.ChecksumData;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+
+/**
+ * An {@link InputStream} used by the REST service in combination with the
+ * SCMClient to read the value of a key from a sequence of container chunks.
+ * All bytes of the key value are stored in container chunks. Each chunk may
+ * contain multiple underlying {@link ByteBuffer} instances. This class
+ * encapsulates all state management for iterating through the sequence of
+ * buffers within each chunk.
+ */
+public class ChunkInputStream extends InputStream implements Seekable {
+
+ private final ChunkInfo chunkInfo;
+ private final long length;
+ private final BlockID blockID;
+ private final String traceID;
+ private XceiverClientSpi xceiverClient;
+ private final boolean verifyChecksum;
+ private boolean allocated = false;
+
+ // Buffer to store the chunk data read from the DN container
+ private List buffers;
+
+ // Index of the buffers corresponding to the current postion of the buffers
+ private int bufferIndex;
+
Review comment:
whitespace:end of line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248276)
Time Spent: 4h 50m (was: 4h 40m)
> Support partial chunk reads and checksum verification
> -
>
> Key: HDDS-1496
> URL: https://issues.apache.org/jira/browse/HDDS-1496
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> BlockInputStream#readChunkFromContainer() reads the whole chunk from disk
> even if we need to read only a part of the chunk.
> This Jira aims to improve readChunkFromContainer so that only that part of
> the chunk file is read which is needed by client plus the part of chunk file
> which is required to verify the checksum.
> For example, lets say the client is reading from index 120 to 450 in the
> chunk. And let's say checksum is stored for every 100 bytes in the chunk i.e.
> the first checksum is for bytes from index 0 to 99, the next for bytes from
> index 100 to 199 and so on. To verify bytes from 120 to 450, we would need to
> read from bytes 100 to 499 so that checksum verification can be done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1496) Support partial chunk reads and checksum verification
[
https://issues.apache.org/jira/browse/HDDS-1496?focusedWorklogId=248274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248274
]
ASF GitHub Bot logged work on HDDS-1496:
Author: ASF GitHub Bot
Created on: 24/May/19 20:27
Start Date: 24/May/19 20:27
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #804: HDDS-1496.
Support partial chunk reads and checksum verification
URL: https://github.com/apache/hadoop/pull/804#discussion_r287505221
##
File path:
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ChunkInputStream.java
##
@@ -0,0 +1,536 @@
+package org.apache.hadoop.hdds.scm.storage;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import org.apache.hadoop.fs.Seekable;
+import org.apache.hadoop.hdds.client.BlockID;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerCommandResponseProto;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ChunkInfo;
+import
org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ReadChunkResponseProto;
+import org.apache.hadoop.hdds.scm.XceiverClientReply;
+import org.apache.hadoop.hdds.scm.XceiverClientSpi;
+import
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
+import org.apache.hadoop.ozone.common.Checksum;
+import org.apache.hadoop.ozone.common.ChecksumData;
+import org.apache.ratis.thirdparty.com.google.protobuf.ByteString;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutionException;
+
+/**
+ * An {@link InputStream} used by the REST service in combination with the
+ * SCMClient to read the value of a key from a sequence of container chunks.
+ * All bytes of the key value are stored in container chunks. Each chunk may
+ * contain multiple underlying {@link ByteBuffer} instances. This class
+ * encapsulates all state management for iterating through the sequence of
+ * buffers within each chunk.
+ */
+public class ChunkInputStream extends InputStream implements Seekable {
+
+ private final ChunkInfo chunkInfo;
+ private final long length;
+ private final BlockID blockID;
+ private final String traceID;
+ private XceiverClientSpi xceiverClient;
+ private final boolean verifyChecksum;
+ private boolean allocated = false;
+
+ // Buffer to store the chunk data read from the DN container
+ private List buffers;
+
+ // Index of the buffers corresponding to the current postion of the buffers
+ private int bufferIndex;
+
+ // The offset of the current data residing in the buffers w.r.t the start
+ // of chunk data
+ private long bufferOffset;
+
Review comment:
whitespace:end of line
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248274)
Time Spent: 4.5h (was: 4h 20m)
> Support partial chunk reads and checksum verification
> -
>
> Key: HDDS-1496
> URL: https://issues.apache.org/jira/browse/HDDS-1496
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> BlockInputStream#readChunkFromContainer() reads the whole chunk from disk
> even if we need to read only a part of the chunk.
> This Jira aims to improve readChunkFromContainer so that only that part of
> the chunk file is read which is needed by client plus the part of chunk file
> which is required to verify the checksum.
> For example, lets say the client is reading from index 120 to 450 in the
> chunk. And let's say checksum is stored for every 100 bytes in the chunk i.e.
> the first checksum is for bytes from index 0 to 99, the next for bytes from
> index 100 to 199 and so on. To verify bytes from 120 to 450, we would need to
> read from bytes 100 to 499 so that checksum verification can be done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1551) Implement Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1551: - Description: Implement Bucket write requests to use OM Cache, double buffer. And also in OM previously we used to Ratis client for communication to Ratis server, instead of that use Ratis server API's. In this Jira will add the changes to implement bucket operations, and HA/Non-HA will have a different code path, but once all requests are implemented will have a single code path. was: Implement Bucket write requests to use OM Cache, double buffer. And also in OM previously we used to Ratis client for communication to Ratis server, instead of that use Ratis server API's. # Implement checkAcl's method with new Request classes. As in Grpc Context, we shall not have UGI object, we need to set userName and remotehostAddress during pre-Execute step, and use this information to construct UGI and InetAddress and then call checkAcl. # Implement takeSnapshot once after flush to OM DB is completed. In this Jira will add the changes to implement bucket operations, and HA/Non-HA will have a different code path, but once all requests are implemented will have a single code path. > Implement Bucket Write Requests to use Cache and DoubleBuffer > - > > Key: HDDS-1551 > URL: https://issues.apache.org/jira/browse/HDDS-1551 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1580) Obtain Handler reference in ContainerScrubber
[
https://issues.apache.org/jira/browse/HDDS-1580?focusedWorklogId=248267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248267
]
ASF GitHub Bot logged work on HDDS-1580:
Author: ASF GitHub Bot
Created on: 24/May/19 20:26
Start Date: 24/May/19 20:26
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #842:
HDDS-1580.Obtain Handler reference in ContainerScrubber
URL: https://github.com/apache/hadoop/pull/842#discussion_r287504751
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerScrubber.java
##
@@ -130,12 +128,13 @@ private void throttleScrubber(TimeStamp startTime) {
private void scrub() {
-Iterator containerIt = containerSet.getContainerIterator();
+Iterator containerIt = controller.getContainerSetIterator();
long count = 0;
while (containerIt.hasNext()) {
TimeStamp startTime = new TimeStamp(System.currentTimeMillis());
Container container = containerIt.next();
+ Handler containerHandler = controller.getHandler(container);
Review comment:
containerHandler is not used here. Is it for upcoming patches?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248267)
Time Spent: 50m (was: 40m)
> Obtain Handler reference in ContainerScrubber
> -
>
> Key: HDDS-1580
> URL: https://issues.apache.org/jira/browse/HDDS-1580
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Affects Versions: 0.5.0
>Reporter: Shweta
>Assignee: Shweta
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Obtain reference to Handler based on containerType in scrub() in
> ContainerScrubber.java
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14486) The exception classes in some throw statements do not accurately describe why they are thrown
[
https://issues.apache.org/jira/browse/HDFS-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847868#comment-16847868
]
Hadoop QA commented on HDFS-14486:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14486 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969663/HDFS-14486-04.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 654e66071bab 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c9393dd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26841/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26841/testReport/ |
| Max. process+thread count | 2767 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26841/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
[jira] [Reopened] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen reopened HDFS-14500:
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14500.000.patch, HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14500:
---
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Just committed this to {{trunk}}. Thanks for the review [~vagarychen]!
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14500.000.patch, HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14502) keepResults option in NNThroughputBenchmark should call saveNamespace()
[
https://issues.apache.org/jira/browse/HDFS-14502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847855#comment-16847855
]
Hudson commented on HDFS-14502:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16603 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16603/])
HDFS-14502. keepResults option in NNThroughputBenchmark should call (shv: rev
c31b7b8d525b88cba7079b29d5ed9615286f5f59)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
> keepResults option in NNThroughputBenchmark should call saveNamespace()
> ---
>
> Key: HDFS-14502
> URL: https://issues.apache.org/jira/browse/HDFS-14502
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, hdfs
>Affects Versions: 2.7.6
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
>Priority: Major
> Attachments: HDFS-14502.001.patch
>
>
> {{-keepResults}} option is usually used to make it possible to rerun
> NNThroughputBenchmark with existing namespace state {{-useExisting}}. E.g.
> first generate files with {{create}} command and then run {{fileStatus}}
> command on generated files.
> NNThroughputBenchmark should call {{saveNamespace()}} when {{-keepResults}}
> option is specified. Otherwise NN startup takes a while since it needs to
> digest large edits file starting from the empty image.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1590) Add per pipeline metrics
Siddharth Wagle created HDDS-1590: - Summary: Add per pipeline metrics Key: HDDS-1590 URL: https://issues.apache.org/jira/browse/HDDS-1590 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: SCM Reporter: Siddharth Wagle Add metrics on per pipeline basis: - bytes read/written - container metrics (state changes) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14500) NameNode StartupProgress continues to report edit log segments after the LOADING_EDITS phase is finished
[
https://issues.apache.org/jira/browse/HDFS-14500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847796#comment-16847796
]
Chen Liang commented on HDFS-14500:
---
+1 to v001 patch.
> NameNode StartupProgress continues to report edit log segments after the
> LOADING_EDITS phase is finished
>
>
> Key: HDFS-14500
> URL: https://issues.apache.org/jira/browse/HDFS-14500
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 2.9.2, 3.0.3, 2.8.5, 3.1.2
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14500.000.patch, HDFS-14500.001.patch
>
>
> When testing out a cluster with the edit log tailing fast path feature
> enabled (HDFS-13150), an unrelated issue caused the NameNode to remain in
> safe mode for an extended period of time, preventing the NameNode from fully
> completing its startup sequence. We noticed that the Startup Progress web UI
> displayed many edit log segments (millions of them).
> I traced this problem back to {{StartupProgress}}. Within
> {{FSEditLogLoader}}, the loader continually tries to update the startup
> progress with a new {{Step}} any time that it loads edits. Per the Javadoc
> for {{StartupProgress}}, this should be a no-op once startup is completed:
> {code:title=StartupProgress.java}
> * After startup completes, the tracked data is frozen. Any subsequent
> updates
> * or counter increments are no-ops.
> {code}
> However, {{StartupProgress}} only implements that logic once the _entire_
> startup sequence has been completed. When {{FSEditLogLoader}} calls
> {{addStep()}}, it adds it into the {{LOADING_EDITS}} phase:
> {code:title=FSEditLogLoader.java}
> StartupProgress prog = NameNode.getStartupProgress();
> Step step = createStartupProgressStep(edits);
> prog.beginStep(Phase.LOADING_EDITS, step);
> {code}
> This phase, in our case, ended long before, so it is nonsensical to continue
> to add steps to it. I believe it is a bug that {{StartupProgress}} accepts
> such steps instead of ignoring them; once a phase is complete, it should no
> longer change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847788#comment-16847788 ] Hudson commented on HDDS-1512: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16602 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16602/]) HDDS-1512. Implement DoubleBuffer in OzoneManager. (#810) (github: rev 6d0e79c1216f342b4463131f6f997fea8e3ce247) * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/helpers/DoubleBufferEntry.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketDeleteResponse.java * (add) hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithOMResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMVolumeDeleteResponse.java * (add) hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBufferWithDummyResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMClientResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/package-info.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMBucketCreateResponse.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/helpers/package-info.java * (add) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/OMVolumeCreateResponse.java > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248186 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 18:00 Start Date: 24/May/19 18:00 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495729519 Thank You, everyone, for the review. I will commit this shortly. @linyiqun provided an explanation for doing this way. If you have any more comments, we can open new Jira and address them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248186) Time Spent: 10h 10m (was: 10h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h 10m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14451) Incorrect header or version mismatch log message
[
https://issues.apache.org/jira/browse/HDFS-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847784#comment-16847784
]
Shweta commented on HDFS-14451:
---
Thank you for the commit [~jojochuang].
> Incorrect header or version mismatch log message
>
>
> Key: HDFS-14451
> URL: https://issues.apache.org/jira/browse/HDFS-14451
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Shweta
>Priority: Minor
> Labels: noob
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14451.001.patch, HDFS-14451.002.patch
>
>
> {code:java|title=Server.java}
> if (!RpcConstants.HEADER.equals(dataLengthBuffer)
> || version != CURRENT_VERSION) {
> //Warning is ok since this is not supposed to happen.
> LOG.warn("Incorrect header or version mismatch from " +
> hostAddress + ":" + remotePort +
> " got version " + version +
> " expected version " + CURRENT_VERSION);
> setupBadVersionResponse(version);
> return -1;
> {code}
> This message should include the value of {{RpcConstants.HEADER}} and
> {{dataLengthBuffer}} in addition to just the version information or else that
> data is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1512: - Resolution: Fixed Fix Version/s: 0.5.0 Status: Resolved (was: Patch Available) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248176 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:51 Start Date: 24/May/19 17:51 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248176) Time Spent: 10h (was: 9h 50m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 10h > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248175 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:50 Start Date: 24/May/19 17:50 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495729519 Thank You, everyone, for the review. I will commit this shortly. @linyiqun provided my reason for doing this way. If you have any more comments, we can open new Jira and address them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248175) Time Spent: 9h 50m (was: 9h 40m) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 50m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1512) Implement DoubleBuffer in OzoneManager
[ https://issues.apache.org/jira/browse/HDDS-1512?focusedWorklogId=248169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248169 ] ASF GitHub Bot logged work on HDDS-1512: Author: ASF GitHub Bot Created on: 24/May/19 17:44 Start Date: 24/May/19 17:44 Worklog Time Spent: 10m Work Description: arp7 commented on issue #810: HDDS-1512. Implement DoubleBuffer in OzoneManager. URL: https://github.com/apache/hadoop/pull/810#issuecomment-495727082 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 248169) Time Spent: 9h 40m (was: 9.5h) > Implement DoubleBuffer in OzoneManager > -- > > Key: HDDS-1512 > URL: https://issues.apache.org/jira/browse/HDDS-1512 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 9h 40m > Remaining Estimate: 0h > > This Jira is created to implement DoubleBuffer in OzoneManager to flush > transactions to OM DB. > > h2. Flushing Transactions to RocksDB: > We propose using an implementation similar to the HDFS EditsDoubleBuffer. We > shall flush RocksDB transactions in batches, instead of current way of using > rocksdb.put() after every operation. At a given time only one batch will be > outstanding for flush while newer transactions are accumulated in memory to > be flushed later. > > In DoubleBuffer it will have 2 buffers one is currentBuffer, and the other is > readyBuffer. We add entry to current buffer, and we check if another flush > call is outstanding. If not, we flush to disk Otherwise we add entries to > otherBuffer while sync is happening. > > In this if sync is happening, we shall add new requests to other buffer and > when we can sync we use *RocksDB batch commit to sync to disk, instead of > rocksdb put.* > > Note: If flush to disk is failed on any OM, we shall terminate the > OzoneManager, so that OM DB’s will not diverge. Flush failure should be > considered as catastrophic failure. > > Scope of this Jira is to add DoubleBuffer implementation, integrating to > current OM will be done in further jira's. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14486) The exception classes in some throw statements do not accurately describe why they are thrown
[
https://issues.apache.org/jira/browse/HDFS-14486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-14486:
Attachment: HDFS-14486-04.patch
> The exception classes in some throw statements do not accurately describe why
> they are thrown
> -
>
> Key: HDFS-14486
> URL: https://issues.apache.org/jira/browse/HDFS-14486
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: eBugs
>Assignee: Ayush Saxena
>Priority: Minor
> Attachments: HDFS-14486-01.patch, HDFS-14486-02.patch,
> HDFS-14486-03.patch, HDFS-14486-04.patch
>
>
> Dear HDFS developers, we are developing a tool to detect exception-related
> bugs in Java. Our prototype has spotted a few {{throw}} statements whose
> exception class does not accurately describe why they are thrown. This can be
> dangerous since it makes correctly handling them challenging. For example, in
> an old bug, HDFS-8224, throwing a general {{IOException}} makes it difficult
> to perform data recovery specifically when a metadata file is corrupted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847767#comment-16847767 ] Hudson commented on HDFS-14353: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16601 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16601/]) Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to (inigoiri: rev c9393dd17f25ee62ceac0c66b96ce26b2789cc8a) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1588) Update Ozone dist image to use jdk11
[
https://issues.apache.org/jira/browse/HDDS-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847753#comment-16847753
]
Hadoop QA commented on HDDS-1588:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} hadolint {color} | {color:green} 0m
2s{color} | {color:green} The patch generated 0 new + 1 unchanged - 1 fixed = 1
total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 47s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 41s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestBCSID |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2708/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1588 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969660/HDDS-1588.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient hadolint shellcheck shelldocs |
| uname | Linux 58a37d80a3aa 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 460ba7f |
| Default Java | 1.8.0_212 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2708/artifact/out/patch-unit-hadoop-hdds.txt
[jira] [Reopened] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reopened HDFS-14353: > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847740#comment-16847740 ] Íñigo Goiri edited comment on HDFS-14353 at 5/24/19 5:22 PM: - [~maobaolong] can you take a look? It looks like the new wait for the number of xceivers is not happening. Actually, as [~ayushtkn] mentioned, this actually failed in our builds... my bad here. I'm reverting this. was (Author: elgoiri): [~maobaolong] can you take a look? It looks like the new wait for the number of xceivers is not happening. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847740#comment-16847740 ] Íñigo Goiri commented on HDFS-14353: [~maobaolong] can you take a look? It looks like the new wait for the number of xceivers is not happening. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch, > HDFS-14353.009.patch, screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847738#comment-16847738
]
Íñigo Goiri commented on HDFS-13255:
A couple minor comments:
* The new message when failing the delete could also report the mount points in
addition to the path.
* For the capturing of the expected exceptions let's use
LambaTestUtils#intercept and try to capture the message which ideally should
have the federated path.
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14451) Incorrect header or version mismatch log message
[
https://issues.apache.org/jira/browse/HDFS-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847728#comment-16847728
]
Hudson commented on HDFS-14451:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16599 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16599/])
HDFS-14451. Incorrect header or version mismatch log message. (weichiu: rev
93d7630062247793a4860704fad4a31437693de1)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
> Incorrect header or version mismatch log message
>
>
> Key: HDFS-14451
> URL: https://issues.apache.org/jira/browse/HDFS-14451
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Shweta
>Priority: Minor
> Labels: noob
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14451.001.patch, HDFS-14451.002.patch
>
>
> {code:java|title=Server.java}
> if (!RpcConstants.HEADER.equals(dataLengthBuffer)
> || version != CURRENT_VERSION) {
> //Warning is ok since this is not supposed to happen.
> LOG.warn("Incorrect header or version mismatch from " +
> hostAddress + ":" + remotePort +
> " got version " + version +
> " expected version " + CURRENT_VERSION);
> setupBadVersionResponse(version);
> return -1;
> {code}
> This message should include the value of {{RpcConstants.HEADER}} and
> {{dataLengthBuffer}} in addition to just the version information or else that
> data is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14475) RBF: Expose router security enabled status on the UI
[ https://issues.apache.org/jira/browse/HDFS-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847724#comment-16847724 ] CR Hota commented on HDFS-14475: [~tasanuma] Thanks for looking into this. I would just leave this as is and see how we can consolidate in the refactoring jira. As I had mentioned earlier, there is few things which should be re-looked at wrt metrics. For now federation can continue to expose status, token counts, security etc. Can you help start the conversation in HDFS-14508. Fundamentally since router is becoming more intelligent, it may be a good idea to expose metrics separately and not necessarily implement NamenodeBeanMetrics which has methods specific to namenode and not applicable to Routers. > RBF: Expose router security enabled status on the UI > > > Key: HDFS-14475 > URL: https://issues.apache.org/jira/browse/HDFS-14475 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14475-HDFS-13891.001.patch, > HDFS-14475-HDFS-13891.002.patch > > > This is a branched off Jira to expose metric so that router's security status > can be displayed on the UI. We are still unclear if more work needs to be > done for dealing with CORS etc. > https://issues.apache.org/jira/browse/HDFS-12510 will continue to track that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14451) Incorrect header or version mismatch log message
[
https://issues.apache.org/jira/browse/HDFS-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14451:
---
Resolution: Fixed
Fix Version/s: 3.1.3
3.2.1
3.3.0
Status: Resolved (was: Patch Available)
Pushed to trunk ~ branch-3.1. Thanks a lot!
> Incorrect header or version mismatch log message
>
>
> Key: HDFS-14451
> URL: https://issues.apache.org/jira/browse/HDFS-14451
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Shweta
>Priority: Minor
> Labels: noob
> Fix For: 3.3.0, 3.2.1, 3.1.3
>
> Attachments: HDFS-14451.001.patch, HDFS-14451.002.patch
>
>
> {code:java|title=Server.java}
> if (!RpcConstants.HEADER.equals(dataLengthBuffer)
> || version != CURRENT_VERSION) {
> //Warning is ok since this is not supposed to happen.
> LOG.warn("Incorrect header or version mismatch from " +
> hostAddress + ":" + remotePort +
> " got version " + version +
> " expected version " + CURRENT_VERSION);
> setupBadVersionResponse(version);
> return -1;
> {code}
> This message should include the value of {{RpcConstants.HEADER}} and
> {{dataLengthBuffer}} in addition to just the version information or else that
> data is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1589) CloseContainer transaction on unhealthy replica should fail with CONTAINER_UNHEALTHY exception
[
https://issues.apache.org/jira/browse/HDDS-1589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDDS-1589:
--
Description:
Currently, while trying to close an unhealthy container over Ratis, it fails
with INTERNAL_ERROR which leads to exception as follow:
{code:java}
2019-05-19 22:00:48,386 ERROR commandhandler.CloseContainerCommandHandler
(CloseContainerCommandHandler.java:handle(124)) - Can't close container #125
org.apache.ratis.protocol.StateMachineException:
java.util.concurrent.CompletionException from Server
faea26b0-9c60-4b4c-a0df-bf7c67cc5b48: java.lang.IllegalStateException
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$replyPendingRequest$24(RaftServerImpl.java:1221)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException:
java.lang.IllegalStateException
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
... 3 more
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:300)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$5(ContainerStateMachine.java:613)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
{code}
This happens when , it tries to mark the container unhealthy as the transaction
has failed and tries to mark the container unhealthy where it expects the
container to be in OPEN or CLOSIG state ad hence asserts. It should ideally
fail with CONTAINER_UNHEALTHY so as to not retry to not change the state to be
UNHEALTHY.
was:
Currently, while trying to close an unhealthy container over Ratis, it fails
with INTERNAL_ERROR which leads to exception as follow:
{code:java}
2019-05-19 22:00:48,386 ERROR commandhandler.CloseContainerCommandHandler
(CloseContainerCommandHandler.java:handle(124)) - Can't close container #125
org.apache.ratis.protocol.StateMachineException:
java.util.concurrent.CompletionException from Server
faea26b0-9c60-4b4c-a0df-bf7c67cc5b48: java.lang.IllegalStateException
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$replyPendingRequest$24(RaftServerImpl.java:1221)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException:
java.lang.IllegalStateException
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
... 3 more
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:300)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
[jira] [Updated] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[ https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14510: --- Resolution: Fixed Fix Version/s: 2.9.3 2.8.6 2.10.0 Status: Resolved (was: Patch Available) > Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information > should be immutable) > - > > Key: HDFS-14510 > URL: https://issues.apache.org/jira/browse/HDFS-14510 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption, snapshots >Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7 >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Fix For: 2.10.0, 2.8.6, 2.9.3 > > Attachments: HDFS-14510.branch-2.8.001.patch > > > HDFS-13087 is a great fix. We should bring it into branch-2 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1589) CloseContainer transaction on unhealthy replica should fail with CONTAINER_UNHEALTHY exception
Shashikant Banerjee created HDDS-1589:
-
Summary: CloseContainer transaction on unhealthy replica should
fail with CONTAINER_UNHEALTHY exception
Key: HDDS-1589
URL: https://issues.apache.org/jira/browse/HDDS-1589
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Datanode
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
Currently, while trying to close an unhealthy container over Ratis, it fails
with INTERNAL_ERROR which leads to exception as follow:
{code:java}
2019-05-19 22:00:48,386 ERROR commandhandler.CloseContainerCommandHandler
(CloseContainerCommandHandler.java:handle(124)) - Can't close container #125
org.apache.ratis.protocol.StateMachineException:
java.util.concurrent.CompletionException from Server
faea26b0-9c60-4b4c-a0df-bf7c67cc5b48: java.lang.IllegalStateException
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$replyPendingRequest$24(RaftServerImpl.java:1221)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CompletionException:
java.lang.IllegalStateException
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
... 3 more
Caused by: java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:300)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:149)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:347)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.runCommand(ContainerStateMachine.java:354)
at
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$applyTransaction$5(ContainerStateMachine.java:613)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
{code}
This happens when , it tries to mark the container unhealthy as the transaction
has failed and tries to mark the container unhealthy where it expects the
container to be in OPE or CLOSIG state ad hence asserts. It should ideally fail
with CONTAINER_UNHEATHY so as to not retry to not change the state to be
UNNHEATHY.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[
https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847707#comment-16847707
]
Wei-Chiu Chuang commented on HDFS-14510:
This is the commit message
I added Co-authored information according to
[https://help.github.com/en/articles/creating-a-commit-with-multiple-authors]
to give credits.
{noformat}
commit acbca66c22d9621985be5e6bf0c1b713df775f68
Author: Wei-Chiu Chuang
Date: Fri May 24 18:33:49 2019 +0200
HDFS-13087. Snapshotted encryption zone information should be immutable.
Contributed by LiXin Ge, Siyao Meng.
Signed-off-by: Wei-Chiu Chuang
Co-authored-by: Siyao Meng
(cherry picked from commit 380289a1678fea904d7c7a190dcbc3c4937c86fd)
(cherry picked from commit cd8b5e478071415638a55631c3e20ce2b18cd1fd)
{noformat}
> Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information
> should be immutable)
> -
>
> Key: HDFS-14510
> URL: https://issues.apache.org/jira/browse/HDFS-14510
> Project: Hadoop HDFS
> Issue Type: Task
> Components: encryption, snapshots
>Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14510.branch-2.8.001.patch
>
>
> HDFS-13087 is a great fix. We should bring it into branch-2 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1588) Update Ozone dist image to use jdk11
[ https://issues.apache.org/jira/browse/HDDS-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1588: Attachment: HDDS-1588.001.patch > Update Ozone dist image to use jdk11 > > > Key: HDDS-1588 > URL: https://issues.apache.org/jira/browse/HDDS-1588 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1588.001.patch > > > For some reason, Ozone RPC only works correctly with jdk11 in the container. > The distribution created docker image must match jdk11 for the system to > function correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1588) Update Ozone dist image to use jdk11
[ https://issues.apache.org/jira/browse/HDDS-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDDS-1588: Status: Patch Available (was: Open) > Update Ozone dist image to use jdk11 > > > Key: HDDS-1588 > URL: https://issues.apache.org/jira/browse/HDDS-1588 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Attachments: HDDS-1588.001.patch > > > For some reason, Ozone RPC only works correctly with jdk11 in the container. > The distribution created docker image must match jdk11 for the system to > function correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1588) Update Ozone dist image to use jdk11
Eric Yang created HDDS-1588: --- Summary: Update Ozone dist image to use jdk11 Key: HDDS-1588 URL: https://issues.apache.org/jira/browse/HDDS-1588 Project: Hadoop Distributed Data Store Issue Type: Bug Affects Versions: 0.5.0 Reporter: Eric Yang Assignee: Eric Yang For some reason, Ozone RPC only works correctly with jdk11 in the container. The distribution created docker image must match jdk11 for the system to function correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847649#comment-16847649
]
Eric Yang commented on HDDS-1458:
-
The failed unit tests are not related to patch 013.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Attachments: HDDS-1458.001.patch, HDDS-1458.002.patch,
> HDDS-1458.003.patch, HDDS-1458.004.patch, HDDS-1458.005.patch,
> HDDS-1458.006.patch, HDDS-1458.007.patch, HDDS-1458.008.patch,
> HDDS-1458.009.patch, HDDS-1458.010.patch, HDDS-1458.011.patch,
> HDDS-1458.012.patch, HDDS-1458.013.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14440) RBF: Optimize the file write process in case of multiple destinations.
[ https://issues.apache.org/jira/browse/HDFS-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HDFS-14440: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-13891 Status: Resolved (was: Patch Available) Committed. Thanx [~vinayakumarb] and [~elgoiri] for the review!!! > RBF: Optimize the file write process in case of multiple destinations. > -- > > Key: HDFS-14440 > URL: https://issues.apache.org/jira/browse/HDFS-14440 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14440-HDFS-13891-01.patch, > HDFS-14440-HDFS-13891-02.patch, HDFS-14440-HDFS-13891-03.patch, > HDFS-14440-HDFS-13891-04.patch, HDFS-14440-HDFS-13891-05.patch, > HDFS-14440-HDFS-13891-06.patch > > > In case of multiple destinations, We need to check if the file already exists > in one of the subclusters for which we use the existing getBlockLocation() > API which is by default a sequential Call, > In an ideal scenario where the file needs to be created each subcluster shall > be checked sequentially, this can be done concurrently to save time. > In another case where the file is found and if the last block is null, we > need to do getFileInfo to all the locations to get the location where the > file exists. This also can be prevented by use of ConcurrentCall since we > shall be having the remoteLocation to where the getBlockLocation returned a > non null entry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847633#comment-16847633
]
Ayush Saxena commented on HDFS-13255:
-
Thanx [~aajisaka] for the patch.
* There are couple of test failures you need to handle.
* IIUC regarding Fail remove if under mount point point, you tend to say if
/mnt is a mount point delete operation on /mnt that should fail. If so this
seems genuine because deleting that will make the mount entry nonoperational,
viewFs also I guess doesn't allow such operation, Guess it throws some
internalisReadOnly type exception. Regarding rename, need to check if there
leeds to any complications, well there is Jira tracking the rename operation
stuff. If something else also it restricts I guess we need to think regarding
the impacts.
* If thats only the stuff, may be you can extend a UT for the behaviour that
it stops delete on exact mount entries and others stay unaffected.
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847622#comment-16847622 ] Ayush Saxena commented on HDFS-12733: - Well technically this makes quite sense to me. In HA the local edits I don't find any as such usability for the local edits and can be good to have a configuration to disable them. I saw this patch few days back too for some issue, The patch is quite straightforward and seems fair enough [~brahmareddy] is the best person who can help with some good details or if there was any challenge here. :) > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[
https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847616#comment-16847616
]
Ayush Saxena commented on HDFS-14353:
-
The test changed in the patch
{{TestReconstructStripedFile.testErasureCodingWorkerXmitsWeight}} seems to be
failing after this :
Saw it in a recent build :
[https://builds.apache.org/job/PreCommit-HDFS-Build/26838/testReport/org.apache.hadoop.hdfs/TestReconstructStripedFile/testErasureCodingWorkerXmitsWeight/]
It has even failed in last three reports here too. Pls give a check.
> Erasure Coding: metrics xmitsInProgress become to negative.
> ---
>
> Key: HDFS-14353
> URL: https://issues.apache.org/jira/browse/HDFS-14353
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, erasure-coding
>Affects Versions: 3.3.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch,
> HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch,
> HDFS-14353.006.patch, HDFS-14353.007.patch, HDFS-14353.008.patch,
> HDFS-14353.009.patch, screenshot-1.png
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14451) Incorrect header or version mismatch log message
[
https://issues.apache.org/jira/browse/HDFS-14451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847582#comment-16847582
]
Wei-Chiu Chuang commented on HDFS-14451:
+1
> Incorrect header or version mismatch log message
>
>
> Key: HDFS-14451
> URL: https://issues.apache.org/jira/browse/HDFS-14451
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ipc
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Shweta
>Priority: Minor
> Labels: noob
> Attachments: HDFS-14451.001.patch, HDFS-14451.002.patch
>
>
> {code:java|title=Server.java}
> if (!RpcConstants.HEADER.equals(dataLengthBuffer)
> || version != CURRENT_VERSION) {
> //Warning is ok since this is not supposed to happen.
> LOG.warn("Incorrect header or version mismatch from " +
> hostAddress + ":" + remotePort +
> " got version " + version +
> " expected version " + CURRENT_VERSION);
> setupBadVersionResponse(version);
> return -1;
> {code}
> This message should include the value of {{RpcConstants.HEADER}} and
> {{dataLengthBuffer}} in addition to just the version information or else that
> data is lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14186) blockreport storm slow down namenode restart seriously in large cluster
[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847553#comment-16847553 ] He Xiaoqiao commented on HDFS-14186: ping [~kihwal],[~elgoiri],[~xkrogen],[~daryn], do you have time to give any further more comments or suggestions for this issue and solution? Thanks. > blockreport storm slow down namenode restart seriously in large cluster > --- > > Key: HDFS-14186 > URL: https://issues.apache.org/jira/browse/HDFS-14186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.1 >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14186.001.patch > > > In the current implementation, the datanode sends blockreport immediately > after register to namenode successfully when restart, and the blockreport > storm will make namenode high load to process them. One result is some > received RPC have to skip because queue time is timeout. If some datanodes' > heartbeat RPC are continually skipped for long times (default is > heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to > re-register and send blockreport again, aggravate blockreport storm and trap > in a vicious circle, and slow down (more than one hour and even more) > namenode startup seriously in a large (several thousands of datanodes) and > busy cluster especially. Although there are many work to optimize namenode > startup, the issue still exists. > I propose to postpone dead datanode check when namenode have finished startup. > Any comments and suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14356) Implement HDFS cache on SCM with native PMDK libs
[
https://issues.apache.org/jira/browse/HDFS-14356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847555#comment-16847555
]
Hadoop QA commented on HDFS-14356:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} root: The patch generated 0 new + 271 unchanged - 1
fixed = 271 total (was 272) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 12m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
14s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 56s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}148m 58s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
2s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}291m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Work logged] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[
https://issues.apache.org/jira/browse/HDDS-1530?focusedWorklogId=248016=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248016
]
ASF GitHub Bot logged work on HDDS-1530:
Author: ASF GitHub Bot
Created on: 24/May/19 12:54
Start Date: 24/May/19 12:54
Worklog Time Spent: 10m
Work Description: iamcaoxudong commented on pull request #830: HDDS-1530.
Freon support big files larger than 2GB and add --bufferSize and
--validateWrites options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285967860
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##
@@ -622,7 +642,11 @@ public void run() {
try (Scope writeScope = GlobalTracer.get()
.buildSpan("writeKeyData")
.startActive(true)) {
- os.write(keyValue);
+ for (long nrRemaining = keySize - randomValue.length;
+nrRemaining > 0; nrRemaining -= bufferSize) {
+int curSize = (int)Math.min(bufferSize, nrRemaining);
+os.write(keyValueBuffer, 0, curSize);
Review comment:
Thank you for reviewing, but I have some different opinions:
1. On the client machine, the buffer here will be sent to the Ratis DataNode
via TCP later, so the file system buffer/cache is not involved.
2. On the DataNode, the file system cache is managed by the file's Inode
index. since each key is written on DataNode through different files, so I
think it will not affected by file system buffer/cache too significant.
3. On the client machine, when the data is sent from user space to the
socket in the kernel via system call, or when the data is sent form kernel
socket to peer via TCP, in all two cases, there is no cache involved (this is
in my impression, if not right, please correct me).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248016)
Time Spent: 5h 10m (was: 5h)
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> --
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1530) Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and "--validateWrites" options.
[
https://issues.apache.org/jira/browse/HDDS-1530?focusedWorklogId=248015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-248015
]
ASF GitHub Bot logged work on HDDS-1530:
Author: ASF GitHub Bot
Created on: 24/May/19 12:54
Start Date: 24/May/19 12:54
Worklog Time Spent: 10m
Work Description: iamcaoxudong commented on pull request #830: HDDS-1530.
Freon support big files larger than 2GB and add --bufferSize and
--validateWrites options.
URL: https://github.com/apache/hadoop/pull/830#discussion_r285967860
##
File path:
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/freon/RandomKeyGenerator.java
##
@@ -622,7 +642,11 @@ public void run() {
try (Scope writeScope = GlobalTracer.get()
.buildSpan("writeKeyData")
.startActive(true)) {
- os.write(keyValue);
+ for (long nrRemaining = keySize - randomValue.length;
+nrRemaining > 0; nrRemaining -= bufferSize) {
+int curSize = (int)Math.min(bufferSize, nrRemaining);
+os.write(keyValueBuffer, 0, curSize);
Review comment:
Thank you for reviewing, but I have some different opinions:
1. On the client machine, the buffer here will be sent to the Ratis DataNode
via TCP later, so the file system buffer/cache is not involved.
2. On the DataNode, the file system cache is managed by the file's Inode
index. since each key is written on DataNode through different files, so I
think it will not affected by file system buffer/cache too significant.
3.On the client machine, when the data is sent from user space to the socket
in the kernel via system call, or when the data is sent form kernel socket to
peer via TCP, in all two cases, there is no cache involved (this is in my
impression, if not right, please correct me).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 248015)
Time Spent: 5h (was: 4h 50m)
> Ozone: Freon: Support big files larger than 2GB and add "--bufferSize" and
> "--validateWrites" options.
> --
>
> Key: HDDS-1530
> URL: https://issues.apache.org/jira/browse/HDDS-1530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Xudong Cao
>Assignee: Xudong Cao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> *Current problems:*
> 1. Freon does not support big files larger than 2GB because it use an int
> type "keySize" parameter and also "keyValue" buffer size.
> 2. Freon allocates a entire buffer for each key at once, so if the key size
> is large and the concurrency is high, freon will report OOM exception
> frequently.
> 3. Freon lacks option such as "--validateWrites", thus users cannot manually
> specify that verification is required after writing.
> *Some solutions:*
> 1. Use a long type "keySize" parameter, make sure freon can support big
> files larger than 2GB.
> 2. Use a small buffer repeatedly than allocating the entire key-size buffer
> at once, the default buffer size is 4K and can be configured by "–bufferSize"
> parameter.
> 3. Add a "--validateWrites" option to Freon command line, users can provide
> this option to indicate that a validation is required after write.
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14512) ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)
[ https://issues.apache.org/jira/browse/HDFS-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847523#comment-16847523 ] Wei-Chiu Chuang commented on HDFS-14512: Out of curiosity, why do you set favored node? Is it used by HBase or is it by some other applications? > ONE_SSD policy will be violated while write data with > DistributedFileSystem.create(favoredNodes) > > > Key: HDFS-14512 > URL: https://issues.apache.org/jira/browse/HDFS-14512 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shen Yinjie >Priority: Major > > Reproduce steps: > 1.setStoragePolicy ONE_SSD for a path A; > 2. client write data to path A by > DistributedFileSystem.create(...favoredNodes) and Passing parameter > favoredNodes > then, three replicas of file in this path will be located in 2 SSD and > 1DISK,which is violating the ONE_SSD policy. > Not sure am I clear? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[ https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847515#comment-16847515 ] Wei-Chiu Chuang commented on HDFS-14510: +1 Next time, please use branch-2 as the patch file name suffix. Thank you! > Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information > should be immutable) > - > > Key: HDFS-14510 > URL: https://issues.apache.org/jira/browse/HDFS-14510 > Project: Hadoop HDFS > Issue Type: Task > Components: encryption, snapshots >Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7 >Reporter: Wei-Chiu Chuang >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-14510.branch-2.8.001.patch > > > HDFS-13087 is a great fix. We should bring it into branch-2 as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14512) ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)
[ https://issues.apache.org/jira/browse/HDFS-14512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847506#comment-16847506 ] Ayush Saxena commented on HDFS-14512: - Didi you had enough Disks? > ONE_SSD policy will be violated while write data with > DistributedFileSystem.create(favoredNodes) > > > Key: HDFS-14512 > URL: https://issues.apache.org/jira/browse/HDFS-14512 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shen Yinjie >Priority: Major > > Reproduce steps: > 1.setStoragePolicy ONE_SSD for a path A; > 2. client write data to path A by > DistributedFileSystem.create(...favoredNodes) and Passing parameter > favoredNodes > then, three replicas of file in this path will be located in 2 SSD and > 1DISK,which is violating the ONE_SSD policy. > Not sure am I clear? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14510) Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information should be immutable)
[
https://issues.apache.org/jira/browse/HDFS-14510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847501#comment-16847501
]
Siyao Meng commented on HDFS-14510:
---
I applied the patch, compiled and ran test TestEncryptionZones and
TestOpenFilesWithSnapshot locally on branch-2.8, they passed:
{code:bash|title=TestEncryptionZones}
$ mvn test -fn -Dsurefire.printSummary -Dtest=TestEncryptionZones
...
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.TestEncryptionZones
Tests run: 39, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.282 sec -
in org.apache.hadoop.hdfs.TestEncryptionZones
...
{code}
{code:bash|title=TestOpenFilesWithSnapshot}
$ mvn test -fn -Dsurefire.printSummary -Dtest=TestOpenFilesWithSnapshot
...
---
T E S T S
---
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.353 sec - in
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
Results :
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0
...
{code}
> Backport HDFS-13087 to branch-2 (Snapshotted encryption zone information
> should be immutable)
> -
>
> Key: HDFS-14510
> URL: https://issues.apache.org/jira/browse/HDFS-14510
> Project: Hadoop HDFS
> Issue Type: Task
> Components: encryption, snapshots
>Affects Versions: 2.10.0, 2.9.2, 2.8.5, 2.7.7
>Reporter: Wei-Chiu Chuang
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-14510.branch-2.8.001.patch
>
>
> HDFS-13087 is a great fix. We should bring it into branch-2 as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14402) Use FileChannel.transferTo() method for transferring block to SCM cache
[
https://issues.apache.org/jira/browse/HDFS-14402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847482#comment-16847482
]
Rakesh R commented on HDFS-14402:
-
bq. Upon making checksum configurable, I am thinking maybe to user, the cache
read performance is concerned mostly. The checksum is executed once when
caching data to DRAM/Pmem. It may be tolerable to user with checksum operation
for data verification. I think more discussions are required. I will open a
separate Jira common to DRAM and Pmem cache.
Yes, it makes sense to me to move the discussion to a separate jira. One use
case I can foresee that, if some user has concern about the sanity of data
while {{read-from-cache}} block then they will do checksum computation again at
read call. This is again depends on the NVMe device consistency level, I think.
In that case the checksum we did at the beginning can be skipped and not
worried about the sanity during {{write-to-cache}}.
> Use FileChannel.transferTo() method for transferring block to SCM cache
> ---
>
> Key: HDFS-14402
> URL: https://issues.apache.org/jira/browse/HDFS-14402
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: caching, datanode
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Attachments: HDFS-14402.000.patch, HDFS-14402.001.patch,
> HDFS-14402.002.patch, With-Cache-Improvement-Patch.png,
> Without-Cache-Improvement-Patch.png
>
>
> We will consider to use transferTo API to improve SCM's cach performace.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847461#comment-16847461 ] He Xiaoqiao commented on HDFS-12733: I check test results and 10 failures, all about EC and WebHDFS, I believe it is not related with this patch. Anyone help to kindly reviews？ > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[
https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847431#comment-16847431
]
Hadoop QA commented on HDFS-12733:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 17s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-12733 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969592/HDFS-12733.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 757ea3ed58f8 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 460ba7f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26838/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26838/testReport/ |
| Max. process+thread count | 3555
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847417#comment-16847417
]
Hadoop QA commented on HDFS-13255:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 13s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.fs.contract.router.web.TestRouterWebHDFSContractRootDirectory |
| | hadoop.fs.contract.router.TestRouterHDFSContractRootDirectory |
| | hadoop.fs.contract.router.TestRouterHDFSContractRootDirectorySecure |
| | hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-13255 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12969600/HDFS-13255-HDFS-13891-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 071f5235c95c 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec
10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 4a16a08 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Created] (HDFS-14512) ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes)
Shen Yinjie created HDFS-14512: -- Summary: ONE_SSD policy will be violated while write data with DistributedFileSystem.create(favoredNodes) Key: HDFS-14512 URL: https://issues.apache.org/jira/browse/HDFS-14512 Project: Hadoop HDFS Issue Type: Bug Reporter: Shen Yinjie Reproduce steps: 1.setStoragePolicy ONE_SSD for a path A; 2. client write data to path A by DistributedFileSystem.create(...favoredNodes) and Passing parameter favoredNodes then, three replicas of file in this path will be located in 2 SSD and 1DISK,which is violating the ONE_SSD policy. Not sure am I clear? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847382#comment-16847382 ] He Xiaoqiao commented on HDFS-12733: [~lindongdong] could you like to share any experience about this issue? I just attach some info which I meet recently in HDFS-14511, is it same with yours? look forward to some other different cases. > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14356) Implement HDFS cache on SCM with native PMDK libs
[ https://issues.apache.org/jira/browse/HDFS-14356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847371#comment-16847371 ] Feilong He commented on HDFS-14356: --- [^HDFS-14356.006.patch] has been uploaded, in which cache read operation was optimized with direct byte buffer. A 3-round DFSIO read test on 50*200MB files shows that our cache read optimization can bring *16% performance improvement* in terms of throughput metric. > Implement HDFS cache on SCM with native PMDK libs > - > > Key: HDFS-14356 > URL: https://issues.apache.org/jira/browse/HDFS-14356 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14356.000.patch, HDFS-14356.001.patch, > HDFS-14356.002.patch, HDFS-14356.003.patch, HDFS-14356.004.patch, > HDFS-14356.005.patch, HDFS-14356.006.patch > > > In this implementation, native PMDK libs are used to map HDFS blocks to SCM. > To use this implementation, user should build hadoop with PMDK libs by > specifying a build option. This implementation is only supported on linux > platform. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14356) Implement HDFS cache on SCM with native PMDK libs
[ https://issues.apache.org/jira/browse/HDFS-14356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-14356: -- Attachment: HDFS-14356.006.patch > Implement HDFS cache on SCM with native PMDK libs > - > > Key: HDFS-14356 > URL: https://issues.apache.org/jira/browse/HDFS-14356 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: caching, datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-14356.000.patch, HDFS-14356.001.patch, > HDFS-14356.002.patch, HDFS-14356.003.patch, HDFS-14356.004.patch, > HDFS-14356.005.patch, HDFS-14356.006.patch > > > In this implementation, native PMDK libs are used to map HDFS blocks to SCM. > To use this implementation, user should build hadoop with PMDK libs by > specifying a build option. This implementation is only supported on linux > platform. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847330#comment-16847330
]
Akira Ajisaka commented on HDFS-13255:
--
002 patch:
* Fail remove and rename operations if there are any mount points under the path
* Fixed some tests
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-13255:
-
Status: Patch Available (was: Open)
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HDFS-13255:
-
Attachment: HDFS-13255-HDFS-13891-002.patch
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-002.patch,
> HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13255) RBF: Fail when try to remove mount point paths
[
https://issues.apache.org/jira/browse/HDFS-13255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847326#comment-16847326
]
Akira Ajisaka commented on HDFS-13255:
--
The change is not simple. The "Input/output error" message is generated by the
client if the delete API returns false. If we want to change the message, we
need to change the API to throw an exception with a proper message, which
breaks the existing behavior of the delete API.
> RBF: Fail when try to remove mount point paths
> --
>
> Key: HDFS-13255
> URL: https://issues.apache.org/jira/browse/HDFS-13255
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wu Weiwei
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-13255-HDFS-13891-wip-001.patch
>
>
> when delete a ns-fed path which include mount point paths, will issue a error.
> Need to delete each mount point path independently.
> Operation step:
> {code:java}
> [hadp@root]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> Source Destinations Owner Group Mode Quota/Usage
> /rm-test-all/rm-test-ns10 ns10->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> /rm-test-all/rm-test-ns2 ns1->/rm-test hadp hadp rwxr-xr-x [NsQuota: -/-,
> SsQuota: -/-]
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns2/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 101 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/NOTICE.txt
> -rw-r--r-- 3 hadp supergroup 1366 2018-03-07 16:57
> hdfs://ns-fed/rm-test-all/rm-test-ns2/README.txt
> [hadp@root]$ hdfs dfs -ls hdfs://ns-fed/rm-test-all/rm-test-ns10/
> Found 2 items
> -rw-r--r-- 3 hadp supergroup 3118 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/core-site.xml
> -rw-r--r-- 3 hadp supergroup 7481 2018-03-07 21:52
> hdfs://ns-fed/rm-test-all/rm-test-ns10/hdfs-site.xml
> [hadp@root]$ hdfs dfs -rm -r hdfs://ns-fed/rm-test-all/
> rm: Failed to move to trash: hdfs://ns-fed/rm-test-all. Consider using
> -skipTrash option
> [hadp@root]$ hdfs dfs -rm -r -skipTrash hdfs://ns-fed/rm-test-all/
> rm: `hdfs://ns-fed/rm-test-all': Input/output error
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847317#comment-16847317 ] lindongdong commented on HDFS-12733: Great job！ > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847282#comment-16847282 ] He Xiaoqiao commented on HDFS-12733: rebase original patch to branch trunk and upload [^HDFS-12733.004.patch] which author from [~brahmareddy]. > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12733) Option to disable to namenode local edits
[ https://issues.apache.org/jira/browse/HDFS-12733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Xiaoqiao updated HDFS-12733: --- Attachment: HDFS-12733.004.patch > Option to disable to namenode local edits > - > > Key: HDFS-12733 > URL: https://issues.apache.org/jira/browse/HDFS-12733 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, performance >Reporter: Brahma Reddy Battula >Assignee: Brahma Reddy Battula >Priority: Major > Attachments: HDFS-12733-001.patch, HDFS-12733-002.patch, > HDFS-12733-003.patch, HDFS-12733.004.patch > > > As of now, Edits will be written in local and shared locations which will be > redundant and local edits never used in HA setup. > Disabling local edits gives little performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
