[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864592#comment-16864592
]
Anu Engineer commented on HDDS-1495:
Care to explain why my core build path is slower with this patch? I am telling
you the command that I use regularly to build, and my concern is really for the
commands that I use.
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch,
> HDDS-1495.003.patch, HDDS-1495.004.patch, HDDS-1495.005.patch,
> HDDS-1495.006.patch, HDDS-1495.007.patch, HDDS-1495.008.patch, Hadoop Docker
> Image inline build process.pdf
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14574) [distcp] Add ability to increase the replication factor for fileList.seq
[
https://issues.apache.org/jira/browse/HDFS-14574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14574:
---
Description:
distcp creates fileList.seq with default replication factor = 3.
For large clusters runing distcp job with thousands of mappers, that 3-replica
for the file listing file is not good enough, because DataNodes easily run out
of max number of xceivers.
It looks like we can pass in a distcp option, update replication factor in when
creating the sequence file writer:
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L517-L521]
Like this:
{code:java}
return SequenceFile.createWriter(getConf(),
SequenceFile.Writer.file(pathToListFile),
SequenceFile.Writer.keyClass(Text.class),
SequenceFile.Writer.valueClass(CopyListingFileStatus.class),
SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE),
SequenceFile.Writer.replication((short)100)); <-- this line
{code}
was:
distcp creates fileList.seq with default replication factor = 3.
For large clusters runing distcp job with thousands of mappers, that 3-replica
for the file listing file is not good enough, because DataNodes easily run out
of max number of xceivers.
It looks like we can pass in a distcp option, update replication factor in when
creating the sequence file writer:
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L517-L521]
> [distcp] Add ability to increase the replication factor for fileList.seq
>
>
> Key: HDFS-14574
> URL: https://issues.apache.org/jira/browse/HDFS-14574
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: distcp
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> distcp creates fileList.seq with default replication factor = 3.
> For large clusters runing distcp job with thousands of mappers, that
> 3-replica for the file listing file is not good enough, because DataNodes
> easily run out of max number of xceivers.
>
> It looks like we can pass in a distcp option, update replication factor in
> when creating the sequence file writer:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L517-L521]
>
> Like this:
> {code:java}
> return SequenceFile.createWriter(getConf(),
> SequenceFile.Writer.file(pathToListFile),
> SequenceFile.Writer.keyClass(Text.class),
> SequenceFile.Writer.valueClass(CopyListingFileStatus.class),
> SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE),
> SequenceFile.Writer.replication((short)100)); <-- this line
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14574) [distcp] Add ability to increase the replication factor for fileList.seq
Wei-Chiu Chuang created HDFS-14574: -- Summary: [distcp] Add ability to increase the replication factor for fileList.seq Key: HDFS-14574 URL: https://issues.apache.org/jira/browse/HDFS-14574 Project: Hadoop HDFS Issue Type: Improvement Components: distcp Reporter: Wei-Chiu Chuang distcp creates fileList.seq with default replication factor = 3. For large clusters runing distcp job with thousands of mappers, that 3-replica for the file listing file is not good enough, because DataNodes easily run out of max number of xceivers. It looks like we can pass in a distcp option, update replication factor in when creating the sequence file writer: [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java#L517-L521] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260813=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260813 ] ASF GitHub Bot logged work on HDDS-1601: Author: ASF GitHub Bot Created on: 15/Jun/19 03:48 Start Date: 15/Jun/19 03:48 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #972: HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. URL: https://github.com/apache/hadoop/pull/972#issuecomment-502331622 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 29 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 575 | trunk passed | | +1 | compile | 287 | trunk passed | | +1 | checkstyle | 83 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 944 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 167 | trunk passed | | 0 | spotbugs | 338 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 531 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 462 | the patch passed | | +1 | compile | 284 | the patch passed | | +1 | javac | 284 | the patch passed | | +1 | checkstyle | 86 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 740 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 165 | the patch passed | | +1 | findbugs | 546 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 160 | hadoop-hdds in the patch failed. | | -1 | unit | 1826 | hadoop-ozone in the patch failed. | | +1 | asflicense | 93 | The patch does not generate ASF License warnings. | | | | 7182 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.common.impl.TestHddsDispatcher | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-972/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/972 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 934a1c2721cd 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct 31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cda9f33 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/3/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/3/testReport/ | | Max. process+thread count | 4836 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260813) Time Spent: 2h (was: 1h 50m) > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=260811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260811 ] ASF GitHub Bot logged work on HDDS-1611: Author: ASF GitHub Bot Created on: 15/Jun/19 03:25 Start Date: 15/Jun/19 03:25 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/973#issuecomment-502330456 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 69 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 43 | Maven dependency ordering for branch | | +1 | mvninstall | 502 | trunk passed | | +1 | compile | 278 | trunk passed | | +1 | checkstyle | 81 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 912 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 171 | trunk passed | | 0 | spotbugs | 326 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 513 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 450 | the patch passed | | +1 | compile | 283 | the patch passed | | +1 | cc | 283 | the patch passed | | +1 | javac | 283 | the patch passed | | -0 | checkstyle | 44 | hadoop-ozone: The patch generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 743 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 175 | the patch passed | | +1 | findbugs | 621 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 182 | hadoop-hdds in the patch failed. | | -1 | unit | 1429 | hadoop-ozone in the patch failed. | | +1 | asflicense | 53 | The patch does not generate ASF License warnings. | | | | 6803 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher | | | hadoop.ozone.om.TestOzoneManager | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.web.client.TestBuckets | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.ozShell.TestOzoneShell | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=18.09.5 Server=18.09.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/973 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 3f6a9a66d4ce 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / cda9f33 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/testReport/ | | Max. process+thread count | 4436 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test hadoop-ozone/tools U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-973/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864572#comment-16864572
]
Wei-Chiu Chuang commented on HDFS-12914:
Suggest to rewrite the last two parameters as
{code:java}
HeartbeatResponse hbResponse = rpcServer.sendHeartbeat(
dnRegistration, storages, 0, 0, 0, 0, 0, null, true,
SlowPeerReports.EMPTY_REPORT, SlowDiskReports.EMPTY_REPORT);
{code}
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864571#comment-16864571
]
He Xiaoqiao commented on HDFS-12914:
Thanks for your quick feedback and report, I will check this test fail.
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864567#comment-16864567
]
Wei-Chiu Chuang commented on HDFS-12914:
That is correct, and so I pushed an addendum patch to include the test file in
trunk.
For branch-3.2, I squashed the two commits into one.
Also, please note that the tests fail in my IntelliJ:
{noformat}
ava.lang.IllegalArgumentException: Argument for @Nonnull parameter 'slowPeers'
of org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.sendHeartbeat must
not be null
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.$$$reportNull$$$0(NameNodeRpcServer.java)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java)
at
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockReportLease.testCheckBlockReportLease(TestBlockReportLease.java:91){noformat}
I guess we need to file another Jira to fix the tests.
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864565#comment-16864565 ] He Xiaoqiao commented on HDFS-14563: Thanks [~elgoiri] for your comments. I will update with your suggestions later. > Enhance interface about recommissioning/decommissioning > --- > > Key: HDFS-14563 > URL: https://issues.apache.org/jira/browse/HDFS-14563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14563.001.patch > > > In current implementation, if we need to decommissioning or recommissioning > one datanode, the only way is add the datanode to include or exclude file > under namenode configuration path then execute command `bin/hadoop dfsadmin > -refreshNodes` and trigger namenode to reload include/exclude and start to > recommissioning or decommissioning datanode. > The shortcomings of this approach is that: > a. namenode reload include/exclude configuration file from devices, if I/O > load is high, handler may be blocked. > b. namenode has to process every datnodes in include and exclude > configurations, if there are many datanodes (very common for large cluster) > pending to process, namenode will be hung for hundred seconds to wait > recommision/decommision finish at the worst since holding write lock. > I think we should expose one lightweight interface to support recommissioning > or decommissioning single datanode, thus we can operate datanode using > dfsadmin more smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864562#comment-16864562
]
He Xiaoqiao commented on HDFS-12914:
Thanks [~jojochuang], it seems to miss #TestBlockReportLease file which commit
to branch trunk. Please help to double check.
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260794 ] ASF GitHub Bot logged work on HDDS-1601: Author: ASF GitHub Bot Created on: 15/Jun/19 01:50 Start Date: 15/Jun/19 01:50 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #972: HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. URL: https://github.com/apache/hadoop/pull/972#issuecomment-502325002 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260794) Time Spent: 1h 50m (was: 1h 40m) > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > This Jira is to implement updating lastAppliedIndex in > OzoneManagerStateMachine once after the buffer is flushed to OM DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260792
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 01:47
Start Date: 15/Jun/19 01:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294030591
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
Yes, you are right. I have initially added it for testing purpose. But we
can test that without returning value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260792)
Time Spent: 1.5h (was: 1h 20m)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260793
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 01:47
Start Date: 15/Jun/19 01:47
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294030597
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260793)
Time Spent: 1h 40m (was: 1.5h)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260787
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 01:33
Start Date: 15/Jun/19 01:33
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294028789
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
This is added to make easy for testing purposes in UT. Otherwise, I need to
store lastAppliedIndex also in OzoneManagerDoubleBuffer and set this value and
add a getter method to get this value to use in the tests. As I don't see any
value in storing lastAppliedIndex in OzoneManagerDoubleBuffer did this way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260787)
Time Spent: 1h 20m (was: 1h 10m)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=260786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260786 ] ASF GitHub Bot logged work on HDDS-1611: Author: ASF GitHub Bot Created on: 15/Jun/19 01:32 Start Date: 15/Jun/19 01:32 Worklog Time Spent: 10m Work Description: ajayydv commented on issue #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/973#issuecomment-502323985 Draft patch for initial feedback, will ad robot test and more unit tests soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260786) Time Spent: 20m (was: 10m) > Evaluate ACL on volume bucket key and prefix to authorize access > - > > Key: HDDS-1611 > URL: https://issues.apache.org/jira/browse/HDDS-1611 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1611: - Labels: pull-request-available (was: ) > Evaluate ACL on volume bucket key and prefix to authorize access > - > > Key: HDDS-1611 > URL: https://issues.apache.org/jira/browse/HDDS-1611 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1611) Evaluate ACL on volume bucket key and prefix to authorize access
[ https://issues.apache.org/jira/browse/HDDS-1611?focusedWorklogId=260785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260785 ] ASF GitHub Bot logged work on HDDS-1611: Author: ASF GitHub Bot Created on: 15/Jun/19 01:31 Start Date: 15/Jun/19 01:31 Worklog Time Spent: 10m Work Description: ajayydv commented on pull request #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. URL: https://github.com/apache/hadoop/pull/973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260785) Time Spent: 10m Remaining Estimate: 0h > Evaluate ACL on volume bucket key and prefix to authorize access > - > > Key: HDDS-1611 > URL: https://issues.apache.org/jira/browse/HDDS-1611 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14567) If kms-acls is failed to load, and it will never be reload
[
https://issues.apache.org/jira/browse/HDFS-14567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864552#comment-16864552
]
Wei-Chiu Chuang commented on HDFS-14567:
I think a quick workaround is to create a temporary acl file, once it closes,
move it to kms-acls.xml
> If kms-acls is failed to load, and it will never be reload
> ---
>
> Key: HDFS-14567
> URL: https://issues.apache.org/jira/browse/HDFS-14567
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
>
> Scenario : through one automation tool , we are generating kms-acls , though
> the generation of kms-acls is not completed , the system will detect a
> modification of kms-alcs and it will try to load
> Before getting the configuration we are modifiying last reload time , code
> shown below
> {code:java}
> private Configuration loadACLsFromFile() {
> LOG.debug("Loading ACLs file");
> lastReload = System.currentTimeMillis();
> Configuration conf = KMSConfiguration.getACLsConf();
> // triggering the resource loading.
> conf.get(Type.CREATE.getAclConfigKey());
> return conf;
> }{code}
> if the kms-acls file written within next 100ms , the changes will not be
> loaded as this condition "newer = f.lastModified() - time > 100" never meets
> because we have modified last reload time before getting the configuration
> {code:java}
> public static boolean isACLsFileNewer(long time) {
> boolean newer = false;
> String confDir = System.getProperty(KMS_CONFIG_DIR);
> if (confDir != null) {
> Path confPath = new Path(confDir);
> if (!confPath.isUriPathAbsolute()) {
> throw new RuntimeException("System property '" + KMS_CONFIG_DIR +
> "' must be an absolute path: " + confDir);
> }
> File f = new File(confDir, KMS_ACLS_XML);
> LOG.trace("Checking file {}, modification time is {}, last reload time is"
> + " {}", f.getPath(), f.lastModified(), time);
> // at least 100ms newer than time, we do this to ensure the file
> // has been properly closed/flushed
> newer = f.lastModified() - time > 100;
> }
> return newer;
> } {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-12914:
---
Fix Version/s: 3.2.1
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0, 3.2.1
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14573) Backport Standby Read to branch-3
[ https://issues.apache.org/jira/browse/HDFS-14573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864548#comment-16864548 ] Wei-Chiu Chuang commented on HDFS-14573: I don't think getting this feature into lower 3.x branches is required before getting into 2.x branches. That said, I am happy to see this happening. > Backport Standby Read to branch-3 > - > > Key: HDFS-14573 > URL: https://issues.apache.org/jira/browse/HDFS-14573 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14573-branch-3.0.001.patch, > HDFS-14573-branch-3.1.001.patch, HDFS-14573-branch-3.2.001.patch > > > This Jira tracks backporting the feature consistent read from standby > (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for > backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864547#comment-16864547
]
Hadoop QA commented on HDFS-12914:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
4s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
52s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}104m
6s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 40s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HDFS-12914 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971847/HDFS-12914.branch-3.2.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9b3f5cd02460 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 335aebb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26960/testReport/ |
| Max. process+thread count | 2848 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26960/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Block report leases cause missing blocks until next report
> --
>
>
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260779
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 01:02
Start Date: 15/Jun/19 01:02
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294028789
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
This is added to make easy for testing purposes in UT. Otherwise, I need to
store lastAppliedIndex also in OzoneManagerDoubleBuffer and set this value and
add a getter method to get this value to use in the tests. As I don't see any
value in storing lastAppliedIndex in OzoneManagerDoubleBuffer did this way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260779)
Time Spent: 1h 10m (was: 1h)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260778
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 01:01
Start Date: 15/Jun/19 01:01
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294028789
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
This is added to make easy for testing purposes in UT. Otherwise, I need to
store lastAppliedIndex also in OzoneManagerDoubleBuffer and set this value and
add a getter method to get this value to use in the tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260778)
Time Spent: 1h (was: 50m)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260777
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 15/Jun/19 00:59
Start Date: 15/Jun/19 00:59
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294028789
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
This is added to make easy for testing purposes in UT.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260777)
Time Spent: 50m (was: 40m)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1667) Docker compose file may referring to incorrect docker image name
[
https://issues.apache.org/jira/browse/HDDS-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864542#comment-16864542
]
Hadoop QA commented on HDDS-1667:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} yamllint {color} | {color:blue} 0m
0s{color} | {color:blue} yamllint was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 7s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 16s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
7s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher |
| | hadoop.ozone.om.TestOzoneManagerHA |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.client.rpc.TestBCSID |
| | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
| |
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
|
| | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.5 Server=18.09.5 base:
https://builds.apache.org/job/PreCommit-HDDS-Build/2731/artifact/out/Dockerfile
|
| JIRA Issue | HDDS-1667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971852/HDDS-1667.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml yamllint |
| uname
[jira] [Commented] (HDFS-14564) Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable
[
https://issues.apache.org/jira/browse/HDFS-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864539#comment-16864539
]
Hadoop QA commented on HDFS-14564:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
29s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} branch/hadoop-hdfs-project/hadoop-hdfs-native-client
no findbugs output file (findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} root: The patch generated 0 new + 110 unchanged - 1
fixed = 110 total (was 111) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hadoop-hdfs-project/hadoop-hdfs-native-client has no
data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
48s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 25s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Work logged] (HDDS-1638) Implement Key Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1638?focusedWorklogId=260768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260768 ] ASF GitHub Bot logged work on HDDS-1638: Author: ASF GitHub Bot Created on: 15/Jun/19 00:18 Start Date: 15/Jun/19 00:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #956: HDDS-1638. Implement Key Write Requests to use Cache and DoubleBuffer. URL: https://github.com/apache/hadoop/pull/956#issuecomment-502316805 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 31 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 19 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 47 | Maven dependency ordering for branch | | +1 | mvninstall | 536 | trunk passed | | +1 | compile | 301 | trunk passed | | +1 | checkstyle | 90 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 903 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 183 | trunk passed | | 0 | spotbugs | 337 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 529 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 472 | the patch passed | | +1 | compile | 311 | the patch passed | | +1 | cc | 311 | the patch passed | | +1 | javac | 311 | the patch passed | | +1 | checkstyle | 101 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 692 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 183 | the patch passed | | +1 | findbugs | 537 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 141 | hadoop-hdds in the patch failed. | | -1 | unit | 1155 | hadoop-ozone in the patch failed. | | +1 | asflicense | 56 | The patch does not generate ASF License warnings. | | | | 6517 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-956/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/956 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux c422edbb5455 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / c7554ff | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-956/3/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-956/3/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-956/3/testReport/ | | Max. process+thread count | 4854 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-956/3/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260768) Time Spent: 1h 20m (was: 1h 10m) > Implement Key Write Requests to use Cache and DoubleBuffer > -- > > Key: HDDS-1638 > URL: https://issues.apache.org/jira/browse/HDDS-1638 > Project: Hadoop Distributed Data Store
[jira] [Commented] (HDDS-1567) Define a set of environment variables to configure Ozone docker image
[
https://issues.apache.org/jira/browse/HDDS-1567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864534#comment-16864534
]
Eric Yang commented on HDDS-1567:
-
{quote}Do you suggest to use one keytab option where the value will be
different for each service?
{quote}
In Hadoop, there is minimum a pair of principals to use: Spnego principal and
service principal. They are unique per server. I am open to store both
principals in the same keytab file for the same server.
{quote}Can you please share the details of the planned implementation?
{quote}
My thought on improving this:
# map the global variable into the TAGs format in envtoconf.py.
# import a copy of docker-config as default. This allows user to use docker
run command to start Ozone container image with almost no parameters.
# Refine docker-compose and k8s templates with global environment variables.
# document the usage of the global environment variable in external facing
document.
> Define a set of environment variables to configure Ozone docker image
> -
>
> Key: HDDS-1567
> URL: https://issues.apache.org/jira/browse/HDDS-1567
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Eric Yang
>Priority: Major
>
> For developer that tries to setup docker image by end for testing purpose, it
> would be nice to predefine a set of environment variables that can be passed
> to Ozone docker image to configure the minimum set of configuration to start
> Ozone containers. There is a python script that converts environment
> variables to config, but documentation does not show what setting can be
> passed to configure the system. This task would be a good starting point to
> document the available configuration knobs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[
https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260766
]
ASF GitHub Bot logged work on HDDS-1601:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:55
Start Date: 14/Jun/19 23:55
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #972:
HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB.
URL: https://github.com/apache/hadoop/pull/972#discussion_r294023858
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerRatisSnapshot.java
##
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis;
+
+/**
+ * Functional interface for OM RatisSnapshot.
+ */
+
+public interface OzoneManagerRatisSnapshot {
+
+ /**
+ * Update lastAppliedIndex with the specified value in OzoneManager
+ * StateMachine.
+ * @param lastAppliedIndex
+ * @return lastAppliedIndex
+ */
+ long updateLastAppliedIndex(long lastAppliedIndex);
+}
Review comment:
Do we need a return value here? It is not being used anywhere.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260766)
Time Spent: 40m (was: 0.5h)
> Implement updating lastAppliedIndex after buffer flush to OM DB.
>
>
> Key: HDDS-1601
> URL: https://issues.apache.org/jira/browse/HDDS-1601
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This Jira is to implement updating lastAppliedIndex in
> OzoneManagerStateMachine once after the buffer is flushed to OM DB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864518#comment-16864518
]
Eric Yang commented on HDDS-1495:
-
{quote}> mvn -f pom.ozone.xml clean package -DskipTests -Dmaven.javadoc.skip
-Dskipshade{quote}
The above command does not trigger docker build. It is a full build command
except docker part. In order to trigger a docker build, you need to pass in
-Pdocker-build for the current process. For the patched version, pass in
-Pdocker
The discussion of the performance improvement is about reiteration time spend
for developer. If developer is working on Docker only, he can jump into docker
module, and trigger build:
{code}
cd hadoop/hadoop-ozone/docker
mvn package -Pdocker
{code}
This saves time for repeating the process without doing the full build.
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch,
> HDDS-1495.003.patch, HDDS-1495.004.patch, HDDS-1495.005.patch,
> HDDS-1495.006.patch, HDDS-1495.007.patch, HDDS-1495.008.patch, Hadoop Docker
> Image inline build process.pdf
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260765
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:16
Start Date: 14/Jun/19 23:16
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294019380
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
Review comment:
Named this as totalNumOfFlushIterations because we have one more metric
which says maxNumberOfTransactionsFlushedInOneIteration. This will say till
this point what this the maximum number of transactions flushed in a iteration.
If I change the totalNumOfFlushIterations one as totalnumOfFlushOperations, do
you want to change maxNumberOfTransactionsFlushedInOneIteration to some other
name?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260765)
Time Spent: 2.5h (was: 2h 20m)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260764
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:15
Start Date: 14/Jun/19 23:15
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294019380
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
Review comment:
Named this as totalNumOfFlushIterations because we have one more metric
which says maxNumberOfTransactionsFlushedInOneIteration. This will say till
this point what this the maximum number of transactions flushed in a iteration.
If I change the above one as ops, do you want to change below one or just leave
as it is?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260764)
Time Spent: 2h 20m (was: 2h 10m)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260763=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260763
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:14
Start Date: 14/Jun/19 23:14
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294019380
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
Review comment:
Named this as totalNumOfFlushIterations because we have one more metric
which says maxNumberOfTransactionsFlushedInOneIteration. This will say till
this point what this the maximum number of transactions flushed in a iteration.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260763)
Time Spent: 2h 10m (was: 2h)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260762
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:13
Start Date: 14/Jun/19 23:13
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294019380
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
Review comment:
Named this as totalNumOfFlushIterations because we have one more metric
which says max maxNumberOfTransactionsFlushedInOneIteration.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260762)
Time Spent: 2h (was: 1h 50m)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1667) Docker compose file may referring to incorrect docker image name
[
https://issues.apache.org/jira/browse/HDDS-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864511#comment-16864511
]
Eric Yang commented on HDDS-1667:
-
Patch 005 fixes the white space issue, the failed unit tests are not related to
this patch.
> Docker compose file may referring to incorrect docker image name
>
>
> Key: HDDS-1667
> URL: https://issues.apache.org/jira/browse/HDDS-1667
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Trivial
> Fix For: 0.4.1
>
> Attachments: HDDS-1667.001.patch, HDDS-1667.002.patch,
> HDDS-1667.003.patch, HDDS-1667.004.patch, HDDS-1667.005.patch
>
>
> In fault injection test, the docker compose file is templated using:
> ${user.name}/ozone:${project.version}
> If user pass in parameter -Ddocker.image to cause docker build to generate a
> different name. This can cause fault injection test to fail/stuck because it
> could not find the required docker image. The fix is simply use docker.image
> token to filter docker compose file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1667) Docker compose file may referring to incorrect docker image name
[
https://issues.apache.org/jira/browse/HDDS-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1667:
Attachment: HDDS-1667.005.patch
> Docker compose file may referring to incorrect docker image name
>
>
> Key: HDDS-1667
> URL: https://issues.apache.org/jira/browse/HDDS-1667
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Trivial
> Fix For: 0.4.1
>
> Attachments: HDDS-1667.001.patch, HDDS-1667.002.patch,
> HDDS-1667.003.patch, HDDS-1667.004.patch, HDDS-1667.005.patch
>
>
> In fault injection test, the docker compose file is templated using:
> ${user.name}/ozone:${project.version}
> If user pass in parameter -Ddocker.image to cause docker build to generate a
> different name. This can cause fault injection test to fail/stuck because it
> could not find the required docker image. The fix is simply use docker.image
> token to filter docker compose file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260758=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260758
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:04
Start Date: 14/Jun/19 23:04
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294017433
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
Review comment:
NIT: Can we rename it to numOfFlushOperations. Iteration gives the
impression that we iterate through a list.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260758)
Time Spent: 1.5h (was: 1h 20m)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260759
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:04
Start Date: 14/Jun/19 23:04
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294017596
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/metrics/OzoneManagerDoubleBufferMetrics.java
##
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.ratis.metrics;
+
+import org.apache.hadoop.metrics2.MetricsSystem;
+import org.apache.hadoop.metrics2.annotation.Metric;
+import org.apache.hadoop.metrics2.lib.DefaultMetricsSystem;
+import org.apache.hadoop.metrics2.lib.MutableCounterLong;
+
+/**
+ * Class which maintains metrics related to OzoneManager DoubleBuffer.
+ */
+public class OzoneManagerDoubleBufferMetrics {
+
+ private static final String SOURCE_NAME =
+ OzoneManagerDoubleBufferMetrics.class.getSimpleName();
+
+ @Metric(about = "Total Number of flush iterations happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushIterations;
+
+ @Metric(about = "Total Number of flushed transactions happened in " +
+ "OzoneManagerDoubleBuffer.")
+ private MutableCounterLong totalNumOfFlushedTransactions;
+
+ @Metric(about = "Max Number of transactions flushed in a iteration in " +
+ "OzoneManagerDoubleBuffer. This will provide a value which is maximum " +
+ "number of transactions flushed in a single flush iteration till now.")
+ private MutableCounterLong maxNumberOfTransactionsFlushedInOneIteration;
+
+
+ public static OzoneManagerDoubleBufferMetrics create() {
+MetricsSystem ms = DefaultMetricsSystem.instance();
+return ms.register(SOURCE_NAME,
+"OzoneManager DoubleBuffer Metrics",
+new OzoneManagerDoubleBufferMetrics());
+ }
+
+ public void incTotalNumOfFlushIterations() {
+this.totalNumOfFlushIterations.incr();
+ }
+
+ public void setTotalSizeOfFlushedTransactions(
+ long flushedTransactions) {
Review comment:
NIT: Can we rename this to incrTotal as we are incrementing by the input
value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260759)
Time Spent: 1h 40m (was: 1.5h)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1579) Create OMDoubleBuffer metrics
[
https://issues.apache.org/jira/browse/HDDS-1579?focusedWorklogId=260760=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260760
]
ASF GitHub Bot logged work on HDDS-1579:
Author: ASF GitHub Bot
Created on: 14/Jun/19 23:04
Start Date: 14/Jun/19 23:04
Worklog Time Spent: 10m
Work Description: hanishakoneru commented on pull request #871:
HDDS-1579. Create OMDoubleBuffer metrics.
URL: https://github.com/apache/hadoop/pull/871#discussion_r294017908
##
File path:
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
##
@@ -149,6 +160,23 @@ private void cleanupCache(long lastRatisTransactionIndex)
{
omMetadataManager.getBucketTable().cleanupCache(lastRatisTransactionIndex);
}
+ /**
+ * Set OzoneManagerDoubleBuffer metrics values.
+ * @param flushedTransactionsSize
+ */
+ private void setOzoneManagerDoubleBufferMetrics(
+ long flushedTransactionsSize) {
Review comment:
NIT: can we rename this method to something like updateMetrics?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 260760)
Time Spent: 1h 50m (was: 1h 40m)
> Create OMDoubleBuffer metrics
> -
>
> Key: HDDS-1579
> URL: https://issues.apache.org/jira/browse/HDDS-1579
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This Jira is to implement OMDoubleBuffer metrics, to show metrics like.
> # flushIterations.
> # totalTransactionsflushed.
>
> Any other related metrics. This Jira is created based on the comment by
> [~anu] during HDDS-1512 review.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260755 ] ASF GitHub Bot logged work on HDDS-1601: Author: ASF GitHub Bot Created on: 14/Jun/19 23:01 Start Date: 14/Jun/19 23:01 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #972: HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. URL: https://github.com/apache/hadoop/pull/972#issuecomment-502305424 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 42 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 547 | trunk passed | | +1 | compile | 287 | trunk passed | | +1 | checkstyle | 89 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 974 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 176 | trunk passed | | 0 | spotbugs | 359 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 572 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 495 | the patch passed | | +1 | compile | 328 | the patch passed | | +1 | javac | 328 | the patch passed | | -0 | checkstyle | 46 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 830 | patch has no errors when building and testing our client artifacts. | | -1 | javadoc | 99 | hadoop-ozone generated 14 new + 9 unchanged - 0 fixed = 23 total (was 9) | | +1 | findbugs | 584 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 180 | hadoop-hdds in the patch failed. | | -1 | unit | 1500 | hadoop-ozone in the patch failed. | | +1 | asflicense | 48 | The patch does not generate ASF License warnings. | | | | 7088 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher | | | hadoop.ozone.om.TestOmInit | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestFailureHandlingByClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/972 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9dfc1cb72e5e 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b24efa1 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/testReport/ | | Max. process+thread count | 4816 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260755) Time Spent: 0.5h (was:
[jira] [Commented] (HDFS-14573) Backport Standby Read to branch-3
[ https://issues.apache.org/jira/browse/HDFS-14573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864503#comment-16864503 ] Chen Liang commented on HDFS-14573: --- The patches were created by cherry-picking from trunk, with minimum conflicts. Mind taking a look [~shv]? > Backport Standby Read to branch-3 > - > > Key: HDFS-14573 > URL: https://issues.apache.org/jira/browse/HDFS-14573 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14573-branch-3.0.001.patch, > HDFS-14573-branch-3.1.001.patch, HDFS-14573-branch-3.2.001.patch > > > This Jira tracks backporting the feature consistent read from standby > (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for > backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864502#comment-16864502
]
Anu Engineer commented on HDDS-1495:
I did;
Without this patch, I am able to do a build in 04:38 on my machine.
With this patch, it takes 05:30 on the same machine. Looks like a 20% overhead
to me, that is a very significant time consider that we are talking about the
end-to-end build time.
Here is the build command that I used.
> mvn -f pom.ozone.xml clean package -DskipTests -Dmaven.javadoc.skip
-Dskipshade
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch,
> HDDS-1495.003.patch, HDDS-1495.004.patch, HDDS-1495.005.patch,
> HDDS-1495.006.patch, HDDS-1495.007.patch, HDDS-1495.008.patch, Hadoop Docker
> Image inline build process.pdf
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14570) Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled property back into the hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864501#comment-16864501 ] Wei-Chiu Chuang commented on HDFS-14570: Looks like this behavior is changed by HDFS-7985 (WebHDFS should be always enabled). > Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled > property back into the hdfs-site.xml > -- > > Key: HDFS-14570 > URL: https://issues.apache.org/jira/browse/HDFS-14570 > Project: Hadoop HDFS > Issue Type: Wish > Components: webhdfs >Affects Versions: 3.0.0, 3.1.0, 3.0.1, 3.0.2, 3.2.0, 3.1.1, 3.0.3, 3.1.2 >Reporter: Scott A. Wehner >Priority: Major > Labels: webhdfs > Original Estimate: 6h > Remaining Estimate: 6h > > We don't want to enable security for viewing namenode http page, but we don't > want people to be able to modify the contents of hdfs through anonymous > access to the namenode page. in Hadoop 3 we lost the ability to totally > disable webhdfs. want to bring this back, doesn't seem to hard to do, but > makes it important in our environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1675) Cleanup Volume Request 2 phase old code
[ https://issues.apache.org/jira/browse/HDDS-1675?focusedWorklogId=260754=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260754 ] ASF GitHub Bot logged work on HDDS-1675: Author: ASF GitHub Bot Created on: 14/Jun/19 22:57 Start Date: 14/Jun/19 22:57 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #964: HDDS-1675. Cleanup Volume Request 2 phase old code. URL: https://github.com/apache/hadoop/pull/964#issuecomment-502304754 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 52 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 17 | Maven dependency ordering for branch | | +1 | mvninstall | 524 | trunk passed | | +1 | compile | 298 | trunk passed | | +1 | checkstyle | 82 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 943 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 179 | trunk passed | | 0 | spotbugs | 385 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 592 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 21 | Maven dependency ordering for patch | | +1 | mvninstall | 475 | the patch passed | | +1 | compile | 304 | the patch passed | | +1 | cc | 304 | the patch passed | | +1 | javac | 304 | the patch passed | | +1 | checkstyle | 84 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 718 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 192 | the patch passed | | +1 | findbugs | 668 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 205 | hadoop-hdds in the patch failed. | | -1 | unit | 224 | hadoop-ozone in the patch failed. | | +1 | asflicense | 66 | The patch does not generate ASF License warnings. | | | | 5828 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.common.impl.TestHddsDispatcher | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-964/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/964 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux fb7c3d8c5ac9 4.4.0-144-generic #170~14.04.1-Ubuntu SMP Mon Mar 18 15:02:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b24efa1 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-964/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-964/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-964/2/testReport/ | | Max. process+thread count | 1297 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-964/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260754) Time Spent: 0.5h (was: 20m) > Cleanup Volume Request 2 phase old code > --- > > Key: HDDS-1675 > URL: https://issues.apache.org/jira/browse/HDDS-1675 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining
[jira] [Updated] (HDFS-14570) Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled property back into the hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-14570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14570: --- Fix Version/s: (was: 2.9.2) > Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled > property back into the hdfs-site.xml > -- > > Key: HDFS-14570 > URL: https://issues.apache.org/jira/browse/HDFS-14570 > Project: Hadoop HDFS > Issue Type: Wish > Components: webhdfs >Affects Versions: 3.0.0, 3.1.0, 3.0.1, 3.0.2, 3.2.0, 3.1.1, 3.0.3, 3.1.2 >Reporter: Scott A. Wehner >Priority: Major > Labels: webhdfs > Original Estimate: 6h > Remaining Estimate: 6h > > We don't want to enable security for viewing namenode http page, but we don't > want people to be able to modify the contents of hdfs through anonymous > access to the namenode page. in Hadoop 3 we lost the ability to totally > disable webhdfs. want to bring this back, doesn't seem to hard to do, but > makes it important in our environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14573) Backport Standby Read to branch-3
[ https://issues.apache.org/jira/browse/HDFS-14573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14573: -- Attachment: HDFS-14573-branch-3.2.001.patch > Backport Standby Read to branch-3 > - > > Key: HDFS-14573 > URL: https://issues.apache.org/jira/browse/HDFS-14573 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14573-branch-3.0.001.patch, > HDFS-14573-branch-3.1.001.patch, HDFS-14573-branch-3.2.001.patch > > > This Jira tracks backporting the feature consistent read from standby > (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for > backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14573) Backport Standby Read to branch-3
[ https://issues.apache.org/jira/browse/HDFS-14573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14573: -- Attachment: HDFS-14573-branch-3.1.001.patch > Backport Standby Read to branch-3 > - > > Key: HDFS-14573 > URL: https://issues.apache.org/jira/browse/HDFS-14573 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14573-branch-3.0.001.patch, > HDFS-14573-branch-3.1.001.patch > > > This Jira tracks backporting the feature consistent read from standby > (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for > backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14573) Backport Standby Read to branch-3
[ https://issues.apache.org/jira/browse/HDFS-14573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-14573: -- Attachment: HDFS-14573-branch-3.0.001.patch > Backport Standby Read to branch-3 > - > > Key: HDFS-14573 > URL: https://issues.apache.org/jira/browse/HDFS-14573 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-14573-branch-3.0.001.patch > > > This Jira tracks backporting the feature consistent read from standby > (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for > backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14573) Backport Standby Read to branch-3
Chen Liang created HDFS-14573: - Summary: Backport Standby Read to branch-3 Key: HDFS-14573 URL: https://issues.apache.org/jira/browse/HDFS-14573 Project: Hadoop HDFS Issue Type: Task Components: hdfs Reporter: Chen Liang Assignee: Chen Liang This Jira tracks backporting the feature consistent read from standby (HDFS-12943) to branch-3.x, including 3.0, 3.1, 3.2. This is required for backporting to branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260751 ] ASF GitHub Bot logged work on HDDS-1601: Author: ASF GitHub Bot Created on: 14/Jun/19 22:50 Start Date: 14/Jun/19 22:50 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #972: HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. URL: https://github.com/apache/hadoop/pull/972#issuecomment-502303570 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 52 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 | mvninstall | 485 | trunk passed | | +1 | compile | 262 | trunk passed | | +1 | checkstyle | 66 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 796 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 152 | trunk passed | | 0 | spotbugs | 326 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 517 | trunk passed | ||| _ Patch Compile Tests _ | | +1 | mvninstall | 458 | the patch passed | | +1 | compile | 294 | the patch passed | | +1 | javac | 294 | the patch passed | | -0 | checkstyle | 49 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 686 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 178 | the patch passed | | +1 | findbugs | 538 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 182 | hadoop-hdds in the patch failed. | | -1 | unit | 1471 | hadoop-ozone in the patch failed. | | +1 | asflicense | 49 | The patch does not generate ASF License warnings. | | | | 6491 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.client.rpc.TestWatchForCommit | | | hadoop.hdds.scm.pipeline.TestSCMPipelineManager | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/972 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7c2550f8b8f4 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / b24efa1 | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/testReport/ | | Max. process+thread count | 5016 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-972/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260751) Time Spent: 20m (was: 10m) > Implement updating lastAppliedIndex after buffer flush to OM DB. >
[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864494#comment-16864494
]
Hadoop QA commented on HDDS-1495:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} hadolint {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 4 new + 0 unchanged - 0 fixed = 4
total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} pylint {color} | {color:orange} 0m
5s{color} | {color:orange} Error running pylint. Please check pylint stderr
files. {color} |
| {color:green}+1{color} | {color:green} pylint {color} | {color:green} 0m
5s{color} | {color:green} There were no new pylint issues. {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
0s{color} | {color:red} The patch generated 5 new + 0 unchanged - 0 fixed = 5
total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 13s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 10s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.container.common.impl.TestHddsDispatcher |
| | hadoop.ozone.client.rpc.TestOzoneAtRestEncryption |
| |
[jira] [Created] (HDDS-1690) ContainerController should provide a way to retrieve containers per volume
Hrishikesh Gadre created HDDS-1690: -- Summary: ContainerController should provide a way to retrieve containers per volume Key: HDDS-1690 URL: https://issues.apache.org/jira/browse/HDDS-1690 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Datanode Affects Versions: 0.4.0 Reporter: Hrishikesh Gadre Assignee: Hrishikesh Gadre Ozone data scrubber needs to start a dedicated thread per volume to perform checksum validation for all the containers in the given volume. But currently ContainerController does not expose an API to fetch the list of containers for a given volume. This Jira is to add the relevant API to ContainerController class so that we can retrieve list of containers for a given HDDS volume. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1689) Implement S3 Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1689: - Description: Implement S3 Bucket write requests to use OM Cache, double buffer. In this Jira will add the changes to implement S3 bucket operations, and HA/Non-HA will have a different code path, but once all requests are implemented will have a single code path. was: Implement Bucket write requests to use OM Cache, double buffer. And also in OM previously we used to Ratis client for communication to Ratis server, instead of that use Ratis server API's. In this Jira will add the changes to implement bucket operations, and HA/Non-HA will have a different code path, but once all requests are implemented will have a single code path. > Implement S3 Bucket Write Requests to use Cache and DoubleBuffer > > > Key: HDDS-1689 > URL: https://issues.apache.org/jira/browse/HDDS-1689 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.5.0 > > > Implement S3 Bucket write requests to use OM Cache, double buffer. > > In this Jira will add the changes to implement S3 bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1689) Implement S3 Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1689: - Labels: (was: pull-request-available) > Implement S3 Bucket Write Requests to use Cache and DoubleBuffer > > > Key: HDDS-1689 > URL: https://issues.apache.org/jira/browse/HDDS-1689 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 0.5.0 > > > Implement Bucket write requests to use OM Cache, double buffer. > And also in OM previously we used to Ratis client for communication to Ratis > server, instead of that use Ratis server API's. > > In this Jira will add the changes to implement bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1689) Implement S3 Bucket Write Requests to use Cache and DoubleBuffer
[ https://issues.apache.org/jira/browse/HDDS-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1689: - Fix Version/s: (was: 0.5.0) > Implement S3 Bucket Write Requests to use Cache and DoubleBuffer > > > Key: HDDS-1689 > URL: https://issues.apache.org/jira/browse/HDDS-1689 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Implement S3 Bucket write requests to use OM Cache, double buffer. > > In this Jira will add the changes to implement S3 bucket operations, and > HA/Non-HA will have a different code path, but once all requests are > implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1689) Implement S3 Bucket Write Requests to use Cache and DoubleBuffer
Bharat Viswanadham created HDDS-1689: Summary: Implement S3 Bucket Write Requests to use Cache and DoubleBuffer Key: HDDS-1689 URL: https://issues.apache.org/jira/browse/HDDS-1689 Project: Hadoop Distributed Data Store Issue Type: Sub-task Components: Ozone Manager Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Fix For: 0.5.0 Implement Bucket write requests to use OM Cache, double buffer. And also in OM previously we used to Ratis client for communication to Ratis server, instead of that use Ratis server API's. In this Jira will add the changes to implement bucket operations, and HA/Non-HA will have a different code path, but once all requests are implemented will have a single code path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-12914:
---
Attachment: HDFS-12914.branch-3.2.patch
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[
https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864484#comment-16864484
]
Hadoop QA commented on HDFS-14563:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-hdfs-project: The patch generated 9 new +
614 unchanged - 0 fixed = 623 total (was 614) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
23s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14563 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971835/HDFS-14563.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 6e45b3568bf7 3.13.0-153-generic
[jira] [Commented] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864461#comment-16864461
]
Eric Yang commented on HDDS-1495:
-
[~anu] patch 8 is rebased to current trunk. 400mb saving were from excluding
byteman, robot framework, dumb-init and async-profiler. Developer tools were
the reason to make the image more bloated. Patch 8 added back those
dependencies and keep the image as close to original as possible. Therefore,
space saving doesn't exist. The speed of rebuilding is very close, current
method takes about 14-35 seconds. Where, docker module build time is 19-21
seconds.
The only improvements using this approach are:
# It would be possible for someone to re-roll a image based on released tarball
# Faster rebuild time by separating binary download from configuration steps.
if there is more shell script commands to add to setup-image.sh
# Reduced number of layers to download.
Let me know if you want to give it a try.
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch,
> HDDS-1495.003.patch, HDDS-1495.004.patch, HDDS-1495.005.patch,
> HDDS-1495.006.patch, HDDS-1495.007.patch, HDDS-1495.008.patch, Hadoop Docker
> Image inline build process.pdf
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14564) Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable
[
https://issues.apache.org/jira/browse/HDFS-14564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864460#comment-16864460
]
Sahil Takiar commented on HDFS-14564:
-
[~smeng] addressed the checkstyle issues. Ran the failed unit tests locally and
they pass.
> Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable
> -
>
> Key: HDFS-14564
> URL: https://issues.apache.org/jira/browse/HDFS-14564
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> Splitting this out from HDFS-14478
> The {{PositionedReadable#readFully}} APIs have existed for a while, but have
> never been exposed via libhdfs.
> HDFS-3246 added a new interface called {{ByteBufferPositionedReadable}} that
> provides a {{ByteBuffer}} version of {{PositionedReadable}}, but it does not
> contain a {{readFully}} method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1684) OM should create Ratis related dirs only if ratis is enabled
[ https://issues.apache.org/jira/browse/HDDS-1684?focusedWorklogId=260719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260719 ] ASF GitHub Bot logged work on HDDS-1684: Author: ASF GitHub Bot Created on: 14/Jun/19 21:24 Start Date: 14/Jun/19 21:24 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #965: HDDS-1684. OM should create Ratis related dirs only if ratis is enabled URL: https://github.com/apache/hadoop/pull/965#issuecomment-502267835 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 36 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 71 | Maven dependency ordering for branch | | +1 | mvninstall | 521 | trunk passed | | +1 | compile | 271 | trunk passed | | +1 | checkstyle | 73 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 819 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 166 | trunk passed | | 0 | spotbugs | 336 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 532 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 31 | Maven dependency ordering for patch | | +1 | mvninstall | 459 | the patch passed | | +1 | compile | 280 | the patch passed | | +1 | javac | 280 | the patch passed | | -0 | checkstyle | 41 | hadoop-ozone: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 625 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 170 | the patch passed | | +1 | findbugs | 574 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 244 | hadoop-hdds in the patch failed. | | -1 | unit | 1434 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 6617 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.node.TestNodeReportHandler | | | hadoop.hdds.scm.block.TestBlockManager | | | hadoop.hdds.scm.pipeline.TestRatisPipelineProvider | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.TestSecureOzoneCluster | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/965 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 02bad4253e97 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ae4143a | | Default Java | 1.8.0_212 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/artifact/out/patch-unit-hadoop-hdds.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/testReport/ | | Max. process+thread count | 3828 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/container-service hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-965/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260719) Time Spent: 50m (was: 40m) > OM should create Ratis related dirs only if ratis is enabled > > > Key: HDDS-1684 > URL:
[jira] [Updated] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1601: - Target Version/s: 0.5.0 > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement updating lastAppliedIndex in > OzoneManagerStateMachine once after the buffer is flushed to OM DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1601: - Status: Patch Available (was: In Progress) > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement updating lastAppliedIndex in > OzoneManagerStateMachine once after the buffer is flushed to OM DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1495) Create hadoop/ozone docker images with inline build process
[
https://issues.apache.org/jira/browse/HDDS-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDDS-1495:
Attachment: HDDS-1495.008.patch
> Create hadoop/ozone docker images with inline build process
> ---
>
> Key: HDDS-1495
> URL: https://issues.apache.org/jira/browse/HDDS-1495
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Eric Yang
>Priority: Major
> Attachments: HADOOP-16091.001.patch, HADOOP-16091.002.patch,
> HDDS-1495.003.patch, HDDS-1495.004.patch, HDDS-1495.005.patch,
> HDDS-1495.006.patch, HDDS-1495.007.patch, HDDS-1495.008.patch, Hadoop Docker
> Image inline build process.pdf
>
>
> This is proposed by [~eyang] in
> [this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
> mailing thread.
> {quote}1, 3. There are 38 Apache projects hosting docker images on Docker hub
> using Apache Organization. By browsing Apache github mirror. There are only 7
> projects using a separate repository for docker image build. Popular projects
> official images are not from Apache organization, such as zookeeper, tomcat,
> httpd. We may not disrupt what other Apache projects are doing, but it looks
> like inline build process is widely employed by majority of projects such as
> Nifi, Brooklyn, thrift, karaf, syncope and others. The situation seems a bit
> chaotic for Apache as a whole. However, Hadoop community can decide what is
> best for Hadoop. My preference is to remove ozone from source tree naming, if
> Ozone is intended to be subproject of Hadoop for long period of time. This
> enables Hadoop community to host docker images for various subproject without
> having to check out several source tree to trigger a grand build. However,
> inline build process seems more popular than separated process. Hence, I
> highly recommend making docker build inline if possible.
> {quote}
> The main challenges are also discussed in the thread:
> {code:java}
> 3. Technically it would be possible to add the Dockerfile to the source
> tree and publish the docker image together with the release by the
> release manager but it's also problematic:
> {code}
> a) there is no easy way to stage the images for the vote
> c) it couldn't be flagged as automated on dockerhub
> d) It couldn't support the critical updates.
> * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> * Creating image for older releases (We would like to provide images,
> for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
> with different versions).
> {code:java}
> {code}
> The a) can be solved (as [~eyang] suggested) with using a personal docker
> image during the vote and publish it to the dockerhub after the vote (in case
> the permission can be set by the INFRA)
> Note: based on LEGAL-270 and linked discussion both approaches (inline build
> process / external build process) are compatible with the apache release.
> Note: HDDS-851 and HADOOP-14898 contains more information about these
> problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?focusedWorklogId=260710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260710 ] ASF GitHub Bot logged work on HDDS-1601: Author: ASF GitHub Bot Created on: 14/Jun/19 21:01 Start Date: 14/Jun/19 21:01 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #972: HDDS-1601. Implement updating lastAppliedIndex after buffer flush to OM DB. URL: https://github.com/apache/hadoop/pull/972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260710) Time Spent: 10m Remaining Estimate: 0h > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement updating lastAppliedIndex in > OzoneManagerStateMachine once after the buffer is flushed to OM DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1601) Implement updating lastAppliedIndex after buffer flush to OM DB.
[ https://issues.apache.org/jira/browse/HDDS-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1601: - Labels: pull-request-available (was: ) > Implement updating lastAppliedIndex after buffer flush to OM DB. > > > Key: HDDS-1601 > URL: https://issues.apache.org/jira/browse/HDDS-1601 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This Jira is to implement updating lastAppliedIndex in > OzoneManagerStateMachine once after the buffer is flushed to OM DB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[
https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864438#comment-16864438
]
Íñigo Goiri commented on HDFS-14563:
I had forgotten that even {{refreshNodes()}} goes through the
{{DistributedFileSystem}}.
I would have expected a separate admin interface but I guess that's the
framework we have.
* What do we want to do with the REST interfaces?
* I think we should have a full test adding a node, checking that is there,
adding another, removing, checking, etc.
* What do we want to do with the state of this? Do we write to the file? ZK?
> Enhance interface about recommissioning/decommissioning
> ---
>
> Key: HDFS-14563
> URL: https://issues.apache.org/jira/browse/HDFS-14563
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14563.001.patch
>
>
> In current implementation, if we need to decommissioning or recommissioning
> one datanode, the only way is add the datanode to include or exclude file
> under namenode configuration path then execute command `bin/hadoop dfsadmin
> -refreshNodes` and trigger namenode to reload include/exclude and start to
> recommissioning or decommissioning datanode.
> The shortcomings of this approach is that:
> a. namenode reload include/exclude configuration file from devices, if I/O
> load is high, handler may be blocked.
> b. namenode has to process every datnodes in include and exclude
> configurations, if there are many datanodes (very common for large cluster)
> pending to process, namenode will be hung for hundred seconds to wait
> recommision/decommision finish at the worst since holding write lock.
> I think we should expose one lightweight interface to support recommissioning
> or decommissioning single datanode, thus we can operate datanode using
> dfsadmin more smooth.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14549) EditLogTailer shouldn't output full stack trace when interrupted
[
https://issues.apache.org/jira/browse/HDFS-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864428#comment-16864428
]
Giovanni Matteo Fumarola commented on HDFS-14549:
-
Thanks [~elgoiri] for the patch. Committed to trunk.
I saw it a few times in the log files.
> EditLogTailer shouldn't output full stack trace when interrupted
>
>
> Key: HDFS-14549
> URL: https://issues.apache.org/jira/browse/HDFS-14549
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14549.000.patch
>
>
> Currently, when the NN transitions to active, it interrupts the EditLogTailer
> and outputs the full stack trace.
> {code}
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> {code}
> I don't think this adds any value and it should just print the message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14549) EditLogTailer shouldn't output full stack trace when interrupted
[
https://issues.apache.org/jira/browse/HDFS-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated HDFS-14549:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> EditLogTailer shouldn't output full stack trace when interrupted
>
>
> Key: HDFS-14549
> URL: https://issues.apache.org/jira/browse/HDFS-14549
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14549.000.patch
>
>
> Currently, when the NN transitions to active, it interrupts the EditLogTailer
> and outputs the full stack trace.
> {code}
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> {code}
> I don't think this adds any value and it should just print the message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14549) EditLogTailer shouldn't output full stack trace when interrupted
[
https://issues.apache.org/jira/browse/HDFS-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giovanni Matteo Fumarola updated HDFS-14549:
Fix Version/s: 3.3.0
> EditLogTailer shouldn't output full stack trace when interrupted
>
>
> Key: HDFS-14549
> URL: https://issues.apache.org/jira/browse/HDFS-14549
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14549.000.patch
>
>
> Currently, when the NN transitions to active, it interrupts the EditLogTailer
> and outputs the full stack trace.
> {code}
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> {code}
> I don't think this adds any value and it should just print the message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14549) EditLogTailer shouldn't output full stack trace when interrupted
[
https://issues.apache.org/jira/browse/HDFS-14549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864424#comment-16864424
]
Hudson commented on HDFS-14549:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16748 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16748/])
HDFS-14549. EditLogTailer shouldn't output full stack trace when (gifuma: rev
b24efa11ea2b3ecbae6578058aea89b6823d18d8)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java
> EditLogTailer shouldn't output full stack trace when interrupted
>
>
> Key: HDFS-14549
> URL: https://issues.apache.org/jira/browse/HDFS-14549
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.0
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-14549.000.patch
>
>
> Currently, when the NN transitions to active, it interrupts the EditLogTailer
> and outputs the full stack trace.
> {code}
> 2013-10-02 00:13:49,701 WARN ha.EditLogTailer
> (EditLogTailer.java:doWork(336)) - Edit log tailer interrupted
> java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:334)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:296)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1463)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:454)
> at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTail
> {code}
> I don't think this adds any value and it should just print the message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864418#comment-16864418
]
Hadoop QA commented on HDFS-12914:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:63396be |
| JIRA Issue | HDFS-12914 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971832/HDFS-12914.branch-3.2.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28ef5c5510ba 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 335aebb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26958/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26958/testReport/ |
| Max. process+thread count | 3967 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HDFS-14568) The quota and consume of the file's ancestors are not handled when the storage policy of the file is changed.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864401#comment-16864401
]
Hadoop QA commented on HDFS-14568:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}165m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14568 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12971801/HDFS-14568-unit-test.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 23c49dde4915 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3ba090f |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26957/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26957/testReport/ |
| Max. process+thread count | 3405 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864396#comment-16864396
]
Hudson commented on HDFS-12914:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16747 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16747/])
HDFS-12914. Addendum patch. Block report leases cause missing blocks (weichiu:
rev cdc5de6448e429d6cb523b8a61bed8b1cb2fc263)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-12914:
---
Attachment: (was: HDFS-12914.branch-3.2.patch)
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1684) OM should create Ratis related dirs only if ratis is enabled
[ https://issues.apache.org/jira/browse/HDDS-1684?focusedWorklogId=260674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-260674 ] ASF GitHub Bot logged work on HDDS-1684: Author: ASF GitHub Bot Created on: 14/Jun/19 19:36 Start Date: 14/Jun/19 19:36 Worklog Time Spent: 10m Work Description: hanishakoneru commented on issue #965: HDDS-1684. OM should create Ratis related dirs only if ratis is enabled URL: https://github.com/apache/hadoop/pull/965#issuecomment-502237146 /retest This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 260674) Time Spent: 40m (was: 0.5h) > OM should create Ratis related dirs only if ratis is enabled > > > Key: HDDS-1684 > URL: https://issues.apache.org/jira/browse/HDDS-1684 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > In OM, Ratis related dirs (storage, snapshot etc.) should only be created if > OM ratis is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864352#comment-16864352 ] He Xiaoqiao edited comment on HDFS-14563 at 6/14/19 6:50 PM: - Thanks [~sodonnell],[~elgoiri] for your detailed analysis. I believe that we can improve it through move DNS resolution out of holding write lock. I will watch the new JIRA and I would like to supply some contribution if [~elgoiri] need. Just upload the draft version for the interface, I want to state that this issue still open discussion and welcome any suggestions. Thanks again. was (Author: hexiaoqiao): Thanks [~sodonnell] for your detailed analysis. I believe that we can improve it through move DNS resolution out of holding write lock. I will watch the new JIRA and I would like to supply some contribution if [~elgoiri] need. Just upload the draft version for the interface, I want to state that this issue still open and welcome any suggestions. Thanks again. > Enhance interface about recommissioning/decommissioning > --- > > Key: HDFS-14563 > URL: https://issues.apache.org/jira/browse/HDFS-14563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14563.001.patch > > > In current implementation, if we need to decommissioning or recommissioning > one datanode, the only way is add the datanode to include or exclude file > under namenode configuration path then execute command `bin/hadoop dfsadmin > -refreshNodes` and trigger namenode to reload include/exclude and start to > recommissioning or decommissioning datanode. > The shortcomings of this approach is that: > a. namenode reload include/exclude configuration file from devices, if I/O > load is high, handler may be blocked. > b. namenode has to process every datnodes in include and exclude > configurations, if there are many datanodes (very common for large cluster) > pending to process, namenode will be hung for hundred seconds to wait > recommision/decommision finish at the worst since holding write lock. > I think we should expose one lightweight interface to support recommissioning > or decommissioning single datanode, thus we can operate datanode using > dfsadmin more smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864352#comment-16864352 ] He Xiaoqiao commented on HDFS-14563: Thanks [~sodonnell] for your detailed analysis. I believe that we can improve it through move DNS resolution out of holding write lock. I will watch the new JIRA and I would like to supply some contribution if [~elgoiri] need. Just upload the draft version for the interface, I want to state that this issue still open and welcome any suggestions. Thanks again. > Enhance interface about recommissioning/decommissioning > --- > > Key: HDFS-14563 > URL: https://issues.apache.org/jira/browse/HDFS-14563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14563.001.patch > > > In current implementation, if we need to decommissioning or recommissioning > one datanode, the only way is add the datanode to include or exclude file > under namenode configuration path then execute command `bin/hadoop dfsadmin > -refreshNodes` and trigger namenode to reload include/exclude and start to > recommissioning or decommissioning datanode. > The shortcomings of this approach is that: > a. namenode reload include/exclude configuration file from devices, if I/O > load is high, handler may be blocked. > b. namenode has to process every datnodes in include and exclude > configurations, if there are many datanodes (very common for large cluster) > pending to process, namenode will be hung for hundred seconds to wait > recommision/decommision finish at the worst since holding write lock. > I think we should expose one lightweight interface to support recommissioning > or decommissioning single datanode, thus we can operate datanode using > dfsadmin more smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864349#comment-16864349
]
Wei-Chiu Chuang commented on HDFS-12914:
I'm sorry I forgot to add the test code in to git commit. Doing that now.
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Xiaoqiao updated HDFS-14563: --- Attachment: HDFS-14563.001.patch Status: Patch Available (was: Open) > Enhance interface about recommissioning/decommissioning > --- > > Key: HDFS-14563 > URL: https://issues.apache.org/jira/browse/HDFS-14563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > Attachments: HDFS-14563.001.patch > > > In current implementation, if we need to decommissioning or recommissioning > one datanode, the only way is add the datanode to include or exclude file > under namenode configuration path then execute command `bin/hadoop dfsadmin > -refreshNodes` and trigger namenode to reload include/exclude and start to > recommissioning or decommissioning datanode. > The shortcomings of this approach is that: > a. namenode reload include/exclude configuration file from devices, if I/O > load is high, handler may be blocked. > b. namenode has to process every datnodes in include and exclude > configurations, if there are many datanodes (very common for large cluster) > pending to process, namenode will be hung for hundred seconds to wait > recommision/decommision finish at the worst since holding write lock. > I think we should expose one lightweight interface to support recommissioning > or decommissioning single datanode, thus we can operate datanode using > dfsadmin more smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14572) Disk Maintenance
Scott A. Wehner created HDFS-14572: -- Summary: Disk Maintenance Key: HDFS-14572 URL: https://issues.apache.org/jira/browse/HDFS-14572 Project: Hadoop HDFS Issue Type: New Feature Components: balancer mover, block placement, datanode Environment: hot swappable disks Reporter: Scott A. Wehner Like putting datanodes into maintenance mode, I don't want to have to reduce capacity to swap out failing drives in a hot swappable environment. I'd like to have a list of datanodes and storageIDs or datanode and mountpoints in a file that can act like the nodes in mainenance mode. This can ensure that blocks are replicated correctly and all replicas that exist on failing but not failed disks can be read off and replicated while the volumes are read only in hdfs land. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14571) Command line to force volume failures
Scott A. Wehner created HDFS-14571: -- Summary: Command line to force volume failures Key: HDFS-14571 URL: https://issues.apache.org/jira/browse/HDFS-14571 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, hdfs Environment: Linux Reporter: Scott A. Wehner Datanodes that have failed hard drives reports to the namenode that it has a failed volume in line with enabling slow datanode detection and we have a failing drive that has not failed, or has uncorrectable sectors, I want to be able to run a command to force fail a datanode volume based on storageID or Target Storage location (a.k.a mount point). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14570) Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled property back into the hdfs-site.xml
Scott A. Wehner created HDFS-14570: -- Summary: Bring back ability to totally disable webhdfs by bringing dfs.webhdfs.enabled property back into the hdfs-site.xml Key: HDFS-14570 URL: https://issues.apache.org/jira/browse/HDFS-14570 Project: Hadoop HDFS Issue Type: Wish Components: webhdfs Affects Versions: 3.1.2, 3.0.3, 3.1.1, 3.2.0, 3.0.2, 3.0.1, 3.1.0, 3.0.0 Reporter: Scott A. Wehner Fix For: 2.9.2 We don't want to enable security for viewing namenode http page, but we don't want people to be able to modify the contents of hdfs through anonymous access to the namenode page. in Hadoop 3 we lost the ability to totally disable webhdfs. want to bring this back, doesn't seem to hard to do, but makes it important in our environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-12914:
---
Fix Version/s: 3.3.0
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-12914:
---
Attachment: HDFS-12914.branch-3.2.patch
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864317#comment-16864317
]
Wei-Chiu Chuang commented on HDFS-12914:
Pushed v008 to trunk. Branch-3.2 doesn't compile because of HDFS-13898. I'm
attaching an updated patch.
The only difference is the addition of
{{FSNamesystem#setBlockManagerForTesting()}}
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch,
> HDFS-12914.branch-3.2.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12914) Block report leases cause missing blocks until next report
[
https://issues.apache.org/jira/browse/HDFS-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864303#comment-16864303
]
Hudson commented on HDFS-12914:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16746 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16746/])
HDFS-12914. Block report leases cause missing blocks until next report.
(weichiu: rev ae4143a529d74d94f205ca627c31360abfa11bfa)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
> Block report leases cause missing blocks until next report
> --
>
> Key: HDFS-12914
> URL: https://issues.apache.org/jira/browse/HDFS-12914
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.8.0, 2.9.2
>Reporter: Daryn Sharp
>Assignee: Santosh Marella
>Priority: Critical
> Attachments: HDFS-12914-branch-2.001.patch,
> HDFS-12914-trunk.00.patch, HDFS-12914-trunk.01.patch, HDFS-12914.005.patch,
> HDFS-12914.006.patch, HDFS-12914.007.patch, HDFS-12914.008.patch
>
>
> {{BlockReportLeaseManager#checkLease}} will reject FBRs from DNs for
> conditions such as "unknown datanode", "not in pending set", "lease has
> expired", wrong lease id, etc. Lease rejection does not throw an exception.
> It returns false which bubbles up to {{NameNodeRpcServer#blockReport}} and
> interpreted as {{noStaleStorages}}.
> A re-registering node whose FBR is rejected from an invalid lease becomes
> active with _no blocks_. A replication storm ensues possibly causing DNs to
> temporarily go dead (HDFS-12645), leading to more FBR lease rejections on
> re-registration. The cluster will have many "missing blocks" until the DNs
> next FBR is sent and/or forced.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14565) RBF: updateMount should validate the newEntry in RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14565: --- Resolution: Duplicate Status: Resolved (was: Patch Available) > RBF: updateMount should validate the newEntry in RouterAdmin > > > Key: HDFS-14565 > URL: https://issues.apache.org/jira/browse/HDFS-14565 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: xuzq >Priority: Major > Labels: RBF > Attachments: HDFS-14565-HDFS-13891-001.patch > > > RouterAdmin should validate the new mountEntry in updateMount. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14565) RBF: updateMount should validate the newEntry in RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864288#comment-16864288 ] Íñigo Goiri commented on HDFS-14565: Actually, this validate() call is already added by HDFS-14316 in HDFS-13891. > RBF: updateMount should validate the newEntry in RouterAdmin > > > Key: HDFS-14565 > URL: https://issues.apache.org/jira/browse/HDFS-14565 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: xuzq >Priority: Major > Labels: RBF > Attachments: HDFS-14565-HDFS-13891-001.patch > > > RouterAdmin should validate the new mountEntry in updateMount. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14565) RBF: updateMount should validate the newEntry in RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14565: --- Issue Type: Sub-task (was: Bug) Parent: HDFS-13891 > RBF: updateMount should validate the newEntry in RouterAdmin > > > Key: HDFS-14565 > URL: https://issues.apache.org/jira/browse/HDFS-14565 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: xuzq >Priority: Major > Labels: RBF > Attachments: HDFS-14565-HDFS-13891-001.patch > > > RouterAdmin should validate the new mountEntry in updateMount. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14565) RBF: updateMount should validate the newEntry in RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14565: --- Status: Patch Available (was: Open) > RBF: updateMount should validate the newEntry in RouterAdmin > > > Key: HDFS-14565 > URL: https://issues.apache.org/jira/browse/HDFS-14565 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: xuzq >Priority: Major > Labels: RBF > Attachments: HDFS-14565-HDFS-13891-001.patch > > > RouterAdmin should validate the new mountEntry in updateMount. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14565) RBF: updateMount should validate the newEntry in RouterAdmin
[ https://issues.apache.org/jira/browse/HDFS-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864287#comment-16864287 ] Íñigo Goiri commented on HDFS-14565: Is there a unit test we can add to validate this? > RBF: updateMount should validate the newEntry in RouterAdmin > > > Key: HDFS-14565 > URL: https://issues.apache.org/jira/browse/HDFS-14565 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf >Reporter: xuzq >Priority: Major > Labels: RBF > Attachments: HDFS-14565-HDFS-13891-001.patch > > > RouterAdmin should validate the new mountEntry in updateMount. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864285#comment-16864285
]
CR Hota commented on HDFS-14090:
[~elgoiri] Thanks for your previous detailed review. Took care of most except
the below.
005.patch looks ok from Yetus perspective.
{quote}I think we can use ReflectionUtils instead of managing the constructor
and so on in FairnessPolicyController.
{quote}
This doesn't allow passing constructors with objects. Rather I think
FederationUtil is a good place to modify and utilize which is also used for
namenode resolver, state store etc. Needs changes to FederationUtil.
FederationUtil either allows constructor with no params or params with context
class and not just conf. Would like to track that through a separate jira.
{quote}Should we assign fairnessPolicyController to null when shutting down?
{quote}
Left it as is, to keep it consistent with other objects created. Added a
shutdown interface, to allow different implementations of FairnessPolicy to
manage any resources if needed and cleanly shutdown.
> RBF: Improved isolation for downstream name nodes.
> --
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14545) RBF: Router should support GetUserMappingsProtocol
[ https://issues.apache.org/jira/browse/HDFS-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864280#comment-16864280 ] Íñigo Goiri commented on HDFS-14545: Thanks [~ayushtkn] for working on this and [~shenyinjie] and [~lukmajercak] for the reviews. Committed to HDFS-13891. > RBF: Router should support GetUserMappingsProtocol > -- > > Key: HDFS-14545 > URL: https://issues.apache.org/jira/browse/HDFS-14545 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Ayush Saxena >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14545-HDFS-13891-01.patch, > HDFS-14545-HDFS-13891-02.patch, HDFS-14545-HDFS-13891-03.patch, > HDFS-14545-HDFS-13891-04.patch, HDFS-14545-HDFS-13891-05.patch, > HDFS-14545-HDFS-13891-06.patch, HDFS-14545-HDFS-13891-07.patch, > HDFS-14545-HDFS-13891-08.patch, HDFS-14545-HDFS-13891-09.patch, > HDFS-14545-HDFS-13891-10.patch, HDFS-14545-HDFS-13891.000.patch > > > We should be able to check the groups for a user from a Router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14545) RBF: Router should support GetUserMappingsProtocol
[ https://issues.apache.org/jira/browse/HDFS-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14545: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-13891 Status: Resolved (was: Patch Available) > RBF: Router should support GetUserMappingsProtocol > -- > > Key: HDFS-14545 > URL: https://issues.apache.org/jira/browse/HDFS-14545 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Ayush Saxena >Priority: Major > Fix For: HDFS-13891 > > Attachments: HDFS-14545-HDFS-13891-01.patch, > HDFS-14545-HDFS-13891-02.patch, HDFS-14545-HDFS-13891-03.patch, > HDFS-14545-HDFS-13891-04.patch, HDFS-14545-HDFS-13891-05.patch, > HDFS-14545-HDFS-13891-06.patch, HDFS-14545-HDFS-13891-07.patch, > HDFS-14545-HDFS-13891-08.patch, HDFS-14545-HDFS-13891-09.patch, > HDFS-14545-HDFS-13891-10.patch, HDFS-14545-HDFS-13891.000.patch > > > We should be able to check the groups for a user from a Router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14545) RBF: Router should support GetUserMappingsProtocol
[ https://issues.apache.org/jira/browse/HDFS-14545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864275#comment-16864275 ] Íñigo Goiri commented on HDFS-14545: +1 on [^HDFS-14545-HDFS-13891-10.patch]. Committing to the branch. > RBF: Router should support GetUserMappingsProtocol > -- > > Key: HDFS-14545 > URL: https://issues.apache.org/jira/browse/HDFS-14545 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14545-HDFS-13891-01.patch, > HDFS-14545-HDFS-13891-02.patch, HDFS-14545-HDFS-13891-03.patch, > HDFS-14545-HDFS-13891-04.patch, HDFS-14545-HDFS-13891-05.patch, > HDFS-14545-HDFS-13891-06.patch, HDFS-14545-HDFS-13891-07.patch, > HDFS-14545-HDFS-13891-08.patch, HDFS-14545-HDFS-13891-09.patch, > HDFS-14545-HDFS-13891-10.patch, HDFS-14545-HDFS-13891.000.patch > > > We should be able to check the groups for a user from a Router. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14563) Enhance interface about recommissioning/decommissioning
[ https://issues.apache.org/jira/browse/HDFS-14563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864271#comment-16864271 ] Íñigo Goiri commented on HDFS-14563: I would open a new JIRA to move the DNS resolution out of this synchronized area. I still think changing the interface to add/remove nodes into decommission would be useful. > Enhance interface about recommissioning/decommissioning > --- > > Key: HDFS-14563 > URL: https://issues.apache.org/jira/browse/HDFS-14563 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: He Xiaoqiao >Assignee: He Xiaoqiao >Priority: Major > > In current implementation, if we need to decommissioning or recommissioning > one datanode, the only way is add the datanode to include or exclude file > under namenode configuration path then execute command `bin/hadoop dfsadmin > -refreshNodes` and trigger namenode to reload include/exclude and start to > recommissioning or decommissioning datanode. > The shortcomings of this approach is that: > a. namenode reload include/exclude configuration file from devices, if I/O > load is high, handler may be blocked. > b. namenode has to process every datnodes in include and exclude > configurations, if there are many datanodes (very common for large cluster) > pending to process, namenode will be hung for hundred seconds to wait > recommision/decommision finish at the worst since holding write lock. > I think we should expose one lightweight interface to support recommissioning > or decommissioning single datanode, thus we can operate datanode using > dfsadmin more smooth. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14568) The quota and consume of the file's ancestors are not handled when the storage policy of the file is changed.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-14568:
---
Status: Patch Available (was: Open)
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed.
> -
>
> Key: HDFS-14568
> URL: https://issues.apache.org/jira/browse/HDFS-14568
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-14568-unit-test.patch
>
>
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed. For example:
> First I set quota StorageType.SSD 2*fileSpace to the parent dir;
> Then I create a file with len fileSpace with \{DISK,DISK,DISK} under it;
> Finally I change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME
> and expect a QuotaByStorageTypeExceededException. But there isn't any
> exception.
> I'm not sure it is a bug or originally designed to skip the quota check. But
> having the check does make sense. I'll upload a unit test to show the case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14566) Quota improvement in HDFS
[ https://issues.apache.org/jira/browse/HDFS-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14566: --- Summary: Quota improvement in HDFS (was: Quota improvment in HDFS) > Quota improvement in HDFS > - > > Key: HDFS-14566 > URL: https://issues.apache.org/jira/browse/HDFS-14566 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Priority: Major > > The backgroud is we are considering upgrading our HDFS clusters from 2.6 to > 3.1. I'm doing a review about the new code related to quota and find there > might be some improvments about the new features. This is a parent jira for > the improvements and issues about quota. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14090) RBF: Improved isolation for downstream name nodes.
[
https://issues.apache.org/jira/browse/HDFS-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864267#comment-16864267
]
CR Hota commented on HDFS-14090:
[~linyiqun] Thanks for the review.
{quote}I'm not sure if this was implemented now. From my impression, the client
side only connect to the one specific Router. And the StandbyException will
trigger the router client to failover to another NN not another Router.
{quote}
Client side change was done in HDFS-6648 to randomize selecting a router while
connecting.
{quote}Even though the client side can failover to another Router, the slowness
issue can still exist in downstream NN. That is say, the failover behavior
across Routers is meaningless to improve the slowness in downstream NN when NN
is busy. The new Router will also quickly reach the permit limitation when NN
is in busy state. So the correct behavior maybe that throw a normal IOException
like Fair Call Queue bakoff logic did.
{quote}
The use case here is to not solve slowness, router doesn't intend to solve
that. The main use case here is if a namenode is slow, healthy clients
connecting to healthy clusters should be ok. Also requests to unhealthy
clusters should be as distributed as possible across all routers to help get
the maximum possible throughput. A normal IOException would mean clogging on
specific routers even though permits may become available on others. A bouncing
behavior distributes load appropriately. Faircall queue backoff isn't quite
applicable here as routers are NOT the ultimate destination. For NN it is and
hence a backoff works ok since clients have to come back to the same NN with
request. In case of routers the same request can be taken by any other router
in the fleet.
> RBF: Improved isolation for downstream name nodes.
> --
>
> Key: HDFS-14090
> URL: https://issues.apache.org/jira/browse/HDFS-14090
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14090-HDFS-13891.001.patch,
> HDFS-14090-HDFS-13891.002.patch, HDFS-14090-HDFS-13891.003.patch,
> HDFS-14090-HDFS-13891.004.patch, HDFS-14090-HDFS-13891.005.patch, RBF_
> Isolation design.pdf
>
>
> Router is a gateway to underlying name nodes. Gateway architectures, should
> help minimize impact of clients connecting to healthy clusters vs unhealthy
> clusters.
> For example - If there are 2 name nodes downstream, and one of them is
> heavily loaded with calls spiking rpc queue times, due to back pressure the
> same with start reflecting on the router. As a result of this, clients
> connecting to healthy/faster name nodes will also slow down as same rpc queue
> is maintained for all calls at the router layer. Essentially the same IPC
> thread pool is used by router to connect to all name nodes.
> Currently router uses one single rpc queue for all calls. Lets discuss how we
> can change the architecture and add some throttling logic for
> unhealthy/slow/overloaded name nodes.
> One way could be to read from current call queue, immediately identify
> downstream name node and maintain a separate queue for each underlying name
> node. Another simpler way is to maintain some sort of rate limiter configured
> for each name node and let routers drop/reject/send error requests after
> certain threshold.
> This won’t be a simple change as router’s ‘Server’ layer would need redesign
> and implementation. Currently this layer is the same as name node.
> Opening this ticket to discuss, design and implement this feature.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
