[jira] [Updated] (HDDS-1289) get Key failed on SCM restart
[
https://issues.apache.org/jira/browse/HDDS-1289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1289:
---
Target Version/s: 0.4.0
> get Key failed on SCM restart
> -
>
> Key: HDDS-1289
> URL: https://issues.apache.org/jira/browse/HDDS-1289
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Critical
> Attachments:
> hadoop-hdfs-scm-ctr-e139-1542663976389-86524-01-03.log
>
>
> Seeing ContainerNotFoundException in scm log when get key operation tried
> after scm restart.
> scm.log:
> [^hadoop-hdfs-scm-ctr-e139-1542663976389-86524-01-03.log]
>
> {noformat}
>
>
> ozone version :
>
> Source code repository g...@github.com:hortonworks/ozone.git -r
> 67b7c4fd071b3f557bdb54be2a266b8a611cbce6
> Compiled by jenkins on 2019-03-06T22:02Z
> Compiled with protoc 2.5.0
> From source with checksum 65be9a337d178cd3855f5c5a2f111
> Using HDDS 0.4.0.3.0.100.0-348
> Source code repository g...@github.com:hortonworks/ozone.git -r
> 67b7c4fd071b3f557bdb54be2a266b8a611cbce6
> Compiled by jenkins on 2019-03-06T22:01Z
> Compiled with protoc 2.5.0
> From source with checksum 324109cb3e8b188c1b89dc0b328c3a
> root@ctr-e139-1542663976389-86524-01-06 hdfs# hadoop version
> Hadoop 3.1.1.3.0.100.0-348
> Source code repository g...@github.com:hortonworks/hadoop.git -r
> 484434b1c2480bdc9314a7ee1ade8a0f4db1758f
> Compiled by jenkins on 2019-03-06T22:14Z
> Compiled with protoc 2.5.0
> From source with checksum ba6aad94c14256ef3ad8634e3b5086
> This command was run using
> /usr/hdp/3.0.100.0-348/hadoop/hadoop-common-3.1.1.3.0.100.0-348.jar
> {noformat}
>
>
>
> {noformat}
> 2019-03-13 17:00:54,348 ERROR container.ContainerReportHandler
> (ContainerReportHandler.java:processContainerReplicas(173)) - Received
> container report for an unknown container 22 from datanode
> 80f046cb-6fe2-4a05-bb67-9bf46f48723b{ip: 172.27.69.155, host:
> ctr-e139-1542663976389-86524-01-05.hwx.site} {}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #22 at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:543)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.updateContainerReplica(ContainerStateMap.java:230)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerReplica(ContainerStateManager.java:565)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerReplica(SCMContainerManager.java:393)
> at
> org.apache.hadoop.hdds.scm.container.ReportHandlerHelper.processContainerReplica(ReportHandlerHelper.java:74)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:159)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:110)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:51)
> at
> org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) 2019-03-13 17:00:54,349 ERROR
> container.ContainerReportHandler
> (ContainerReportHandler.java:processContainerReplicas(173)) - Received
> container report for an unknown container 23 from datanode
> 80f046cb-6fe2-4a05-bb67-9bf46f48723b{ip: 172.27.69.155, host:
> ctr-e139-1542663976389-86524-01-05.hwx.site} {}
> org.apache.hadoop.hdds.scm.container.ContainerNotFoundException: #23 at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.checkIfContainerExist(ContainerStateMap.java:543)
> at
> org.apache.hadoop.hdds.scm.container.states.ContainerStateMap.updateContainerReplica(ContainerStateMap.java:230)
> at
> org.apache.hadoop.hdds.scm.container.ContainerStateManager.updateContainerReplica(ContainerStateManager.java:565)
> at
> org.apache.hadoop.hdds.scm.container.SCMContainerManager.updateContainerReplica(SCMContainerManager.java:393)
> at
> org.apache.hadoop.hdds.scm.container.ReportHandlerHelper.processContainerReplica(ReportHandlerHelper.java:74)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.processContainerReplicas(ContainerReportHandler.java:159)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:110)
> at
> org.apache.hadoop.hdds.scm.container.ContainerReportHandler.onMessage(ContainerReportHandler.java:51)
> at
>
[jira] [Commented] (HDDS-1138) OzoneManager should return the pipeline info of the allocated block along with block info
[ https://issues.apache.org/jira/browse/HDDS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793762#comment-16793762 ] Jitendra Nath Pandey commented on HDDS-1138: I am +1 with the patch if jenkins runs are satisfactory. > OzoneManager should return the pipeline info of the allocated block along > with block info > - > > Key: HDDS-1138 > URL: https://issues.apache.org/jira/browse/HDDS-1138 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Manager >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Attachments: HDDS-1138.001.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently, while a block is allocated from OM, the request is forwarded to > SCM. However, even though the pipeline information is present with the OM for > block allocation, this information is passed through to the client. > This optimization will help in reducing the number of hops for the client by > reducing 1 RPC round trip for each block allocated. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1257) Incorrect object because of mismatch in block lengths
[
https://issues.apache.org/jira/browse/HDDS-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792837#comment-16792837
]
Jitendra Nath Pandey commented on HDDS-1257:
{code:java}
+ // totalAckDataLength replicated yet should always be less than equal to
+ // the current length being returned from commitIndex2flushedDataMap.
+ // The below precondition would ensure commitIndex2flushedDataMap entries
+ // are removed in order of the insertion to the map.
+ Preconditions.checkArgument(totalAckDataLength < length);{code}
The comment says the totalAckDataLength should be less than or equal, but the
check enforces strictly less.
> Incorrect object because of mismatch in block lengths
> -
>
> Key: HDDS-1257
> URL: https://issues.apache.org/jira/browse/HDDS-1257
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Fix For: 0.4.0, 0.5.0
>
> Attachments: HDDS-1257.000.patch, HDDS-1257.001.patch,
> HDDS-1257.002.patch,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient-output.txt,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.txt
>
>
> TestCloseContainerHandlingByClient fails with mismatch in block lengths. This
> mismatch is wrt to the written file length and committed.
> {code}
> TestCloseContainerHandlingByClient.testBlockWrites:466 expected:<5242880> but
> was:<4194304>
> {code}
> the test is failing with the following exception
> {code}
> [ERROR]
> testBlockWrites(org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient)
> Time elapsed: 1.777 s <<< FAILURE!
> java.lang.AssertionError: expected:<5242880> but was:<4194304>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.testBlockWrites(TestCloseContainerHandlingByClient.java:466)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To
[jira] [Updated] (HDDS-1257) Incorrect object because of mismatch in block lengths
[
https://issues.apache.org/jira/browse/HDDS-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1257:
---
Summary: Incorrect object because of mismatch in block lengths (was:
Failure in reading object because of mismatch in block lengths)
> Incorrect object because of mismatch in block lengths
> --
>
> Key: HDDS-1257
> URL: https://issues.apache.org/jira/browse/HDDS-1257
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Attachments: HDDS-1257.000.patch,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient-output.txt,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.txt
>
>
> TestCloseContainerHandlingByClient fails with mismatch in block lengths. This
> mismatch is wrt to the written file length and committed.
> {code}
> TestCloseContainerHandlingByClient.testBlockWrites:466 expected:<5242880> but
> was:<4194304>
> {code}
> the test is failing with the following exception
> {code}
> [ERROR]
> testBlockWrites(org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient)
> Time elapsed: 1.777 s <<< FAILURE!
> java.lang.AssertionError: expected:<5242880> but was:<4194304>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.testBlockWrites(TestCloseContainerHandlingByClient.java:466)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1257) Failure in reading object because of mismatch in block lengths
[
https://issues.apache.org/jira/browse/HDDS-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1257:
---
Summary: Failure in reading object because of mismatch in block lengths
(was: TestCloseContainerHandlingByClient.testBlockWrites fails becaue of
mismatch is block lengths)
> Failure in reading object because of mismatch in block lengths
> ---
>
> Key: HDDS-1257
> URL: https://issues.apache.org/jira/browse/HDDS-1257
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Attachments: HDDS-1257.000.patch,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient-output.txt,
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.txt
>
>
> TestCloseContainerHandlingByClient fails with mismatch in block lengths. This
> mismatch is wrt to the written file length and committed.
> {code}
> TestCloseContainerHandlingByClient.testBlockWrites:466 expected:<5242880> but
> was:<4194304>
> {code}
> the test is failing with the following exception
> {code}
> [ERROR]
> testBlockWrites(org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient)
> Time elapsed: 1.777 s <<< FAILURE!
> java.lang.AssertionError: expected:<5242880> but was:<4194304>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.testBlockWrites(TestCloseContainerHandlingByClient.java:466)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1254) Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer
[ https://issues.apache.org/jira/browse/HDDS-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16792107#comment-16792107 ] Jitendra Nath Pandey commented on HDDS-1254: [~ajayydv], is it going to be backported to 0.4.0? Please update the fix-version accordingly. > Fix failure in TestOzoneManagerHttpServer & > TestStorageContainerManagerHttpServer > - > > Key: HDDS-1254 > URL: https://issues.apache.org/jira/browse/HDDS-1254 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: blocker, pull-request-available, security > Time Spent: 1.5h > Remaining Estimate: 0h > > Investigate failure of Fix failure in TestOzoneManagerHttpServer & > TestStorageContainerManagerHttpServer in jenkins run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1254) Fix failure in TestOzoneManagerHttpServer & TestStorageContainerManagerHttpServer
[ https://issues.apache.org/jira/browse/HDDS-1254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791998#comment-16791998 ] Jitendra Nath Pandey commented on HDDS-1254: +1 for the patch in the PR. > Fix failure in TestOzoneManagerHttpServer & > TestStorageContainerManagerHttpServer > - > > Key: HDDS-1254 > URL: https://issues.apache.org/jira/browse/HDDS-1254 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: blocker, pull-request-available, security > Time Spent: 1h 20m > Remaining Estimate: 0h > > Investigate failure of Fix failure in TestOzoneManagerHttpServer & > TestStorageContainerManagerHttpServer in jenkins run. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-726) Ozone Client should update SCM to move the container out of allocation path in case a write transaction fails
[ https://issues.apache.org/jira/browse/HDDS-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-726: -- Fix Version/s: 0.4.0 > Ozone Client should update SCM to move the container out of allocation path > in case a write transaction fails > - > > Key: HDDS-726 > URL: https://issues.apache.org/jira/browse/HDDS-726 > Project: Hadoop Distributed Data Store > Issue Type: Test >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-726.000.patch, HDDS-726.001.patch, > HDDS-726.002.patch, HDDS-726.003.patch, HDDS-726.004.patch, > HDDS-726.005.patch, HDDS-726.006.patch, HDDS-726.007.patch, > HDDS-726.008.patch, HDDS-726.009.patch, HDDS-726.010.patch, HDDS-726.011.patch > > > Once an container write transaction fails, it will be marked corrupted. Once > Ozone client gets an exception in such case it should tell SCM to move the > container out of allocation path. SCM will eventually close the container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1241) Update ozone to latest ratis snapshot build (0.4.0-5680cf5-SNAPSHOT)
[ https://issues.apache.org/jira/browse/HDDS-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790738#comment-16790738 ] Jitendra Nath Pandey commented on HDDS-1241: +1, The patch looks good to me. > Update ozone to latest ratis snapshot build (0.4.0-5680cf5-SNAPSHOT) > > > Key: HDDS-1241 > URL: https://issues.apache.org/jira/browse/HDDS-1241 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Attachments: HDDS-1241.001.patch, HDDS-1241.002.patch > > > Update Ozone to latest ratis release 0.4.0-55030b2-SNAPSHOT. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1240) Fix check style issues caused by HDDS-1196
[ https://issues.apache.org/jira/browse/HDDS-1240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788856#comment-16788856 ] Jitendra Nath Pandey commented on HDDS-1240: [~bharatviswa], please push it to 0.4.0 as well, if it is relevant. And please update the fix version. > Fix check style issues caused by HDDS-1196 > -- > > Key: HDDS-1240 > URL: https://issues.apache.org/jira/browse/HDDS-1240 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Jenkins has not reported any issues, but found when working on another jira. > https://github.com/apache/hadoop/pull/567 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1221) Introduce fine grained lock in Ozone Manager for key operations
[ https://issues.apache.org/jira/browse/HDDS-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1221: --- Target Version/s: 0.5.0 (was: 0.4.0) > Introduce fine grained lock in Ozone Manager for key operations > --- > > Key: HDDS-1221 > URL: https://issues.apache.org/jira/browse/HDDS-1221 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > > Currently ozone manager acquires bucket lock for key operations in OM. We can > introduce fine grained lock for key operations in ozone manager. This would > help in increasing throughput for key operations in a bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1208) ContainerStateMachine should set chunk data as state machine data for ratis
[ https://issues.apache.org/jira/browse/HDDS-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1208: --- Fix Version/s: 0.5.0 > ContainerStateMachine should set chunk data as state machine data for ratis > --- > > Key: HDDS-1208 > URL: https://issues.apache.org/jira/browse/HDDS-1208 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1208.001.patch, HDDS-1208.002.patch, > HDDS-1208.003.patch > > > Currently ContainerStateMachine sets ContainerCommandRequestProto as state > machine data. This requires converting the ContainerCommandRequestProto to a > bytestring which leads to redundant buffer copy in case of write chunk > request. This can be avoided by setting the chunk data as the state machine > data for a log entry in ratis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1098) Introduce Retry Policy in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784980#comment-16784980 ] Jitendra Nath Pandey commented on HDDS-1098: [~shashikant], please rebase the patch. It doesn't apply even with HDDS-726 included. > Introduce Retry Policy in Ozone Client > -- > > Key: HDDS-1098 > URL: https://issues.apache.org/jira/browse/HDDS-1098 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Critical > Fix For: 0.4.0 > > Attachments: HDDS-1098.000.patch > > > Currently , ozone client indefinitely retries to failover to a new block > write in case it encounters exceptions during write. The aim here is to bring > in retry policy in the client so that indefinite retries should go away. > Ozone client code will also be refactored as a part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-725) Exception thrown in loop while trying to write a file in ozonefs
[
https://issues.apache.org/jira/browse/HDDS-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey resolved HDDS-725.
---
Resolution: Fixed
Fix Version/s: 0.4.0
I think this has been fixed. Please re-open if the issue re-surfaces.
> Exception thrown in loop while trying to write a file in ozonefs
>
>
> Key: HDDS-725
> URL: https://issues.apache.org/jira/browse/HDDS-725
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
>Affects Versions: 0.3.0
> Environment:
>
>Reporter: Nilotpal Nandi
>Priority: Blocker
> Labels: test-badlands
> Fix For: 0.4.0
>
> Attachments: all-node-ozone-logs-1540375264.tar.gz
>
>
> Ran the following command :
>
> ozone fs -put 2GB /testdir5/
> Exceptions are thrown continuously in loop. Please note that there are 8
> datanodes alive in the cluster.
> {noformat}
> root@ctr-e138-1518143905142-53-01-08 logs]# /root/allssh.sh 'jps -l |
> grep Datanode'
>
> Host::172.27.20.96
>
> 411564 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.20.91
>
> 472897 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.38.9
>
> 351139 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.24.90
>
> 314304 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.15.139
>
> 324820 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.10.199
>
>
> Host::172.27.15.131
>
>
> Host::172.27.57.0
>
>
> Host::172.27.23.139
>
> 627053 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.68.65
>
> 557443 org.apache.hadoop.ozone.HddsDatanodeService
>
> Host::172.27.19.74
>
>
> Host::172.27.85.64
>
> 508121 org.apache.hadoop.ozone.HddsDatanodeService{noformat}
>
> {noformat}
>
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: Election REJECTED; received 0
> response(s) [] and 2 exception(s);
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:t16296, leader=null,
> voted=7c3b2fb1-cf16-4e5f-94dc-8a089492ad57, raftlog=[(t:37, i:271)],
> conf=271: [7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:172.27.85.64:9858,
> 86f9e313-ae49-4675-95d7-27856641aee1:172.27.15.131:9858,
> 9524f4e2-9031-4852-ab7c-11c2da3460db:172.27.57.0:9858], old=null
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
> exception
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 1:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io
> exception
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from CANDIDATE to FOLLOWER
> at term 16296 for changeToFollower
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown LeaderElection
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start FollowerState
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.FollowerState:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes to CANDIDATE, lastRpcTime:1078,
> electionTimeout:1078ms
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RoleInfo:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown FollowerState
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from FOLLOWER to CANDIDATE
> at term 16296 for changeToCandidate
> 2018-10-24 09:49:48,172 INFO org.apache.ratis.server.impl.RoleInfo:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start LeaderElection
> 2018-10-24 09:49:48,173 INFO org.apache.ratis.server.impl.LeaderElection:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: begin an election in Term 16297
> 2018-10-24 09:49:48,174 INFO org.apache.ratis.server.impl.LeaderElection:
> 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 got exception when requesting votes: {}
>
[jira] [Commented] (HDDS-1210) Ratis pipeline creation doesn't check raft client reply status during initialization
[
https://issues.apache.org/jira/browse/HDDS-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784681#comment-16784681
]
Jitendra Nath Pandey commented on HDDS-1210:
+1 for the patch. Test failures could be related because now pipeline failures
are not ignored. However, please commit if they are unrelated.
> Ratis pipeline creation doesn't check raft client reply status during
> initialization
> -
>
> Key: HDDS-1210
> URL: https://issues.apache.org/jira/browse/HDDS-1210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1210.001.patch, HDDS-1210.002.patch,
> HDDS-1210.003.patch, HDDS-1210.004.patch, HDDS-1210.005.patch,
> HDDS-1210.006.patch
>
>
> Ratis pipeline are initialized using `raftClient.groupAdd`. However the
> pipeline initialization can fail and this can only be determined by
> raftClientReply status.
> {code}
> callRatisRpc(pipeline.getNodes(), ozoneConf,
> (raftClient, peer) -> raftClient.groupAdd(group, peer.getId()));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-935) Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart
[ https://issues.apache.org/jira/browse/HDDS-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784529#comment-16784529 ] Jitendra Nath Pandey commented on HDDS-935: --- +1 for the latest patch. > Avoid creating an already created container on a datanode in case of disk > removal followed by datanode restart > -- > > Key: HDDS-935 > URL: https://issues.apache.org/jira/browse/HDDS-935 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Rakesh R >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-935.000.patch, HDDS-935.001.patch, > HDDS-935.002.patch, HDDS-935.003.patch, HDDS-935.004.patch, > HDDS-935.005.patch, HDDS-935.006.patch, HDDS-935.007.patch, HDDS-935.008.patch > > > Currently, a container gets created when a writeChunk request comes to > HddsDispatcher and if the container does not exist already. In case a disk on > which a container exists gets removed and datanode restarts and now, if a > writeChunkRequest comes , it might end up creating the same container again > with an updated BCSID as it won't detect the disk is removed. This won't be > detected by SCM as well as it will have the latest BCSID. This Jira aims to > address this issue. > The proposed fix would be to persist the all the containerIds existing in the > containerSet when a ratis snapshot is taken in the snapshot file. If the disk > is removed and dn gets restarted, the container set will be rebuild after > scanning all the available disks and the the container list stored in the > snapshot file will give all the containers created in the datanode. The diff > between these two will give the exact list of containers which were created > but were not detected after the restart. Any writeChunk request now should > validate the container Id from the list of missing containers. Also, we need > to ensure container creation does not happen as part of applyTransaction of > writeChunk request in Ratis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-530) Fix Ozone fs valid uri pattern
[
https://issues.apache.org/jira/browse/HDDS-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783755#comment-16783755
]
Jitendra Nath Pandey commented on HDDS-530:
---
Yes, this can be closed. O3fs only accepts bucket.volume as the authority in
the URI.
> Fix Ozone fs valid uri pattern
> --
>
> Key: HDDS-530
> URL: https://issues.apache.org/jira/browse/HDDS-530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Reporter: Hanisha Koneru
>Assignee: Anu Engineer
>Priority: Blocker
> Labels: alpha2
>
> The valid uri pattern for an Ozone fs uri should be
> {{o3fs://://}}.
> But OzoneFileSystem accepts uri's of the form {{o3fs://.}}
> only.
> {code:java}
> # In OzoneFileSyste.java
> private static final Pattern URL_SCHEMA_PATTERN =
> Pattern.compile("(.+)\\.([^\\.]+)");
> if (!matcher.matches()) {
> throw new IllegalArgumentException("Ozone file system url should be "
> + "in the form o3fs://bucket.volume");
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDDS-530) Fix Ozone fs valid uri pattern
[
https://issues.apache.org/jira/browse/HDDS-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey resolved HDDS-530.
---
Resolution: Done
> Fix Ozone fs valid uri pattern
> --
>
> Key: HDDS-530
> URL: https://issues.apache.org/jira/browse/HDDS-530
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: documentation
>Reporter: Hanisha Koneru
>Assignee: Anu Engineer
>Priority: Blocker
> Labels: alpha2
>
> The valid uri pattern for an Ozone fs uri should be
> {{o3fs://://}}.
> But OzoneFileSystem accepts uri's of the form {{o3fs://.}}
> only.
> {code:java}
> # In OzoneFileSyste.java
> private static final Pattern URL_SCHEMA_PATTERN =
> Pattern.compile("(.+)\\.([^\\.]+)");
> if (!matcher.matches()) {
> throw new IllegalArgumentException("Ozone file system url should be "
> + "in the form o3fs://bucket.volume");
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-935) Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart
[ https://issues.apache.org/jira/browse/HDDS-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783657#comment-16783657 ] Jitendra Nath Pandey commented on HDDS-935: --- Please look at test failures and findbugs issue. > Avoid creating an already created container on a datanode in case of disk > removal followed by datanode restart > -- > > Key: HDDS-935 > URL: https://issues.apache.org/jira/browse/HDDS-935 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Rakesh R >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-935.000.patch, HDDS-935.001.patch, > HDDS-935.002.patch, HDDS-935.003.patch, HDDS-935.004.patch, > HDDS-935.005.patch, HDDS-935.006.patch > > > Currently, a container gets created when a writeChunk request comes to > HddsDispatcher and if the container does not exist already. In case a disk on > which a container exists gets removed and datanode restarts and now, if a > writeChunkRequest comes , it might end up creating the same container again > with an updated BCSID as it won't detect the disk is removed. This won't be > detected by SCM as well as it will have the latest BCSID. This Jira aims to > address this issue. > The proposed fix would be to persist the all the containerIds existing in the > containerSet when a ratis snapshot is taken in the snapshot file. If the disk > is removed and dn gets restarted, the container set will be rebuild after > scanning all the available disks and the the container list stored in the > snapshot file will give all the containers created in the datanode. The diff > between these two will give the exact list of containers which were created > but were not detected after the restart. Any writeChunk request now should > validate the container Id from the list of missing containers. Also, we need > to ensure container creation does not happen as part of applyTransaction of > writeChunk request in Ratis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-338) Using ozone shell and o3fs, possible to create file and directory with same name
[
https://issues.apache.org/jira/browse/HDDS-338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-338:
--
Target Version/s: 0.5.0 (was: 0.4.0)
> Using ozone shell and o3fs, possible to create file and directory with same
> name
>
>
> Key: HDDS-338
> URL: https://issues.apache.org/jira/browse/HDDS-338
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
>Reporter: Nilotpal Nandi
>Assignee: Hanisha Koneru
>Priority: Critical
> Attachments: HDDS-338.001.patch
>
>
> steps taken :
> --
> 1. created a directory through ozoneFS interface.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -mkdir /temp1/
> 2018-08-08 13:50:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:09:59 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> drwxrwxrwx - 0 2018-08-08 13:51 /temp1{noformat}
> 2. create a new key with name 'temp1' at same bucket.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/temp1
> -file /etc/passwd
> 2018-08-08 14:10:34 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 2:10:36 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_3.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Observed that there are multiple entries of 'temp1' when ozone fs -ls command
> is run. Also . both the entries are considered as file . '/temp1' directory
> is not visible anymore.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:10:41 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for
[jira] [Updated] (HDDS-338) Using ozone shell and o3fs, possible to create file and directory with same name
[
https://issues.apache.org/jira/browse/HDDS-338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-338:
--
Sprint: (was: HDDS BadLands)
> Using ozone shell and o3fs, possible to create file and directory with same
> name
>
>
> Key: HDDS-338
> URL: https://issues.apache.org/jira/browse/HDDS-338
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
>Reporter: Nilotpal Nandi
>Assignee: Hanisha Koneru
>Priority: Critical
> Attachments: HDDS-338.001.patch
>
>
> steps taken :
> --
> 1. created a directory through ozoneFS interface.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -mkdir /temp1/
> 2018-08-08 13:50:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:09:59 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> drwxrwxrwx - 0 2018-08-08 13:51 /temp1{noformat}
> 2. create a new key with name 'temp1' at same bucket.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/temp1
> -file /etc/passwd
> 2018-08-08 14:10:34 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 2:10:36 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_3.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Observed that there are multiple entries of 'temp1' when ozone fs -ls command
> is run. Also . both the entries are considered as file . '/temp1' directory
> is not visible anymore.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:10:41 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your
[jira] [Updated] (HDDS-338) Using ozone shell and o3fs, possible to create file and directory with same name
[
https://issues.apache.org/jira/browse/HDDS-338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-338:
--
Summary: Using ozone shell and o3fs, possible to create file and directory
with same name (was: ozoneFS allows to create file key and directory key with
same keyname)
> Using ozone shell and o3fs, possible to create file and directory with same
> name
>
>
> Key: HDDS-338
> URL: https://issues.apache.org/jira/browse/HDDS-338
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
>Reporter: Nilotpal Nandi
>Assignee: Hanisha Koneru
>Priority: Critical
> Attachments: HDDS-338.001.patch
>
>
> steps taken :
> --
> 1. created a directory through ozoneFS interface.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -mkdir /temp1/
> 2018-08-08 13:50:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:09:59 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> drwxrwxrwx - 0 2018-08-08 13:51 /temp1{noformat}
> 2. create a new key with name 'temp1' at same bucket.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/temp1
> -file /etc/passwd
> 2018-08-08 14:10:34 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 2:10:36 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_3.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Observed that there are multiple entries of 'temp1' when ozone fs -ls command
> is run. Also . both the entries are considered as file . '/temp1' directory
> is not visible anymore.
> {noformat}
>
[jira] [Commented] (HDDS-338) ozoneFS allows to create file key and directory key with same keyname
[
https://issues.apache.org/jira/browse/HDDS-338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783014#comment-16783014
]
Jitendra Nath Pandey commented on HDDS-338:
---
The problem has been addressed for o3fs, i.e. using o3fs we cannot create file
and directory of same name. However, the issue is still possible by mixing o3fs
access and ozone shell. I will update the summary.
> ozoneFS allows to create file key and directory key with same keyname
> -
>
> Key: HDDS-338
> URL: https://issues.apache.org/jira/browse/HDDS-338
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
>Reporter: Nilotpal Nandi
>Assignee: Hanisha Koneru
>Priority: Critical
> Attachments: HDDS-338.001.patch
>
>
> steps taken :
> --
> 1. created a directory through ozoneFS interface.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -mkdir /temp1/
> 2018-08-08 13:50:26 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> hadoop@1a1fa8a11332:~/bin$ ./ozone fs -ls /
> 2018-08-08 14:09:59 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Found 1 items
> drwxrwxrwx - 0 2018-08-08 13:51 /temp1{noformat}
> 2. create a new key with name 'temp1' at same bucket.
> {noformat}
> hadoop@1a1fa8a11332:~/bin$ ./ozone oz -putKey root-volume/root-bucket/temp1
> -file /etc/passwd
> 2018-08-08 14:10:34 WARN NativeCodeLoader:60 - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.rpc.type = GRPC (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.retryInterval = 300
> ms (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 -
> raft.client.async.outstanding-requests.max = 100 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.async.scheduler-threads =
> 3 (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.flow.control.window = 1MB
> (=1048576) (default)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.grpc.message.size.max = 33554432
> (custom)
> 2018-08-08 14:10:35 INFO ConfUtils:41 - raft.client.rpc.request.timeout =
> 3000 ms (default)
> Aug 08, 2018 2:10:36 PM
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl detectProxy
> WARNING: Failed to construct URI for proxy lookup, proceeding without proxy
> java.net.URISyntaxException: Illegal character in hostname at index 13:
> https://ozone_datanode_3.ozone_default:9858
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.parseHostname(URI.java:3387)
> at java.net.URI$Parser.parseServer(URI.java:3236)
> at java.net.URI$Parser.parseAuthority(URI.java:3155)
> at java.net.URI$Parser.parseHierarchical(URI.java:3097)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:673)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.detectProxy(ProxyDetectorImpl.java:128)
> at
> org.apache.ratis.shaded.io.grpc.internal.ProxyDetectorImpl.proxyFor(ProxyDetectorImpl.java:118)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:207)
> at
> org.apache.ratis.shaded.io.grpc.internal.InternalSubchannel.obtainActiveTransport(InternalSubchannel.java:188)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$SubchannelImpl.requestConnection(ManagedChannelImpl.java:1130)
> at
> org.apache.ratis.shaded.io.grpc.PickFirstBalancerFactory$PickFirstBalancer.handleResolvedAddressGroups(PickFirstBalancerFactory.java:79)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl$1NamesResolved.run(ManagedChannelImpl.java:1032)
> at
> org.apache.ratis.shaded.io.grpc.internal.ChannelExecutor.drain(ChannelExecutor.java:73)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$LbHelperImpl.runSerialized(ManagedChannelImpl.java:1000)
> at
> org.apache.ratis.shaded.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl.onAddresses(ManagedChannelImpl.java:1044)
> at
> org.apache.ratis.shaded.io.grpc.internal.DnsNameResolver$1.run(DnsNameResolver.java:201)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Observed that there are multiple entries of 'temp1' when ozone fs -ls command
> is run. Also . both the entries are considered as file . '/temp1'
[jira] [Resolved] (HDDS-772) ratis retries infinitely and does not timeout when datanode goes down
[
https://issues.apache.org/jira/browse/HDDS-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey resolved HDDS-772.
---
Resolution: Fixed
This got fixed as part of retry policy for ratis client.
> ratis retries infinitely and does not timeout when datanode goes down
> -
>
> Key: HDDS-772
> URL: https://issues.apache.org/jira/browse/HDDS-772
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.3.0
>Reporter: Nilotpal Nandi
>Priority: Critical
>
> steps taken :
> -
> # Ran ozonefs client operations.
> # Some of the datanodes were down.
> # client operations did not fail and are in waiting/hung state.
> reason: RATIS retries infinitely.
> datanode.log
>
>
> {noformat}
> 2018-10-31 11:13:28,423 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
> 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0
> org.apache.ratis.protocol.GroupMismatchException:
> 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found.
> at
> org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
> at
> org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-31 11:13:29,574 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
> 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0
> org.apache.ratis.protocol.GroupMismatchException:
> 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found.
> at
> org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
> at
> org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-31 11:13:30,772 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
>
[jira] [Updated] (HDDS-772) ratis retries infinitely and does not timeout when datanode goes down
[
https://issues.apache.org/jira/browse/HDDS-772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-772:
--
Fix Version/s: 0.4.0
> ratis retries infinitely and does not timeout when datanode goes down
> -
>
> Key: HDDS-772
> URL: https://issues.apache.org/jira/browse/HDDS-772
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.3.0
>Reporter: Nilotpal Nandi
>Priority: Critical
> Fix For: 0.4.0
>
>
> steps taken :
> -
> # Ran ozonefs client operations.
> # Some of the datanodes were down.
> # client operations did not fail and are in waiting/hung state.
> reason: RATIS retries infinitely.
> datanode.log
>
>
> {noformat}
> 2018-10-31 11:13:28,423 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
> 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0
> org.apache.ratis.protocol.GroupMismatchException:
> 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found.
> at
> org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
> at
> org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-31 11:13:29,574 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
> 54026017-a738-45f5-92f9-c50a0fc24a9f->046351fe-bb76-4f86-b296-c682746981c4#0
> org.apache.ratis.protocol.GroupMismatchException:
> 046351fe-bb76-4f86-b296-c682746981c4: group-FF58136AA1BA not found.
> at
> org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:257)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:266)
> at
> org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
> at
> org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:428)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2018-10-31 11:13:30,772 WARN
> org.apache.ratis.grpc.server.GrpcServerProtocolService:
> 046351fe-bb76-4f86-b296-c682746981c4: Failed requestVote
>
[jira] [Assigned] (HDDS-1162) Data Scrubbing for Ozone Containers
[ https://issues.apache.org/jira/browse/HDDS-1162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-1162: -- Assignee: Supratim Deka > Data Scrubbing for Ozone Containers > --- > > Key: HDDS-1162 > URL: https://issues.apache.org/jira/browse/HDDS-1162 > Project: Hadoop Distributed Data Store > Issue Type: New Feature > Components: Ozone Datanode, SCM >Reporter: Supratim Deka >Assignee: Supratim Deka >Priority: Major > Attachments: Container Scanner Redux.docx > > > General Reference: > [https://en.wikipedia.org/wiki/Data_scrubbing] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1210) Ratis pipeline creation doesn't raft client reply status during initialization
[
https://issues.apache.org/jira/browse/HDDS-1210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781861#comment-16781861
]
Jitendra Nath Pandey commented on HDDS-1210:
Thanks for the patch [~msingh], please add a unit test.
> Ratis pipeline creation doesn't raft client reply status during initialization
> --
>
> Key: HDDS-1210
> URL: https://issues.apache.org/jira/browse/HDDS-1210
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1210.001.patch
>
>
> Ratis pipeline are initialized using `raftClient.groupAdd`. However the
> pipeline initialization can fail and this can only be determined by
> raftClientReply status.
> {code}
> callRatisRpc(pipeline.getNodes(), ozoneConf,
> (raftClient, peer) -> raftClient.groupAdd(group, peer.getId()));
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1102) docker datanode stopped when new datanodes are added to the cluster
[
https://issues.apache.org/jira/browse/HDDS-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1102:
---
Target Version/s: 0.4.0
> docker datanode stopped when new datanodes are added to the cluster
>
>
> Key: HDDS-1102
> URL: https://issues.apache.org/jira/browse/HDDS-1102
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: allnode.log, datanode.log
>
>
> steps taken:
>
> # created 5 datanode cluster.
> # shutdown 2 datanodes
> # started the datanodes again.
> One of the datanodes was shut down.
> exception seen :
>
> {noformat}
> 2019-02-14 07:37:26 INFO LeaderElection:230 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at
> org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187)
> at
> org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-14 07:37:26 INFO LeaderElection:46 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s)
> [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7]
> and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null,
> voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8,
> raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3:
> [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858,
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858,
> 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null
> 2019-02-14 07:37:26 INFO LeaderElection:52 - 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8:
> shutdown LeaderElection
> 2019-02-14 07:37:26 INFO RaftServerImpl:161 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at
> term 7 for changeToLeader
> 2019-02-14 07:37:26 INFO RaftServerImpl:258 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader
> elected after 1066ms
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.staging.catchup.gap = 1000 (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time
> = 25ms (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout
> = 10s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.watch.timeout.denomination = 1s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
>
[jira] [Updated] (HDDS-1102) docker datanode stopped when new datanodes are added to the cluster
[
https://issues.apache.org/jira/browse/HDDS-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1102:
---
Component/s: Ozone Datanode
> docker datanode stopped when new datanodes are added to the cluster
>
>
> Key: HDDS-1102
> URL: https://issues.apache.org/jira/browse/HDDS-1102
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Nilotpal Nandi
>Priority: Major
> Attachments: allnode.log, datanode.log
>
>
> steps taken:
>
> # created 5 datanode cluster.
> # shutdown 2 datanodes
> # started the datanodes again.
> One of the datanodes was shut down.
> exception seen :
>
> {noformat}
> 2019-02-14 07:37:26 INFO LeaderElection:230 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at
> org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException:
> INTERNAL: a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
> at
> org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at
> org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:83)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:187)
> at
> org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-14 07:37:26 INFO LeaderElection:46 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: Election PASSED; received 1 response(s)
> [6a0522ba-019e-4b77-ac1f-a9322cd525b8<-61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5#0:OK-t7]
> and 1 exception(s); 6a0522ba-019e-4b77-ac1f-a9322cd525b8:t7, leader=null,
> voted=6a0522ba-019e-4b77-ac1f-a9322cd525b8,
> raftlog=6a0522ba-019e-4b77-ac1f-a9322cd525b8-SegmentedRaftLog:OPENED, conf=3:
> [61ad3bf3-e9b1-48e5-90e3-3b78c8b5bba5:172.20.0.8:9858,
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8:172.20.0.6:9858,
> 0f377918-aafa-4d8a-972a-6ead54048fba:172.20.0.3:9858], old=null
> 2019-02-14 07:37:26 INFO LeaderElection:52 - 0:
> java.util.concurrent.ExecutionException:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
> a3d1dd2d-554e-4e87-a2cf-076a229af352: group-FD6FA533F1FB not found.
> 2019-02-14 07:37:26 INFO RoleInfo:130 - 6a0522ba-019e-4b77-ac1f-a9322cd525b8:
> shutdown LeaderElection
> 2019-02-14 07:37:26 INFO RaftServerImpl:161 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 changes role from CANDIDATE to LEADER at
> term 7 for changeToLeader
> 2019-02-14 07:37:26 INFO RaftServerImpl:258 -
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8: change Leader from null to
> 6a0522ba-019e-4b77-ac1f-a9322cd525b8 at term 7 for becomeLeader, leader
> elected after 1066ms
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.staging.catchup.gap = 1000 (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.rpc.sleep.time
> = 25ms (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 - raft.server.watch.timeout
> = 10s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.watch.timeout.denomination = 1s (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
> 2019-02-14 07:37:26 INFO RaftServerConfigKeys:43 -
> raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
> 2019-02-14 07:37:26 INFO
[jira] [Updated] (HDDS-1124) java.lang.IllegalStateException exception in datanode log
[
https://issues.apache.org/jira/browse/HDDS-1124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1124:
---
Fix Version/s: (was: 0.5.0)
> java.lang.IllegalStateException exception in datanode log
> -
>
> Key: HDDS-1124
> URL: https://issues.apache.org/jira/browse/HDDS-1124
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Major
>
> steps taken :
>
> # created 12 datanodes cluster and running workload on all the nodes
> exception seen :
> ---
>
> {noformat}
> 2019-02-15 10:15:53,355 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3036
> to
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3036-3047
> 2019-02-15 10:15:53,367 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 943007c8-4fdd-4926-89e2-2c8c52c05073: set configuration 3048:
> [a40a7b01-a30b-469c-b373-9fcb20a126ed:172.27.54.212:9858,
> 8c77b16b-8054-49e3-b669-1ff759cfd271:172.27.23.196:9858,
> 943007c8-4fdd-4926-89e2-2c8c52c05073:172.27.76.72:9858], old=null at 3048
> 2019-02-15 10:15:53,523 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048
> 2019-02-15 10:15:53,580 ERROR org.apache.ratis.grpc.server.GrpcLogAppender:
> Failed onNext serverReply {
> requestorId: "943007c8-4fdd-4926-89e2-2c8c52c05073"
> replyId: "a40a7b01-a30b-469c-b373-9fcb20a126ed"
> raftGroupId {
> id: "\001\323\357*\221,O\300\200\266\001#C\327j\333"
> }
> success: true
> }
> term: 3
> nextIndex: 3049
> followerCommit: 3047
> java.lang.IllegalStateException: reply's next index is 3049, request's
> previous is term: 1
> index: 3047
> at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.onSuccess(GrpcLogAppender.java:285)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNextImpl(GrpcLogAppender.java:230)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:215)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:197)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment
> log-3048_3066 to index:3066
> 2019-02-15 10:15:56,442 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolled log segment from
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3048
> to
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_3048-3066
> 2019-02-15 10:15:56,564 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: created new log segment
> /data/disk1/ozone/meta/ratis/01d3ef2a-912c-4fc0-80b6-012343d76adb/current/log_inprogress_3067
> 2019-02-15 10:16:45,420 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 943007c8-4fdd-4926-89e2-2c8c52c05073-RaftLogWorker: Rolling segment
> log-3067_3077 to index:3077
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-710) Optimize the bits used to represent Blocks
[ https://issues.apache.org/jira/browse/HDDS-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-710: -- Summary: Optimize the bits used to represent Blocks (was: Make Block Commit Sequence (BCS) opaque to clients) > Optimize the bits used to represent Blocks > -- > > Key: HDDS-710 > URL: https://issues.apache.org/jira/browse/HDDS-710 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > > An immutable block is identified by the following: > - Container ID > - Local Block ID > - BCS (Block Commit Sequence ID) > All of these values are currently exposed to the client. Instead we can have > a composite block ID that hides these details from the client. A first > thought is a naive implementation that generates a 192-bit (3x64-bit) block > ID. > Proposed by [~anu] in HDDS-676. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1043: --- Target Version/s: 0.4.0 > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, > HDDS-1043.02.patch, HDDS-1043.03.patch, HDDS-1043.04.patch, > HDDS-1043.05.patch, HDDS-1043.06.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1112) Add a ozoneFilesystem related api's to OzoneManager to reduce redundant lookups
[ https://issues.apache.org/jira/browse/HDDS-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1112: --- Priority: Critical (was: Major) > Add a ozoneFilesystem related api's to OzoneManager to reduce redundant > lookups > --- > > Key: HDDS-1112 > URL: https://issues.apache.org/jira/browse/HDDS-1112 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Critical > Fix For: 0.4.0 > > > With the current OzoneFilesystem design, most of the lookups while create > happens via that getFileStatus api, which inturn does a getKey or a list Key > for the keys in the Ozone bucket. > In most of the cases, the files do not exists before creation, and hence > these lookups corresponds to wasted time in lookup. This jira proposes to > optimize the "create" and "getFileState" api in OzoneFileSystem by > introducing OzoneFilesystem friendly apis in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1185) Optimize GetFileStatus in OzoneFileSystem by reducing the number of rpc call to OM.
[ https://issues.apache.org/jira/browse/HDDS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1185: --- Priority: Critical (was: Major) > Optimize GetFileStatus in OzoneFileSystem by reducing the number of rpc call > to OM. > --- > > Key: HDDS-1185 > URL: https://issues.apache.org/jira/browse/HDDS-1185 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Filesystem >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Critical > Fix For: 0.4.0 > > Attachments: HDDS-1185.001.patch, HDDS-1185.002.patch > > > GetFileStatus sends multiple rpc calls to Ozone Manager to fetch the file > status for a given file. This can be optimized by performing all the > processing on the OzoneManager for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1098) Introduce Retry Policy in Ozone Client
[ https://issues.apache.org/jira/browse/HDDS-1098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1098: --- Priority: Critical (was: Major) > Introduce Retry Policy in Ozone Client > -- > > Key: HDDS-1098 > URL: https://issues.apache.org/jira/browse/HDDS-1098 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Critical > Fix For: 0.4.0 > > Attachments: HDDS-1098.000.patch > > > Currently , ozone client indefinitely retries to failover to a new block > write in case it encounters exceptions during write. The aim here is to bring > in retry policy in the client so that indefinite retries should go away. > Ozone client code will also be refactored as a part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1043: --- Priority: Blocker (was: Major) > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Blocker > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, > HDDS-1043.02.patch, HDDS-1043.03.patch, HDDS-1043.04.patch, > HDDS-1043.05.patch, HDDS-1043.06.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1185) Optimize GetFileStatus in OzoneFileSystem by reducing the number of rpc call to OM.
[
https://issues.apache.org/jira/browse/HDDS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781100#comment-16781100
]
Jitendra Nath Pandey commented on HDDS-1185:
A few comments:
# In OzoneFileStatus, blockSize is hardcoded.
# Set permissions to null, instead of hdfs defaults, as they are not supported
yet.
# Owner cannot be UserGroupInformation.getCurrentUser(). Most likely it needs
to be set explicitly. I would expect it to be part of OMKeyInfo, or should come
from OM metadata.
#
{code:java}
if (dirKeyInfo != null) {
return new OzoneFileStatus(dirKeyInfo, true);
}
List keys = metadataManager.listKeys(volumeName, bucketName,
null, dirKey, 1);
if (keys.iterator().hasNext()) {{code}
If the dirKeyInfo is null, can we still expect listKeys to return some keys?
The creation time of the first key returned in this list is used as the
creation-time of the directory. The keys returned may not be in the same order
as they were created.
> Optimize GetFileStatus in OzoneFileSystem by reducing the number of rpc call
> to OM.
> ---
>
> Key: HDDS-1185
> URL: https://issues.apache.org/jira/browse/HDDS-1185
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Filesystem
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1185.001.patch, HDDS-1185.002.patch
>
>
> GetFileStatus sends multiple rpc calls to Ozone Manager to fetch the file
> status for a given file. This can be optimized by performing all the
> processing on the OzoneManager for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780965#comment-16780965 ] Jitendra Nath Pandey commented on HDDS-1061: Updated the fixed version. It is important to update it, so that jiras fixed for a release can be queried. > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch, HDDS-1061.04.patch, > HDDS-1061.05.patch, HDDS-1061.06.patch, HDDS-1061.07.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1061: --- Fix Version/s: 0.4.0 > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch, HDDS-1061.04.patch, > HDDS-1061.05.patch, HDDS-1061.06.patch, HDDS-1061.07.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-451) PutKey failed due to error "Rejecting write chunk request. Chunk overwrite without explicit request"
[
https://issues.apache.org/jira/browse/HDDS-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779913#comment-16779913
]
Jitendra Nath Pandey commented on HDDS-451:
---
Yes, I think we should resolve it. We have made two fixes related to duplicate
transactions in HDDS-1026 and RATIS-489.
> PutKey failed due to error "Rejecting write chunk request. Chunk overwrite
> without explicit request"
>
>
> Key: HDDS-451
> URL: https://issues.apache.org/jira/browse/HDDS-451
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.2.1
>Reporter: Nilotpal Nandi
>Assignee: Shashikant Banerjee
>Priority: Blocker
> Labels: alpha2
> Attachments: all-node-ozone-logs-1536841590.tar.gz
>
>
> steps taken :
> --
> # Ran Put Key command to write 50GB data. Put Key client operation failed
> after 17 mins.
> error seen ozone.log :
>
>
> {code}
> 2018-09-13 12:11:53,734 [ForkJoinPool.commonPool-worker-20] DEBUG
> (ChunkManagerImpl.java:85) - writing
> chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1
> chunk stage:COMMIT_DATA chunk
> file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_1
> tmp chunk file
> 2018-09-13 12:11:56,576 [pool-3-thread-60] DEBUG (ChunkManagerImpl.java:85) -
> writing
> chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
> chunk stage:WRITE_DATA chunk
> file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
> tmp chunk file
> 2018-09-13 12:11:56,739 [ForkJoinPool.commonPool-worker-20] DEBUG
> (ChunkManagerImpl.java:85) - writing
> chunk:bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
> chunk stage:COMMIT_DATA chunk
> file:/tmp/hadoop-root/dfs/data/hdds/de0a9e01-4a12-40e3-b567-51b9bd83248e/current/containerDir0/16/chunks/bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2
> tmp chunk file
> 2018-09-13 12:12:21,410 [Datanode State Machine Thread - 0] DEBUG
> (DatanodeStateMachine.java:148) - Executing cycle Number : 206
> 2018-09-13 12:12:51,411 [Datanode State Machine Thread - 0] DEBUG
> (DatanodeStateMachine.java:148) - Executing cycle Number : 207
> 2018-09-13 12:12:53,525 [BlockDeletingService#1] DEBUG
> (TopNOrderedContainerDeletionChoosingPolicy.java:79) - Stop looking for next
> container, there is no pending deletion block contained in remaining
> containers.
> 2018-09-13 12:12:55,048 [Datanode ReportManager Thread - 1] DEBUG
> (ContainerSet.java:191) - Starting container report iteration.
> 2018-09-13 12:13:02,626 [pool-3-thread-1] ERROR (ChunkUtils.java:244) -
> Rejecting write chunk request. Chunk overwrite without explicit request.
> ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
> offset=0, len=16777216}
> 2018-09-13 12:13:03,035 [pool-3-thread-1] INFO (ContainerUtils.java:149) -
> Operation: WriteChunk : Trace ID: 54834b29-603d-4ba9-9d68-0885215759d8 :
> Message: Rejecting write chunk request. OverWrite flag
> required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
> offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
> 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] ERROR
> (ChunkUtils.java:244) - Rejecting write chunk request. Chunk overwrite
> without explicit request.
> ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
> offset=0, len=16777216}
> 2018-09-13 12:13:03,037 [ForkJoinPool.commonPool-worker-11] INFO
> (ContainerUtils.java:149) - Operation: WriteChunk : Trace ID:
> 54834b29-603d-4ba9-9d68-0885215759d8 : Message: Rejecting write chunk
> request. OverWrite flag
> required.ChunkInfo{chunkName='bd80b58a5eba888200a4832a0f2aafb3_stream_5f3b2505-6964-45c9-a7ad-827388a1e6a0_chunk_2,
> offset=0, len=16777216} : Result: OVERWRITE_FLAG_REQUIRED
>
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-787) UI page to show all the buckets for a user
[ https://issues.apache.org/jira/browse/HDDS-787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-787: - Assignee: Siddharth Wagle > UI page to show all the buckets for a user > -- > > Key: HDDS-787 > URL: https://issues.apache.org/jira/browse/HDDS-787 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Siddharth Wagle >Priority: Major > > This Jira is to develop a UI page to list all buckets for a user. > Currently, we have the code to list all buckets for a user. (But the user is > taken from authentication header), > # We should have a rest endpoint to take the username as path param, which > should return all the buckets for that. > # Develop a UI page, if browser=true, just show all the buckets and bucket > info on the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1177) Add validation to AuthorizationHeaderV4
[ https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1177: --- Fix Version/s: 0.4.0 > Add validation to AuthorizationHeaderV4 > --- > > Key: HDDS-1177 > URL: https://issues.apache.org/jira/browse/HDDS-1177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-1177.00.patch, HDDS-1177.01.patch, > HDDS-1177.02.patch > > > Add validation to AuthorizationHeaderV4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1184) Parallelization of write chunks in datanodes is broken
[ https://issues.apache.org/jira/browse/HDDS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1184: --- Priority: Blocker (was: Major) > Parallelization of write chunks in datanodes is broken > --- > > Key: HDDS-1184 > URL: https://issues.apache.org/jira/browse/HDDS-1184 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1184.000.patch > > > After the HDDS-4 branch merge, parallelization in write chunks and > applyTransaction is broken in ContainerStateMachine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1153) Make tracing instrumentation configurable
[ https://issues.apache.org/jira/browse/HDDS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-1153: -- Assignee: Kitti Nanasi > Make tracing instrumentation configurable > - > > Key: HDDS-1153 > URL: https://issues.apache.org/jira/browse/HDDS-1153 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Kitti Nanasi >Priority: Major > Labels: newbie > > TracingUtil.createProxy is a helper method to create a proxy instance with > tracing support. > The proxy instance implements the same interface as the original class and > delegates all the method calls to the original instance but it also sends > tracing information to the tracing server. > By default it's not a big overhead as the tracing libraries can be configured > to send tracing only with some low probability. > But to make it more safe we can make it optional. With a global > 'hdds.tracing.enabled' configuration variable (can be true by default) we can > adjust the behavior of TracingUtil.createProxy. > If the tracing is disabled the TracingUtil.createProxy should return with the > 'delegate' parameter instead of a proxy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1150) Enable end-to-end distributed tracing for Ozone
[ https://issues.apache.org/jira/browse/HDDS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-1150: -- Assignee: Elek, Marton > Enable end-to-end distributed tracing for Ozone > --- > > Key: HDDS-1150 > URL: https://issues.apache.org/jira/browse/HDDS-1150 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > See the description of HDDS-1017. That was the first step and it provides > some initial data. > But it turned out that we need multiple follow-up Jira to improve context > propagation. > I create this Jira to group/track all the remaining/work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1017) Use distributed tracing to indentify performance problems in Ozone
[ https://issues.apache.org/jira/browse/HDDS-1017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1017: --- Fix Version/s: 0.4.0 > Use distributed tracing to indentify performance problems in Ozone > -- > > Key: HDDS-1017 > URL: https://issues.apache.org/jira/browse/HDDS-1017 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Attachments: HDDS-1017.001.patch, HDDS-1017.002.patch, > HDDS-1017.003.patch, HDDS-1017.004.patch, HDDS-1017.005.patch, > HDDS-1017.006.patch, HDDS-1017.007.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In the recent months multiple performance issues are resolved in OM/SCM and > datanode sides. To identify the remaining problems a distributed tracing > framework would help a lot. > In HADOOP-15566 there is an ongoing discussion to remove the discontinued > HTrace and use something else instead. Until now without any conclusion, but > 1). There is one existing poc in the jira which uses opentracing > 2). It was suggested to "evaluating all the options" before a final decision > As an evaluation step we would like to investigate the performance of ozone > components with opentracing. This patch can help us to find the performance > problem but can be reverted when we will have a final solution in > HADOOP-15566 about the common tracing library. > To make it lightweight we can use the ozone message level tracing identifier > for context propagation instead of modifying the existing hadoop rpc > framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1129) Fix findbug/checkstyle errors in hadoop-hdds projects
[ https://issues.apache.org/jira/browse/HDDS-1129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773829#comment-16773829 ] Jitendra Nath Pandey commented on HDDS-1129: +1 for the patch. Please commit if jenkins results are satisfactory. (Posted the same comment in the pull request). > Fix findbug/checkstyle errors in hadoop-hdds projects > - > > Key: HDDS-1129 > URL: https://issues.apache.org/jira/browse/HDDS-1129 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HDDS-1114 fixed all the findbug/checkstyle problems but in the mean time new > patches are committed with newer error. > Here I would like to cleanup the projects again. > (Except the static field in RatisPipelineProvider which will be ignored in > this patch and tracked in HDDS-1128) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1012) Add Default CertificateClient implementation
[ https://issues.apache.org/jira/browse/HDDS-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-1012: --- Fix Version/s: 0.4.0 > Add Default CertificateClient implementation > > > Key: HDDS-1012 > URL: https://issues.apache.org/jira/browse/HDDS-1012 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-1012.01.patch, HDDS-1012.02.patch, > HDDS-1012.03.patch, HDDS-1012.04.patch, HDDS-1012.05.patch, > HDDS-1012.06.patch, HDDS-1012.07.patch, HDDS-1012.08.patch, HDDS-1012.09.patch > > > Add Default CertificateClient implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1116) Add java profiler servlet to the Ozone web servers
[ https://issues.apache.org/jira/browse/HDDS-1116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769683#comment-16769683 ] Jitendra Nath Pandey commented on HDDS-1116: Excellent! > Add java profiler servlet to the Ozone web servers > -- > > Key: HDDS-1116 > URL: https://issues.apache.org/jira/browse/HDDS-1116 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Thanks to [~gopalv] we learned that [~prasanth_j] implemented a helper > servlet in Hive to initialize new [async > profiler|https://github.com/jvm-profiling-tools/async-profiler] sessions and > provide the svg based flame graph over HTTP. (see HIVE-20202) > It seems to very useful as with this approach the profiling could be very > easy. > This patch imports the servlet from the Hive code base to the Ozone code base > with minor modification (to make it work with our servlet containers) > * The two servlets are unified to one > * Streaming the svg to the browser based on IOUtils.copy > * Output message is improved > By default the profile servlet is turned off, but you can enable it with > 'hdds.profiler.endpoint.enabled=true' ozone-site.xml settings. In that case > you can access the /prof endpoint from scm,om,s3g. > You should upload the async profiler first > (https://github.com/jvm-profiling-tools/async-profiler) and set the > ASYNC_PROFILER_HOME environment variable to find it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-936: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-936.00.patch, HDDS-936.01.patch, HDDS-936.02.patch, > HDDS-936.03.patch, HDDS-936.04.patch, HDDS-936.05.patch, HDDS-936.06.patch, > HDDS-936.07.patch, HDDS-936.08.patch, HDDS-936.09.patch > > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-936: -- Fix Version/s: 0.4.0 > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-936.00.patch, HDDS-936.01.patch, HDDS-936.02.patch, > HDDS-936.03.patch, HDDS-936.04.patch, HDDS-936.05.patch, HDDS-936.06.patch, > HDDS-936.07.patch, HDDS-936.08.patch, HDDS-936.09.patch > > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767705#comment-16767705 ] Jitendra Nath Pandey commented on HDDS-936: --- +1 for the latest patch. I have committed this to trunk. Thanks [~saruntek] for the patch. > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > Attachments: HDDS-936.00.patch, HDDS-936.01.patch, HDDS-936.02.patch, > HDDS-936.03.patch, HDDS-936.04.patch, HDDS-936.05.patch, HDDS-936.06.patch, > HDDS-936.07.patch, HDDS-936.08.patch, HDDS-936.09.patch > > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763965#comment-16763965 ] Jitendra Nath Pandey commented on HDDS-936: --- +1 for the patch. I will commit it, let's follow up with improvements in the next Jira HDDS-1005. > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > Attachments: HDDS-936.00.patch, HDDS-936.01.patch, HDDS-936.02.patch, > HDDS-936.03.patch, HDDS-936.04.patch, HDDS-936.05.patch, HDDS-936.06.patch > > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-69) Add checkBucketAccess to OzoneManger
[ https://issues.apache.org/jira/browse/HDDS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752593#comment-16752593 ] Jitendra Nath Pandey commented on HDDS-69: -- Is this still relevant? > Add checkBucketAccess to OzoneManger > > > Key: HDDS-69 > URL: https://issues.apache.org/jira/browse/HDDS-69 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Ozone Manager >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDFS-12147-HDFS-7240.000.patch, > HDFS-12147-HDFS-7240.001.patch > > > Checks if the caller has access to a given bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-996) Incorrect data length gets updated in OM by client in case it hits exception in multiple successive block writes
[ https://issues.apache.org/jira/browse/HDDS-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750806#comment-16750806 ] Jitendra Nath Pandey commented on HDDS-996: --- +1 > Incorrect data length gets updated in OM by client in case it hits exception > in multiple successive block writes > > > Key: HDDS-996 > URL: https://issues.apache.org/jira/browse/HDDS-996 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-996.000.patch > > > In the retry path, the data which needs to be written to the next block > should always be calculated from the data actually residing in the buffer > list rather than the length of the current stream entry allocated. This leads > to updating incorrect length of key updated in OM when multiple exceptions > occur while doing key writes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-982) Fix TestContainerDataYaml#testIncorrectContainerFile
[ https://issues.apache.org/jira/browse/HDDS-982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDDS-982: - Assignee: Doroszlai, Attila (was: Lokesh Jain) > Fix TestContainerDataYaml#testIncorrectContainerFile > > > Key: HDDS-982 > URL: https://issues.apache.org/jira/browse/HDDS-982 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Reporter: Lokesh Jain >Assignee: Doroszlai, Attila >Priority: Major > Attachments: HDDS-982.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-936: -- Status: Patch Available (was: Open) > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > Attachments: HDDS-936.00.patch, HDDS-936.01.patch, HDDS-936.02.patch > > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-936) Need a tool to map containers to ozone objects
[ https://issues.apache.org/jira/browse/HDDS-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724446#comment-16724446 ] Jitendra Nath Pandey commented on HDDS-936: --- The goal of this Jira is to have a basic tool for now. We could gradually build a service around it that can provide many data management services. This could need full scan processing of OM's metadata, which could run into terrabytes, hence the need to keep it separate from OM or SCM. > Need a tool to map containers to ozone objects > -- > > Key: HDDS-936 > URL: https://issues.apache.org/jira/browse/HDDS-936 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Manager >Reporter: Jitendra Nath Pandey >Assignee: sarun singla >Priority: Major > > Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-936) Need a tool to map containers to ozone objects
Jitendra Nath Pandey created HDDS-936: - Summary: Need a tool to map containers to ozone objects Key: HDDS-936 URL: https://issues.apache.org/jira/browse/HDDS-936 Project: Hadoop Distributed Data Store Issue Type: Bug Components: Ozone Manager Reporter: Jitendra Nath Pandey Assignee: sarun singla Ozone should have a tool to get list of objects that a container contains. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[ https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-870: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Avoid creating block sized buffer in ChunkGroupOutputStream > --- > > Key: HDDS-870 > URL: https://issues.apache.org/jira/browse/HDDS-870 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-870.000.patch, HDDS-870.001.patch, > HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch, > HDDS-870.005.patch, HDDS-870.006.patch, HDDS-870.007.patch, > HDDS-870.008.patch, HDDS-870.009.patch > > > Currently, for a key, we create a block size byteBuffer in order for caching > data. This can be replaced with an array of buffers of size flush buffer size > configured for handling 2 node failures as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[ https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713738#comment-16713738 ] Jitendra Nath Pandey commented on HDDS-870: --- I have committed this to trunk. Thanks [~shashikant] for the contribution, and [~msingh] for the reviews. bq. Filed RATIS-453 to fix the retry behavior in Ratis. Thanks [~szetszwo] for figuring the root cause of the test failure. > Avoid creating block sized buffer in ChunkGroupOutputStream > --- > > Key: HDDS-870 > URL: https://issues.apache.org/jira/browse/HDDS-870 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-870.000.patch, HDDS-870.001.patch, > HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch, > HDDS-870.005.patch, HDDS-870.006.patch, HDDS-870.007.patch, > HDDS-870.008.patch, HDDS-870.009.patch > > > Currently, for a key, we create a block size byteBuffer in order for caching > data. This can be replaced with an array of buffers of size flush buffer size > configured for handling 2 node failures as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713223#comment-16713223 ] Jitendra Nath Pandey commented on HDDS-909: --- I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-909) Default implementation for Ozone acls
[ https://issues.apache.org/jira/browse/HDDS-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713223#comment-16713223 ] Jitendra Nath Pandey edited comment on HDDS-909 at 12/7/18 7:41 PM: I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. The getPath should rather construct the path from these fields in OzoneObj. was (Author: jnp): I wonder, why we need path at all in OzoneObj. We can just track volume/buckets and keys. These fields need to exist only in OzoneObjInfo class. > Default implementation for Ozone acls > - > > Key: HDDS-909 > URL: https://issues.apache.org/jira/browse/HDDS-909 > Project: Hadoop Distributed Data Store > Issue Type: New Feature >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-909.00.patch > > > Default implementation for Ozone acls -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-858) Start a Standalone Ratis Server on OM
[ https://issues.apache.org/jira/browse/HDDS-858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-858: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Start a Standalone Ratis Server on OM > - > > Key: HDDS-858 > URL: https://issues.apache.org/jira/browse/HDDS-858 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-858.002.patch, HDDS-858.003.patch, > HDDS_858.001.patch > > > We propose implementing a standalone Ratis server on OM, as a start. Once the > Ratis server and state machine are integrated into OM, then the replicated > Ratis state machine can be implemented for OM. > This Jira aims to just start a Ratis server on OM start. The client-OM > communication and OM state would not be changed in this Jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-880) Create api for ACL handling in Ozone
[ https://issues.apache.org/jira/browse/HDDS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711937#comment-16711937 ] Jitendra Nath Pandey commented on HDDS-880: --- +1 for the latest patch. > Create api for ACL handling in Ozone > > > Key: HDDS-880 > URL: https://issues.apache.org/jira/browse/HDDS-880 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HDDS-880.00.patch, HDDS-880.01.patch, HDDS-880.02.patch, > HDDS-880.03.patch, HDDS-880.04.patch, HDDS-880.05.patch, HDDS-880.06.patch > > > Create api for ACL handling in Ozone, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[
https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711855#comment-16711855
]
Jitendra Nath Pandey commented on HDDS-870:
---
The test TestFailureHandlingByClient seems to be flaky.
{code:java}
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 137.352
s <<< FAILURE! - in
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
[ERROR]
testMultiBlockWritesWithIntermittentDnFailures(org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient)
Time elapsed: 83.03 s <<< ERROR!
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
Unable to find the chunk file. chunk info
ChunkInfo{chunkName='6fdfe7e7b129c48ecbf39efa5de52c8f_stream_318544a3-ec1d-4a4f-b280-7d2a53ca30bf_chunk_1,
offset=0, len=1048576}
at
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:495)
at
org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:221)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.readChunkFromContainer(ChunkInputStream.java:211)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.prepareRead(ChunkInputStream.java:175)
at
org.apache.hadoop.hdds.scm.storage.ChunkInputStream.read(ChunkInputStream.java:130)
at
org.apache.hadoop.ozone.client.io.ChunkGroupInputStream$ChunkInputStreamEntry.read(ChunkGroupInputStream.java:231)
at
org.apache.hadoop.ozone.client.io.ChunkGroupInputStream.read(ChunkGroupInputStream.java:125)
at
org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:49)
at java.io.InputStream.read(InputStream.java:101)
at
org.apache.hadoop.ozone.container.ContainerTestHelper.validateData(ContainerTestHelper.java:656)
at
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.validateData(TestFailureHandlingByClient.java:246)
at
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testMultiBlockWritesWithIntermittentDnFailures(TestFailureHandlingByClient.java:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]
TestFailureHandlingByClient.testMultiBlockWritesWithIntermittentDnFailures:234->validateData:246
» StorageContainer
{code}
> Avoid creating block sized buffer in ChunkGroupOutputStream
> ---
>
> Key: HDDS-870
> URL: https://issues.apache.org/jira/browse/HDDS-870
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
>
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[ https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711811#comment-16711811 ] Jitendra Nath Pandey commented on HDDS-870: --- +1, I will commit shortly. > Avoid creating block sized buffer in ChunkGroupOutputStream > --- > > Key: HDDS-870 > URL: https://issues.apache.org/jira/browse/HDDS-870 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-870.000.patch, HDDS-870.001.patch, > HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch, > HDDS-870.005.patch, HDDS-870.006.patch, HDDS-870.007.patch, > HDDS-870.008.patch, HDDS-870.009.patch > > > Currently, for a key, we create a block size byteBuffer in order for caching > data. This can be replaced with an array of buffers of size flush buffer size > configured for handling 2 node failures as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[
https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710699#comment-16710699
]
Jitendra Nath Pandey commented on HDDS-870:
---
There seem to be some inconsistencies between int and long. At some places we
use {{long}} to track size of buffered data, while some methods treat it as
{{int}}. For example {{computeBufferData}} returns an {{int}}.
> Avoid creating block sized buffer in ChunkGroupOutputStream
> ---
>
> Key: HDDS-870
> URL: https://issues.apache.org/jira/browse/HDDS-870
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-870.000.patch, HDDS-870.001.patch,
> HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch,
> HDDS-870.005.patch, HDDS-870.006.patch, HDDS-870.007.patch
>
>
> Currently, for a key, we create a block size byteBuffer in order for caching
> data. This can be replaced with an array of buffers of size flush buffer size
> configured for handling 2 node failures as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[ https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709081#comment-16709081 ] Jitendra Nath Pandey commented on HDDS-870: --- +1 > Avoid creating block sized buffer in ChunkGroupOutputStream > --- > > Key: HDDS-870 > URL: https://issues.apache.org/jira/browse/HDDS-870 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-870.000.patch, HDDS-870.001.patch, > HDDS-870.002.patch, HDDS-870.003.patch, HDDS-870.004.patch, HDDS-870.005.patch > > > Currently, for a key, we create a block size byteBuffer in order for caching > data. This can be replaced with an array of buffers of size flush buffer size > configured for handling 2 node failures as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-887) Add StatemachineContext info to Dispatcher from containerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705644#comment-16705644 ] Jitendra Nath Pandey commented on HDDS-887: --- +1 > Add StatemachineContext info to Dispatcher from containerStateMachine > - > > Key: HDDS-887 > URL: https://issues.apache.org/jira/browse/HDDS-887 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-887.000.patch, HDDS-887.001.patch, > HDDS-887.002.patch > > > As a part of transaction like writeChunk, readChunk, putBlock etc, there are > some specific info set which is required for executing the transactions on > the HddsDispatcher. Right now, all these protocol specfic info is added as > part of ContainerCommandRequestProto object which is visible to client. This > Jira aims to add the protocol specfic info in a context object and pass it to > dispatcher and remove the visibility from clinet by removing it out of > ContainerCommandRequestProto. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-882) Provide a config to optionally turn on/off the sync flag during chunk writes
[ https://issues.apache.org/jira/browse/HDDS-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705643#comment-16705643 ] Jitendra Nath Pandey commented on HDDS-882: --- +1, let's commit it. > Provide a config to optionally turn on/off the sync flag during chunk writes > > > Key: HDDS-882 > URL: https://issues.apache.org/jira/browse/HDDS-882 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 0.4.0 > > Attachments: HDDS-882.000.patch, HDDS-882.001.patch > > > Currently, chunk writes happen with sync flag on. We should add a config to > enable/disable this for performance benchmarks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-887) Add StatemachineContext info to Dispatcher from containerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705333#comment-16705333 ] Jitendra Nath Pandey commented on HDDS-887: --- +1 for the patch, if the test failures are not related. > Add StatemachineContext info to Dispatcher from containerStateMachine > - > > Key: HDDS-887 > URL: https://issues.apache.org/jira/browse/HDDS-887 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-887.000.patch, HDDS-887.001.patch > > > As a part of transaction like writeChunk, readChunk, putBlock etc, there are > some specific info set which is required for executing the transactions on > the HddsDispatcher. Right now, all these protocol specfic info is added as > part of ContainerCommandRequestProto object which is visible to client. This > Jira aims to add the protocol specfic info in a context object and pass it to > dispatcher and remove the visibility from clinet by removing it out of > ContainerCommandRequestProto. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-887) Add StatemachineContext info to Dispatcher from containerStateMachine
[ https://issues.apache.org/jira/browse/HDDS-887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705117#comment-16705117 ] Jitendra Nath Pandey commented on HDDS-887: --- Thanks for the patch [~shashikant]. I have only minor comments. # Let's call the new class DispatcherContext because ContainerStateMachineContext is too generic. # Let's add a Builder in DispatcherContext. # writeChunkStage is a type, so better to name it starting with capital case. > Add StatemachineContext info to Dispatcher from containerStateMachine > - > > Key: HDDS-887 > URL: https://issues.apache.org/jira/browse/HDDS-887 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-887.000.patch > > > As a part of transaction like writeChunk, readChunk, putBlock etc, there are > some specific info set which is required for executing the transactions on > the HddsDispatcher. Right now, all these protocol specfic info is added as > part of ContainerCommandRequestProto object which is visible to client. This > Jira aims to add the protocol specfic info in a context object and pass it to > dispatcher and remove the visibility from clinet by removing it out of > ContainerCommandRequestProto. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-882) Provide a config to optionally turn on/off the sync flag during chunk writes
[ https://issues.apache.org/jira/browse/HDDS-882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704279#comment-16704279 ] Jitendra Nath Pandey commented on HDDS-882: --- The patch looks good to me. +1 However, we didn't get clean jenkins run. > Provide a config to optionally turn on/off the sync flag during chunk writes > > > Key: HDDS-882 > URL: https://issues.apache.org/jira/browse/HDDS-882 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Fix For: 0.4.0 > > Attachments: HDDS-882.000.patch > > > Currently, chunk writes happen with sync flag on. We should add a config to > enable/disable this for performance benchmarks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14112) Avoid recursive call to external authorizer for getContentSummary.
[
https://issues.apache.org/jira/browse/HDFS-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703871#comment-16703871
]
Jitendra Nath Pandey commented on HDFS-14112:
-
+1 for the latest patch.
> Avoid recursive call to external authorizer for getContentSummary.
> --
>
> Key: HDFS-14112
> URL: https://issues.apache.org/jira/browse/HDFS-14112
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jitendra Nath Pandey
>Assignee: Tsz Wo Nicholas Sze
>Priority: Critical
> Attachments: h14112_20181128.patch, h14112_20181129.patch
>
>
> HDFS-12130 optimizes permission check, and invokes permission checker
> recursively for each component of the tree, which works well for FSPermission
> checker.
> But for certain external authorizers it may be more efficient to make one
> call with {{subaccess}}, because often they don't have to evaluate for each
> and every component of the path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-880) Create api for ACL handling in Ozone
[ https://issues.apache.org/jira/browse/HDDS-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703671#comment-16703671 ] Jitendra Nath Pandey commented on HDDS-880: --- Let's add default implementation for setAcl and removeAcl, in the interface itself. For now, we can throw unsupported exception. > Create api for ACL handling in Ozone > > > Key: HDDS-880 > URL: https://issues.apache.org/jira/browse/HDDS-880 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HDDS-880.00.patch, HDDS-880.01.patch > > > Create api for ACL handling in Ozone, -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[
https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703676#comment-16703676
]
Jitendra Nath Pandey commented on HDDS-870:
---
It seems error prone to create a bufferList in {{ChunkGroupOutputStream}} and
share it in various {{ChunkOutputStreams}} within. The two streams may start
working on same buffer?
>From the patch, it seems it would be easy to move bufferList creation to
>{{ChunkOutputStreams}}, and each gets its own bufferList. Is there a downside
>of this?
> Avoid creating block sized buffer in ChunkGroupOutputStream
> ---
>
> Key: HDDS-870
> URL: https://issues.apache.org/jira/browse/HDDS-870
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-870.000.patch, HDDS-870.001.patch,
> HDDS-870.002.patch
>
>
> Currently, for a key, we create a block size byteBuffer in order for caching
> data. This can be replaced with an array of buffers of size flush buffer size
> configured for handling 2 node failures as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14112) Avoid recursive call to external authorizer for getContentSummary.
[
https://issues.apache.org/jira/browse/HDFS-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702698#comment-16702698
]
Jitendra Nath Pandey commented on HDFS-14112:
-
+1, pending jenkins.
> Avoid recursive call to external authorizer for getContentSummary.
> --
>
> Key: HDFS-14112
> URL: https://issues.apache.org/jira/browse/HDFS-14112
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Jitendra Nath Pandey
>Assignee: Tsz Wo Nicholas Sze
>Priority: Critical
> Attachments: h14112_20181128.patch
>
>
> HDFS-12130 optimizes permission check, and invokes permission checker
> recursively for each component of the tree, which works well for FSPermission
> checker.
> But for certain external authorizers it may be more efficient to make one
> call with {{subaccess}}, because often they don't have to evaluate for each
> and every component of the path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14112) Avoid recursive call to external authorizer for getContentSummary.
Jitendra Nath Pandey created HDFS-14112:
---
Summary: Avoid recursive call to external authorizer for
getContentSummary.
Key: HDFS-14112
URL: https://issues.apache.org/jira/browse/HDFS-14112
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Jitendra Nath Pandey
Assignee: Tsz Wo Nicholas Sze
HDFS-12130 optimizes permission check, and invokes permission checker
recursively for each component of the tree, which works well for FSPermission
checker.
But for certain external authorizers it may be more efficient to make one call
with {{subaccess}}, because often they don't have to evaluate for each and
every component of the path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-876) add blockade tests for flaky network
[ https://issues.apache.org/jira/browse/HDDS-876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701437#comment-16701437 ] Jitendra Nath Pandey commented on HDDS-876: --- Its a great idea to test with flaky network. +1 for the patch. > add blockade tests for flaky network > > > Key: HDDS-876 > URL: https://issues.apache.org/jira/browse/HDDS-876 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-876.001.patch > > > Blockade is a container utility to simulate network and node failures and > network partitions. https://blockade.readthedocs.io/en/latest/guide.html. > This jira proposes to add a simple test to test freon with a flaky network. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-868) Handle quasi closed container replicas in SCM
[ https://issues.apache.org/jira/browse/HDDS-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699376#comment-16699376 ] Jitendra Nath Pandey commented on HDDS-868: --- +1 for the patch. > Handle quasi closed container replicas in SCM > - > > Key: HDDS-868 > URL: https://issues.apache.org/jira/browse/HDDS-868 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDDS-868.000.patch, HDDS-868.001.patch > > > In case of pipeline failure the containers will be quais closed by datanode. > SCM has to understand that the container replica is quasi closed and based on > the block commit sequence Id SCM should identify the latest replica and force > close them now. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-850) ReadStateMachineData hits OverlappingFileLockException in ContainerStateMachine
[
https://issues.apache.org/jira/browse/HDDS-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698101#comment-16698101
]
Jitendra Nath Pandey commented on HDDS-850:
---
I am +1 for the patch.
One comment: The dispatcher takes request proto as param, and is shared between
client requests and {{readStateMachineData}} coming from Ratis, and therefore
some of the parameters like {{readFromTmpFile}} are exposed in the client
protocol, which are strictly for internal use. We need to restructure the code
a bit, to address this.
I am ok to commit this patch and address the above in a follow up jira.
> ReadStateMachineData hits OverlappingFileLockException in
> ContainerStateMachine
> ---
>
> Key: HDDS-850
> URL: https://issues.apache.org/jira/browse/HDDS-850
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-850.000.patch, HDDS-850.001.patch,
> HDDS-850.002.patch, HDDS-850.003.patch
>
>
> {code:java}
> 2018-11-16 09:54:41,599 ERROR org.apache.ratis.server.impl.LogAppender:
> GrpcLogAppender(0813f1a9-61be-4cab-aa05-d5640f4a8341 ->
> c6ad906f-7e71-4bac-bde3-d22bc1aa8c7d) hit IOException while loading raft log
> org.apache.ratis.server.storage.RaftLogIOException:
> 0813f1a9-61be-4cab-aa05-d5640f4a8341: Failed readStateMachineData for (t:2,
> i:1), STATEMACHINELOGENTRY, client-7D19FB803B1E, cid=0
> at
> org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:370)
> at
> org.apache.ratis.server.impl.LogAppender$LogEntryBuffer.getAppendRequest(LogAppender.java:167)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:216)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
> at
> org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:100)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.OverlappingFileLockException
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at
> sun.nio.ch.AsynchronousFileChannelImpl.addToFileLockTable(AsynchronousFileChannelImpl.java:178)
> at
> sun.nio.ch.SimpleAsynchronousFileChannelImpl.implLock(SimpleAsynchronousFileChannelImpl.java:185)
> at
> sun.nio.ch.AsynchronousFileChannelImpl.lock(AsynchronousFileChannelImpl.java:118)
> at
> org.apache.hadoop.ozone.container.keyvalue.helpers.ChunkUtils.readData(ChunkUtils.java:178)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.ChunkManagerImpl.readChunk(ChunkManagerImpl.java:197)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleReadChunk(KeyValueHandler.java:542)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:174)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:178)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.dispatchCommand(ContainerStateMachine.java:290)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.readStateMachineData(ContainerStateMachine.java:404)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$readStateMachineData$6(ContainerStateMachine.java:462)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ... 1 more
> {code}
> This happens in the Ratis leader where the stateMachineData is not in the
> cached segments in Ratis while it gets a request for ReadStateMachineData
> while writeStateMachineData is not completed yet. The approach would be to
> cache the stateMachineData inside ContainerStateMachine and not cache it
> inside ratis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-868) Handle quasi closed container replicas in SCM
[
https://issues.apache.org/jira/browse/HDDS-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697687#comment-16697687
]
Jitendra Nath Pandey commented on HDDS-868:
---
Patch looks good overall. Minor comments:
# Is {{sequenceId}} same as BCS? It could be different for some replicas for
certain node failure scenarios.
# {{containerInfo.setUsedBytes}} and {{containerInfo.setNumberOfKeys}}: Do
they track maximum number of bytes and keys reported by any replica; different
replicas would report different counts.
> Handle quasi closed container replicas in SCM
> -
>
> Key: HDDS-868
> URL: https://issues.apache.org/jira/browse/HDDS-868
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-868.000.patch
>
>
> In case of pipeline failure the containers will be quais closed by datanode.
> SCM has to understand that the container replica is quasi closed and based on
> the block commit sequence Id SCM should identify the latest replica and force
> close them now.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[
https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697549#comment-16697549
]
Jitendra Nath Pandey edited comment on HDDS-870 at 11/24/18 7:54 AM:
-
I took a quick look at the patch. Instead of tracking {{lastFlushIndex}}, it
may be easier to track an {{AtomicReference}} to the entry in the buffer-list.
{{totalSuccessfulFlushedData}} might run into GBs (for larger block sizes) and
overflow an integer. Let's use {{long}} for this.
was (Author: jnp):
I took a quick look at the patch. Instead of tracking {{lastFlushIndex}}, it
may be easier to track an {{AtomicReference}} to the entry in the buffer-list.
> Avoid creating block sized buffer in ChunkGroupOutputStream
> ---
>
> Key: HDDS-870
> URL: https://issues.apache.org/jira/browse/HDDS-870
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-870.000.patch
>
>
> Currently, for a key, we create a block size byteBuffer in order for caching
> data. This can be replaced with an array of buffers of size flush buffer size
> configured for handling 2 node failures as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-870) Avoid creating block sized buffer in ChunkGroupOutputStream
[
https://issues.apache.org/jira/browse/HDDS-870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697549#comment-16697549
]
Jitendra Nath Pandey commented on HDDS-870:
---
I took a quick look at the patch. Instead of tracking {{lastFlushIndex}}, it
may be easier to track an {{AtomicReference}} to the entry in the buffer-list.
> Avoid creating block sized buffer in ChunkGroupOutputStream
> ---
>
> Key: HDDS-870
> URL: https://issues.apache.org/jira/browse/HDDS-870
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-870.000.patch
>
>
> Currently, for a key, we create a block size byteBuffer in order for caching
> data. This can be replaced with an array of buffers of size flush buffer size
> configured for handling 2 node failures as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-866) Handle RaftRetryFailureException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-866: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Handle RaftRetryFailureException in OzoneClient > --- > > Key: HDDS-866 > URL: https://issues.apache.org/jira/browse/HDDS-866 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-866.000.patch, HDDS-866.001.patch > > > With 2 Node failures or network partition among multiple servers in ratis, > RaftClient retries request and eventually fails with > raftRetryFailureException. This exception needs to be handled in OzoneClient > in order to handle 2 node failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-866) Handle RaftRetryFailureException in OzoneClient
[ https://issues.apache.org/jira/browse/HDDS-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695614#comment-16695614 ] Jitendra Nath Pandey commented on HDDS-866: --- +1 for the patch, pending jenkins. > Handle RaftRetryFailureException in OzoneClient > --- > > Key: HDDS-866 > URL: https://issues.apache.org/jira/browse/HDDS-866 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-866.000.patch, HDDS-866.001.patch > > > With 2 Node failures or network partition among multiple servers in ratis, > RaftClient retries request and eventually fails with > raftRetryFailureException. This exception needs to be handled in OzoneClient > in order to handle 2 node failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-866) Handle RaftRetryFailureException in OzoneClient
[
https://issues.apache.org/jira/browse/HDDS-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695599#comment-16695599
]
Jitendra Nath Pandey commented on HDDS-866:
---
# In {{validateContainerCommand}}, no need to check for null container again.
Also, we don't need to call {{getContainer(containerID)}} again in the next
line after checking for null container.
# {{XceiverServerRatis#isExist}} has unhandled checked exception. It might not
compile.
> Handle RaftRetryFailureException in OzoneClient
> ---
>
> Key: HDDS-866
> URL: https://issues.apache.org/jira/browse/HDDS-866
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-866.000.patch
>
>
> With 2 Node failures or network partition among multiple servers in ratis,
> RaftClient retries request and eventually fails with
> raftRetryFailureException. This exception needs to be handled in OzoneClient
> in order to handle 2 node failures.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-9) Add GRPC protocol interceptors for Ozone Block Token
[ https://issues.apache.org/jira/browse/HDDS-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692444#comment-16692444 ] Jitendra Nath Pandey commented on HDDS-9: - +1 for the latest patch. > Add GRPC protocol interceptors for Ozone Block Token > > > Key: HDDS-9 > URL: https://issues.apache.org/jira/browse/HDDS-9 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Security >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDDS-9-HDDS-4.001.patch, HDDS-9-HDDS-4.002.patch, > HDDS-9-HDDS-4.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-718) Introduce new SCM Commands to list and close Pipelines
[ https://issues.apache.org/jira/browse/HDDS-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-718: -- Fix Version/s: 0.4.0 > Introduce new SCM Commands to list and close Pipelines > -- > > Key: HDDS-718 > URL: https://issues.apache.org/jira/browse/HDDS-718 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Nanda kumar >Assignee: Lokesh Jain >Priority: Blocker > Fix For: 0.4.0 > > Attachments: HDDS-718.001.patch, HDDS-718.002.patch, > HDDS-718.003.patch > > > We need to have a tear-down pipeline command in SCM so that an administrator > can close/destroy a pipeline in the cluster. > HDDS-695 brings in the commands in branch ozone-0.3, this Jira is for porting > them to trunk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-9) Add GRPC protocol interceptors for Ozone Block Token
[ https://issues.apache.org/jira/browse/HDDS-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690819#comment-16690819 ] Jitendra Nath Pandey commented on HDDS-9: - The patch looks ok to me. Just one comment: Block token identifier should also contain the length and block commit sequence (BCS) of the block. The length at the datanode may be more than what is recorded at OM. > Add GRPC protocol interceptors for Ozone Block Token > > > Key: HDDS-9 > URL: https://issues.apache.org/jira/browse/HDDS-9 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Security >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDDS-9-HDDS-4.001.patch, HDDS-9-HDDS-4.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-837) Persist originNodeId as part of .container file in datanode
[
https://issues.apache.org/jira/browse/HDDS-837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690736#comment-16690736
]
Jitendra Nath Pandey commented on HDDS-837:
---
+1 for the patch.
> Persist originNodeId as part of .container file in datanode
> ---
>
> Key: HDDS-837
> URL: https://issues.apache.org/jira/browse/HDDS-837
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDDS-837.000.patch, HDDS-837.wip.patch
>
>
> To differentiate the replica of QUASI_CLOSED containers we need
> {{originNodeId}} field. With this field, we can uniquely identify a
> QUASI_CLOSED container replica. This will be needed when we want to CLOSE a
> QUASI_CLOSED container.
> This field will be set by the node where the container is created and stored
> as part of {{.container}} file and will be sent as part of ContainerReport to
> SCM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-845) Create a new raftClient instance for every watch request for Ratis
[ https://issues.apache.org/jira/browse/HDDS-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690361#comment-16690361 ] Jitendra Nath Pandey commented on HDDS-845: --- +1 > Create a new raftClient instance for every watch request for Ratis > -- > > Key: HDDS-845 > URL: https://issues.apache.org/jira/browse/HDDS-845 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-845.000.patch, HDDS-845.001.patch > > > Currently , watch request go throw sliding window in ratis and hence block as > well as get blocked for other requests submitted before . These are read only > requests and not necessarily require to go throw the sliding window, Until > this gets addressed in Ratis, its better and efficient to create a new raft > Client instance for watch request in XceiverClientRatis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-845) Create a new raftClient instance for every watch request for Ratis
[ https://issues.apache.org/jira/browse/HDDS-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-845: -- Fix Version/s: (was: 0.4.0) > Create a new raftClient instance for every watch request for Ratis > -- > > Key: HDDS-845 > URL: https://issues.apache.org/jira/browse/HDDS-845 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-845.000.patch > > > Currently , watch request go throw sliding window in ratis and hence block as > well as get blocked for other requests submitted before . These are read only > requests and not necessarily require to go throw the sliding window, Until > this gets addressed in Ratis, its better and efficient to create a new raft > Client instance for watch request in XceiverClientRatis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-845) Create a new raftClient instance for every watch request for Ratis
[ https://issues.apache.org/jira/browse/HDDS-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDDS-845: -- Target Version/s: 0.4.0 > Create a new raftClient instance for every watch request for Ratis > -- > > Key: HDDS-845 > URL: https://issues.apache.org/jira/browse/HDDS-845 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDDS-845.000.patch > > > Currently , watch request go throw sliding window in ratis and hence block as > well as get blocked for other requests submitted before . These are read only > requests and not necessarily require to go throw the sliding window, Until > this gets addressed in Ratis, its better and efficient to create a new raft > Client instance for watch request in XceiverClientRatis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-845) Create a new raftClient instance for every watch request for Ratis
[ https://issues.apache.org/jira/browse/HDDS-845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689763#comment-16689763 ] Jitendra Nath Pandey commented on HDDS-845: --- It is a Ratis limitation, but seems weird that for the retry after timeout, we need to create another client. Let's add a TODO, that once RATIS-345 is fixed, this creation of new client can be removed. You could add that while committing. +1 for the patch. > Create a new raftClient instance for every watch request for Ratis > -- > > Key: HDDS-845 > URL: https://issues.apache.org/jira/browse/HDDS-845 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Affects Versions: 0.4.0 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-845.000.patch > > > Currently , watch request go throw sliding window in ratis and hence block as > well as get blocked for other requests submitted before . These are read only > requests and not necessarily require to go throw the sliding window, Until > this gets addressed in Ratis, its better and efficient to create a new raft > Client instance for watch request in XceiverClientRatis. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-8) Add OzoneManager Delegation Token support
[ https://issues.apache.org/jira/browse/HDDS-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688442#comment-16688442 ] Jitendra Nath Pandey commented on HDDS-8: - I am +1 for the latest patch, pending jenkins. Thanks for addressing comments [~ajayydv]. > Add OzoneManager Delegation Token support > - > > Key: HDDS-8 > URL: https://issues.apache.org/jira/browse/HDDS-8 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: Security >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-8-HDDS-4.00.patch, HDDS-8-HDDS-4.01.patch, > HDDS-8-HDDS-4.02.patch, HDDS-8-HDDS-4.03.patch, HDDS-8-HDDS-4.04.patch, > HDDS-8-HDDS-4.05.patch, HDDS-8-HDDS-4.06.patch, HDDS-8-HDDS-4.07.patch, > HDDS-8-HDDS-4.08.patch, HDDS-8-HDDS-4.09.patch, HDDS-8-HDDS-4.10.patch, > HDDS-8-HDDS-4.11.patch, HDDS-8-HDDS-4.12.patch, HDDS-8-HDDS-4.13.patch, > HDDS-8-HDDS-4.14.patch, HDDS-8-HDDS-4.15.patch > > > Add delegation token functionality to Ozone layer. We will re-use hadoop rpc > layer TOKEN authentication. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-8) Add OzoneManager Delegation Token support
[
https://issues.apache.org/jira/browse/HDDS-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687189#comment-16687189
]
Jitendra Nath Pandey commented on HDDS-8:
-
{quote}OzoneDelegationTokenSelector already extends
AbstractDelegationTokenSelector in patch v13. {quote}
I meant we should use generic parameter {{OzoneTokenIdentifier}}.
> Add OzoneManager Delegation Token support
> -
>
> Key: HDDS-8
> URL: https://issues.apache.org/jira/browse/HDDS-8
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Security
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-8-HDDS-4.00.patch, HDDS-8-HDDS-4.01.patch,
> HDDS-8-HDDS-4.02.patch, HDDS-8-HDDS-4.03.patch, HDDS-8-HDDS-4.04.patch,
> HDDS-8-HDDS-4.05.patch, HDDS-8-HDDS-4.06.patch, HDDS-8-HDDS-4.07.patch,
> HDDS-8-HDDS-4.08.patch, HDDS-8-HDDS-4.09.patch, HDDS-8-HDDS-4.10.patch,
> HDDS-8-HDDS-4.11.patch, HDDS-8-HDDS-4.12.patch, HDDS-8-HDDS-4.13.patch
>
>
> Add delegation token functionality to Ozone layer. We will re-use hadoop rpc
> layer TOKEN authentication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-837) Persist originNodeId as part of .container file in datanode
[
https://issues.apache.org/jira/browse/HDDS-837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686848#comment-16686848
]
Jitendra Nath Pandey commented on HDDS-837:
---
I think we should also add pipelineId along with the originNodeId. Even if it
is not used actively, it will be useful for debugging or looking at the history
of a container. In case of multi-raft, with multiple pipelines being served
from the same DN, it will be helpful to be able to distinguish which container
belongs to which pipeline.
> Persist originNodeId as part of .container file in datanode
> ---
>
> Key: HDDS-837
> URL: https://issues.apache.org/jira/browse/HDDS-837
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> To differentiate the replica of QUASI_CLOSED containers we need
> {{originNodeId}} field. With this field, we can uniquely identify a
> QUASI_CLOSED container replica. This will be needed when we want to CLOSE a
> QUASI_CLOSED container.
> This field will be set by the node where the container is created and stored
> as part of {{.container}} file and will be sent as part of ContainerReport to
> SCM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-837) Persist originNodeId as part of .container file in datanode
[
https://issues.apache.org/jira/browse/HDDS-837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686848#comment-16686848
]
Jitendra Nath Pandey edited comment on HDDS-837 at 11/14/18 4:47 PM:
-
I think we should also add originPipelineId along with the originNodeId. Even
if it is not used actively, it will be useful for debugging or looking at the
history of a container. In case of multi-raft, with multiple pipelines being
served from the same DN, it will be helpful to be able to distinguish which
container belongs to which pipeline.
was (Author: jnp):
I think we should also add pipelineId along with the originNodeId. Even if it
is not used actively, it will be useful for debugging or looking at the history
of a container. In case of multi-raft, with multiple pipelines being served
from the same DN, it will be helpful to be able to distinguish which container
belongs to which pipeline.
> Persist originNodeId as part of .container file in datanode
> ---
>
> Key: HDDS-837
> URL: https://issues.apache.org/jira/browse/HDDS-837
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
>
> To differentiate the replica of QUASI_CLOSED containers we need
> {{originNodeId}} field. With this field, we can uniquely identify a
> QUASI_CLOSED container replica. This will be needed when we want to CLOSE a
> QUASI_CLOSED container.
> This field will be set by the node where the container is created and stored
> as part of {{.container}} file and will be sent as part of ContainerReport to
> SCM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
