[jira] [Created] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
Shashikant Banerjee created HDFS-12340: -- Summary: Ozone: C/C++ implementation of ozone client using curl Key: HDFS-12340 URL: https://issues.apache.org/jira/browse/HDFS-12340 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 This Jira is introduced for implementation of ozone client in C/C++ using curl library. All these calls will make use of HTTP protocol and would require libcurl. The libcurl API are referenced from here: https://curl.haxx.se/libcurl/ Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11612) Ozone: Cleanup Checkstyle issues
[ https://issues.apache.org/jira/browse/HDFS-11612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-11612: --- Attachment: HDFS-11612-HDFS-7240.001.patch Addressed most of the indentation, visibilityModifier , Javadoc Package ,naming convention related check style issues. [~anu], please have a look. > Ozone: Cleanup Checkstyle issues > > > Key: HDFS-11612 > URL: https://issues.apache.org/jira/browse/HDFS-11612 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: ozoneMerge > Attachments: HDFS-11612-HDFS-7240.001.patch > > > There is a bunch of check style issues under Ozone tree. We have to clean > them up before we call for a merge of this tree. This jira tracks that work > item. It would be a noisy but mostly content less change. Hence it is easier > to track that in separate patch -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11613) Ozone: Cleanup findbugs issues
[ https://issues.apache.org/jira/browse/HDFS-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee resolved HDFS-11613. Resolution: Not A Problem > Ozone: Cleanup findbugs issues > -- > > Key: HDFS-11613 > URL: https://issues.apache.org/jira/browse/HDFS-11613 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: ozoneMerge > > Some of the ozone checkins happened before Findbugs started running on test > files. This will cause issues when we attempt to merge with trunk. This jira > tracks cleaning up all Findbugs issue under ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11613) Ozone: Cleanup findbugs issues
[ https://issues.apache.org/jira/browse/HDFS-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172850#comment-16172850 ] Shashikant Banerjee commented on HDFS-11613: I took a look at this. ProtoBuf related findBugs issues are suppressed in the Pom settings. There are no other findBugs issues being found. > Ozone: Cleanup findbugs issues > -- > > Key: HDFS-11613 > URL: https://issues.apache.org/jira/browse/HDFS-11613 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: ozoneMerge > > Some of the ozone checkins happened before Findbugs started running on test > files. This will cause issues when we attempt to merge with trunk. This jira > tracks cleaning up all Findbugs issue under ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-11897: --- Attachment: HDFS-11897-HDFS-7240.003.patch Reverted some formatting changes for better readability. > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Attachments: HDFS-11897-HDFS-7240.001.patch, > HDFS-11897-HDFS-7240.002.patch, HDFS-11897-HDFS-7240.003.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-11897: --- Attachment: HDFS-11897-HDFS-7240.002.patch Thanks [~nandakumar131]] for the review comments. The patch attached addresses the review comments and the checkstyle issues. > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Attachments: HDFS-11897-HDFS-7240.001.patch, > HDFS-11897-HDFS-7240.002.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12469) Ozone: Create docker-compose definition to easily test real clusters
[ https://issues.apache.org/jira/browse/HDFS-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171792#comment-16171792 ] Shashikant Banerjee commented on HDFS-12469: I tried on Mac and i am seeing the following errors: Status: Downloaded newer image for flokkr/hadoop-runner:latest Creating ozone_datanode_1 ... Creating ozone_scm_1 ... Creating ozone_ksm_1 ... Creating ozone_namenode_1 ... Creating ozone_datanode_1 Creating ozone_scm_1 Creating ozone_namenode_1 Creating ozone_namenode_1 ... error ERROR: for ozone_namenode_1 Cannot start service namenode: driver failed programming external connectivity on endpoint ozone_namenode_1 (1a17659203f37fbccee4cfb79d97b651044083593a9b8930d572b8d690f8012e): Error starting userland proxy: BiCreating ozone_ksm_1 ... done ERROR: for namenode Cannot start service namenode: driver failed programming external connectivity on endpoint ozone_namenode_1 (1a17659203f37fbccee4cfb79d97b651044083593a9b8930d572b8d690f8012e): Error starting userland proxy: Bind for 0.0.0.0:9870 failed: port is already allocated ERROR: Encountered errors while bringing up the project > Ozone: Create docker-compose definition to easily test real clusters > > > Key: HDFS-12469 > URL: https://issues.apache.org/jira/browse/HDFS-12469 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: HDFS-7240 > > Attachments: HDFS-12469-HDFS-7240.WIP1.patch > > > The goal here is to create a docker-compose definition for ozone > pseudo-cluster with docker (one component per container). > Ideally after a full build the ozone cluster could be started easily with > after a simple docker-compose up command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12469) Ozone: Create docker-compose definition to easily test real clusters
[ https://issues.apache.org/jira/browse/HDFS-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171801#comment-16171801 ] Shashikant Banerjee commented on HDFS-12469: Just corrected the errors, it works . > Ozone: Create docker-compose definition to easily test real clusters > > > Key: HDFS-12469 > URL: https://issues.apache.org/jira/browse/HDFS-12469 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Elek, Marton >Assignee: Elek, Marton > Fix For: HDFS-7240 > > Attachments: HDFS-12469-HDFS-7240.WIP1.patch > > > The goal here is to create a docker-compose definition for ozone > pseudo-cluster with docker (one component per container). > Ideally after a full build the ozone cluster could be started easily with > after a simple docker-compose up command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-11897: --- Status: Patch Available (was: Open) > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Attachments: HDFS-11897-HDFS-7240.001.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11897) Ozone: KSM: Changing log level for client calls in KSM
[ https://issues.apache.org/jira/browse/HDFS-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-11897: --- Attachment: HDFS-11897-HDFS-7240.001.patch Changed most of the error logs to debug logs if the volume, bucket or key related info is not found in the metadata db.Changes are done in : 1)VolumeManagerImpl 2)BucketManagerImpl 3)KeyManagerImpl > Ozone: KSM: Changing log level for client calls in KSM > -- > > Key: HDFS-11897 > URL: https://issues.apache.org/jira/browse/HDFS-11897 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Nandakumar >Assignee: Shashikant Banerjee > Labels: ozoneMerge > Attachments: HDFS-11897-HDFS-7240.001.patch > > > Whenever there is no Volume/Bucker/Key found in MetadataDB for a client call, > KSM logs ERROR which is not necessary. The level of these log messages can be > changed to DEBUG, which will be helpful in debugging. > Changes are to be made in the following classes > * VolumeManagerImpl > * BucketManagerImpl > * KeyManagerImpl -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff . report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: SnapshotDiff_Improvemnets .pdf > SnapshotDiff - snapshotDiff fails if the snapshotDiff . report exceeds the > RPC response limit > - > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Summary: SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit (was: SnapshotDiff - snapshotDiff fails if the snapshotDiff . report exceeds the RPC response limit) > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12544) SnapshotDiff - support diff generation on any snapshot root descendant directory
[
https://issues.apache.org/jira/browse/HDFS-12544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDFS-12544:
--
Assignee: Manoj Govindassamy (was: Shashikant Banerjee)
> SnapshotDiff - support diff generation on any snapshot root descendant
> directory
>
>
> Key: HDFS-12544
> URL: https://issues.apache.org/jira/browse/HDFS-12544
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-12544.01.patch, HDFS-12544.02.patch
>
>
> {noformat}
> # hdfs snapshotDiff
>
> {noformat}
> Using snapshot diff command, we can generate a diff report between any two
> given snapshots under a snapshot root directory. The command today only
> accepts the path that is a snapshot root. There are many deployments where
> the snapshot root is configured at the higher level directory but the diff
> report needed is only for a specific directory under the snapshot root. In
> these cases, the diff report can be filtered for changes pertaining to the
> directory we are interested in. But when the snapshot root directory is very
> huge, the snapshot diff report generation can take minutes even if we are
> interested to know the changes only in a small directory. So, it would be
> highly performant if the diff report calculation can be limited to only the
> interesting sub-directory of the snapshot root instead of the whole snapshot
> root.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff . report exceeds the RPC response limit
Shashikant Banerjee created HDFS-12594: -- Summary: SnapshotDiff - snapshotDiff fails if the snapshotDiff . report exceeds the RPC response limit Key: HDFS-12594 URL: https://issues.apache.org/jira/browse/HDFS-12594 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee The snapshotDiff command fails if the snapshotDiff report size is larger than the configuration value of ipc.maximum.response.length which is by default 128 MB. Worst case, with all Renames ops in sanpshots each with source and target name equal to MAX_PATH_LEN which is 8k characters, this would result in at 8192 renames. SnapshotDiff is currently used by distcp to optimize copy operations and in case of the the diff report exceeding the limit , it fails with the below exception: Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport --- Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) Time elapsed: 111.906 sec <<< ERROR! java.io.IOException: Failed on local exception: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; Host Details : local host is: "hw15685.local/10.200.5.230"; destination host is: "localhost":59808; Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12544) SnapshotDiff - support diff generation on any snapshot root descendant directory
[
https://issues.apache.org/jira/browse/HDFS-12544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee reassigned HDFS-12544:
--
Assignee: Shashikant Banerjee (was: Manoj Govindassamy)
> SnapshotDiff - support diff generation on any snapshot root descendant
> directory
>
>
> Key: HDFS-12544
> URL: https://issues.apache.org/jira/browse/HDFS-12544
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Shashikant Banerjee
> Attachments: HDFS-12544.01.patch, HDFS-12544.02.patch
>
>
> {noformat}
> # hdfs snapshotDiff
>
> {noformat}
> Using snapshot diff command, we can generate a diff report between any two
> given snapshots under a snapshot root directory. The command today only
> accepts the path that is a snapshot root. There are many deployments where
> the snapshot root is configured at the higher level directory but the diff
> report needed is only for a specific directory under the snapshot root. In
> these cases, the diff report can be filtered for changes pertaining to the
> directory we are interested in. But when the snapshot root directory is very
> huge, the snapshot diff report generation can take minutes even if we are
> interested to know the changes only in a small directory. So, it would be
> highly performant if the diff report calculation can be limited to only the
> interesting sub-directory of the snapshot root instead of the whole snapshot
> root.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: (was: HDFS-12594.001.patch) > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Status: Open (was: Patch Available) Missed a few changes in this patch.. Cancelling it and will reupload the updated one. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.001.patch > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Status: Patch Available (was: Open) > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.001.patch The patch implements the design changes to compute snapshotDiffReport over multiple RPC calls in case the no of snapsReport entries exceed the preconfigured limit set by dfs.snapshotdiff-report.limit. The default is set to 1000 (same as that of dfs.ls.limit). There are still some issues existing related to javadoc which need to be worked out and i am working on the same. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: (was: SnapshotDiff_Improvemnets .pdf) > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Status: Patch Available (was: Open) > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: SnapshotDiff_Improvemnets .pdf Replacing the earlier uploaded doc with the updated doc. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.002.patch Addressed most of the checkstyle issues . > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12551) Ozone: Documentation: Add Ozone overview documentation.
[ https://issues.apache.org/jira/browse/HDFS-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181400#comment-16181400 ] Shashikant Banerjee commented on HDFS-12551: It gives a really good overview of Ozone. +1 for the change. > Ozone: Documentation: Add Ozone overview documentation. > --- > > Key: HDFS-12551 > URL: https://issues.apache.org/jira/browse/HDFS-12551 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Labels: ozoneDoc > Attachments: Apache Hadoop 3.1.0-SNAPSHOT –OzoneOverview.pdf, > HDFS-12551-HDFS-7240.001.patch > > > Add an architectural overview doc for ozone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11442) Ozone: Fix the Cluster ID generation code in SCM
[ https://issues.apache.org/jira/browse/HDFS-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee reassigned HDFS-11442: -- Assignee: Shashikant Banerjee (was: Anu Engineer) > Ozone: Fix the Cluster ID generation code in SCM > > > Key: HDFS-11442 > URL: https://issues.apache.org/jira/browse/HDFS-11442 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Shashikant Banerjee >Priority: Blocker > Labels: ozoneMerge > Fix For: HDFS-7240 > > > The Cluster ID is randomly generated right now when SCM is started and we > avoid verifying the clients cluster ID matches what SCM expects. This JIRA is > to track the comments in code. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: ozoneClient.C ozoneClient.h main.C The uploaded patch consists of 3 files: 1)ozoneClient.C - implements the following API's of ozoneClient library: a) CreateVolume b) CreateBucket c) PutKey d) GetKey 2) ozoneClient.h -- Contains the prototypes for the ozoneClient Library API's 3) main. C - Application using the API for the ozoneClient Library The implementation has been done using the libcurl API's . To Do: 1)There are issue with reusing the same curl easy handles in between calls to different API's of the ozoneClient from the application which need to be optimized.Currently , for every API the curl handle is cleaned up and recreated gain as otherwise, the subsequent operations using the same handle hang over HTTP. 2) Needs integeration with maven for building with Ozone source repository. 3) Needs further implementation of remaining API's for ozoneClient. > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: main.C, ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150374#comment-16150374 ] Shashikant Banerjee commented on HDFS-12340: Attached patch addresses: 1) Code Review Comments 2) Maven integration for ozoneclient library. > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, main.C, ozoneClient.C, > ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: HDFS-12340-HDFS-7240.002.patch > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, main.C, ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152736#comment-16152736 ] Shashikant Banerjee commented on HDFS-12340: Addressed : 1) Addressed some coding style issues 2) Fixed white space issues > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, main.C, ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: HDFS-12340-HDFS-7240.001.patch > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, main.C, ozoneClient.C, > ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12340 started by Shashikant Banerjee. -- > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, main.C, > ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12340 stopped by Shashikant Banerjee. -- > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, main.C, > ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: HDFS-12340-HDFS-7240.003.patch This patch addresses the following : 1) review comments given by Anu 2) The ozone Client library now uses a single connection for all the supported operations for the entire user session. To Do: 1) Right now while executing the ozone supported operations through the ozoneClient library, the errors coming as a part of the HTTP response, are directly written to the error stream .Need to implement an error mapping compatible with ozone instead of redirecting the response to error stream. > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, main.C, > ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12340 started by Shashikant Banerjee. -- > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, main.C, > ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Status: Patch Available (was: In Progress) > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, main.C, > ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194334#comment-16194334 ] Shashikant Banerjee commented on HDFS-12594: The solution being proposed here is an iterative approach where the client will send multiple RPCs to the namenode server where each rpc response will consist of 3 lists--containing modified entries , created entries and deleted entries.All of the diff processing will be moved over to the client(Rename , Create , Delete, Modify). This would result in lower compute overhead on namenode for snapshotdiff. For details, please refer to the attached document. I will upload patches through additional jiras. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.003.patch Addressed review comments given by [~szetszwo]. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.003.patch Thanks [~msingh] for the review comments. Patch v3 addresses the same. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12796) SCM should not start if Cluster Version file does not exist
[ https://issues.apache.org/jira/browse/HDFS-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12796: --- Attachment: HDFS-12796-HDFS-7240.003.patch Thanks [~nandakumar131] for the review comments. patch v3 addresses the same. > SCM should not start if Cluster Version file does not exist > --- > > Key: HDFS-12796 > URL: https://issues.apache.org/jira/browse/HDFS-12796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12796-HDFS-7240.001.patch, > HDFS-12796-HDFS-7240.002.patch, HDFS-12796-HDFS-7240.003.patch > > > We have SCM --init command which persist the cluster Version info in the > version file. If SCM gets > started without SCM --init being done even once, it should fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: (was: HDFS-12740-HDFS-7240.003.patch) > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.003.patch Attached correct v3 patch and removed the stale one. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266882#comment-16266882 ] Shashikant Banerjee edited comment on HDFS-12741 at 11/27/17 2:41 PM: -- Thanks [~linyiqun], for the review comments. Patch v2 addresses the same. Please have a look. KSM Version file is actually located inside the path "ozone_metadir_dir/ksm/" The function to read the ozone.metadata.dir config path is getScmMetaDir() which is misleading i guess. KSM doesn't store the version file inside SCM MetaData Directory. was (Author: shashikant): Thanks [~linyiqun], for the review comments. Patch v2 addresses the same. Please have a look. KSM Version file is actually located inside the path "ozone_metadir_dir/ksm/" The function to read the ozone.metadata.dir path is getScmMetaDir() which is misleading i guess. KSM doesn't store the version file inside SCm MetaData Directory. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.009.patch Thanks [~szetszwo], for the review comments. Patch v9 addresses the same. Please have a look. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch, > HDFS-12594.009.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.002.patch Thanks [~linyiqun], for the review comments. Patch v2 addresses the same. Please have a look. KSM Version file is actually located inside the path "ozone_metadir_dir/ksm/" The function to read the ozone.metadata.dir path is getScmMetaDir() which is misleading i guess. KSM doesn't store the version file inside SCm MetaData Directory. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12794: --- Attachment: HDFS-12794-HDFS-7240.005.patch Thanks [~anu], for the review comments. As per our discussion and review comments, i have updated the patch v5. Please have a look. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch, > HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch, > HDFS-12794-HDFS-7240.004.patch, HDFS-12794-HDFS-7240.005.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.001.patch Patch v1 implements the KSM -createObjectStore command. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Status: Patch Available (was: Open) > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.010.patch Thanks [~szetszwo], for the review comments. Patch v10 addresses the same. 1) All checkstyle issues except one which complains about no of parameters . 2)testDiffReportWithMillionFiles takes a really long time to execute and and that's why i add it here in the unit tests. Please have a look. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch, > HDFS-12594.009.patch, HDFS-12594.010.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272586#comment-16272586 ] Shashikant Banerjee edited comment on HDFS-12594 at 11/30/17 12:14 PM: --- Thanks [~szetszwo], for the review comments. Patch v10 addresses the same. 1) All checkstyle issues except one which complains about no of parameters . 2)testDiffReportWithMillionFiles takes a really long time to execute and and that's why i did not add it here in the unit tests. Please have a look. was (Author: shashikant): Thanks [~szetszwo], for the review comments. Patch v10 addresses the same. 1) All checkstyle issues except one which complains about no of parameters . 2)testDiffReportWithMillionFiles takes a really long time to execute and and that's why i add it here in the unit tests. Please have a look. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch, > HDFS-12594.009.patch, HDFS-12594.010.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12890) XceiverClient should have upper bound on async requests
Shashikant Banerjee created HDFS-12890: -- Summary: XceiverClient should have upper bound on async requests Key: HDFS-12890 URL: https://issues.apache.org/jira/browse/HDFS-12890 Project: Hadoop HDFS Issue Type: Sub-task Components: HDFS-7240 Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 XceiverClient-ratis maintains upper bound on the no of outstanding async requests . XceiverClient should also impose an upper bound on the no of outstanding async requests received from client for write. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.005.patch patch v5 : rebased patch v4 > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch, HDFS-12741-HDFS-7240.005.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.006.patch Thanks [~nandakumar131], for the review comments. Patch v6 addresses the review comments and checkstyle issues. Please have a look. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch, HDFS-12741-HDFS-7240.005.patch, > HDFS-12741-HDFS-7240.006.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12890) Ozone: XceiverClient should have upper bound on async requests
[ https://issues.apache.org/jira/browse/HDFS-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12890: --- Attachment: HDFS-12890-HDFS-7240.005.patch Thanks [~msingh], for the review comments. Patch v5 addresses the same. > Ozone: XceiverClient should have upper bound on async requests > -- > > Key: HDFS-12890 > URL: https://issues.apache.org/jira/browse/HDFS-12890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12890-HDFS-7240.001.patch, > HDFS-12890-HDFS-7240.002.patch, HDFS-12890-HDFS-7240.003.patch, > HDFS-12890-HDFS-7240.004.patch, HDFS-12890-HDFS-7240.005.patch > > > XceiverClient-ratis maintains upper bound on the no of outstanding async > requests . XceiverClient > should also impose an upper bound on the no of outstanding async requests > received from client > for write. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12890) Ozone: XceiverClient should have upper bound on async requests
[ https://issues.apache.org/jira/browse/HDFS-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16294826#comment-16294826 ] Shashikant Banerjee edited comment on HDFS-12890 at 12/18/17 11:21 AM: --- Thanks [~msingh], for the review comments. Patch v5 addresses the same. Please have a look. was (Author: shashikant): Thanks [~msingh], for the review comments. Patch v5 addresses the same. > Ozone: XceiverClient should have upper bound on async requests > -- > > Key: HDFS-12890 > URL: https://issues.apache.org/jira/browse/HDFS-12890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12890-HDFS-7240.001.patch, > HDFS-12890-HDFS-7240.002.patch, HDFS-12890-HDFS-7240.003.patch, > HDFS-12890-HDFS-7240.004.patch, HDFS-12890-HDFS-7240.005.patch > > > XceiverClient-ratis maintains upper bound on the no of outstanding async > requests . XceiverClient > should also impose an upper bound on the no of outstanding async requests > received from client > for write. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.009.patch Thanks [~nandakumar131], for the review comments. Patch v9 addresses the same. Please have a look. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch, HDFS-12741-HDFS-7240.005.patch, > HDFS-12741-HDFS-7240.006.patch, HDFS-12741-HDFS-7240.007.patch, > HDFS-12741-HDFS-7240.008.patch, HDFS-12741-HDFS-7240.009.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.008.patch Patch v8: Added the java doc comments. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch, HDFS-12741-HDFS-7240.005.patch, > HDFS-12741-HDFS-7240.006.patch, HDFS-12741-HDFS-7240.007.patch, > HDFS-12741-HDFS-7240.008.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.007.patch ksm -createObjectStore command line parsing seems to be broken with the generic parser code checked in. Fixed the command line option parsing for ksm as exists in SCM in patch v7. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch, HDFS-12741-HDFS-7240.005.patch, > HDFS-12741-HDFS-7240.006.patch, HDFS-12741-HDFS-7240.007.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[
https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255468#comment-16255468
]
Shashikant Banerjee edited comment on HDFS-12594 at 11/16/17 3:20 PM:
--
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
<{}(byte[])->byte[][]{null}.
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/").
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
was (Author: shashikant):
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
<{}(byte[])->byte[][]{null};
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/");
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC
> response limit
> ---
>
> Key: HDFS-12594
> URL: https://issues.apache.org/jira/browse/HDFS-12594
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch,
> HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch,
> HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch,
> SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than
> the configuration value of ipc.maximum.response.length which is by default
> 128 MB.
> Worst case, with all Renames ops in sanpshots each with source and target
> name equal to MAX_PATH_LEN which is 8k characters, this would result in at
> 8192 renames.
>
> SnapshotDiff is currently used by distcp to optimize copy operations and in
> case of the the diff report exceeding the limit , it fails with the below
> exception:
> Test set:
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
> Time elapsed: 111.906 sec <<< ERROR!
> java.io.IOException: Failed on local exception:
> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length;
> Host Details : local host is: "hw15685.local/10.200.5.230"; destination host
> is: "localhost":59808;
> Attached is the proposal for the changes required.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[
https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255468#comment-16255468
]
Shashikant Banerjee edited comment on HDFS-12594 at 11/16/17 3:16 PM:
--
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
<{}(byte[])->byte[][]{null};
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/");
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
was (Author: shashikant):
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
<{}(byte[])->byte[][]{null};
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/")
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC
> response limit
> ---
>
> Key: HDFS-12594
> URL: https://issues.apache.org/jira/browse/HDFS-12594
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch,
> HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch,
> HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch,
> SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than
> the configuration value of ipc.maximum.response.length which is by default
> 128 MB.
> Worst case, with all Renames ops in sanpshots each with source and target
> name equal to MAX_PATH_LEN which is 8k characters, this would result in at
> 8192 renames.
>
> SnapshotDiff is currently used by distcp to optimize copy operations and in
> case of the the diff report exceeding the limit , it fails with the below
> exception:
> Test set:
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
> Time elapsed: 111.906 sec <<< ERROR!
> java.io.IOException: Failed on local exception:
> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length;
> Host Details : local host is: "hw15685.local/10.200.5.230"; destination host
> is: "localhost":59808;
> Attached is the proposal for the changes required.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[
https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255468#comment-16255468
]
Shashikant Banerjee edited comment on HDFS-12594 at 11/16/17 3:14 PM:
--
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
<{}(byte[])->byte[][]{null};
Reverse Mapping:
byte[][]{null}->byte[]{(byte) ("/") }->String("/")
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
was (Author: shashikant):
Thanks [~szetszwo] , for the review comments.
patch v8 addresses the same.
>>DFSUtilClient.bytes2byteArray and DFSUtil.bytes2byteArray are very similar
>>but there is a small difference when len == 0:
DFSUtilClient returns new byte[0][] and
DFSUtil returns new byte[][]{null}.
Is it a bug?
< {}(byte[]) -> byte[][]{null};
Reverse Mapping:
byte[][]{null} -> byte[]{(byte) ("/") } ->String("/")
I have addressed the problems in conversion of byte[][] to byte[] . Please have
a look.
> SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC
> response limit
> ---
>
> Key: HDFS-12594
> URL: https://issues.apache.org/jira/browse/HDFS-12594
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch,
> HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch,
> HDFS-12594.006.patch, HDFS-12594.007.patch, HDFS-12594.008.patch,
> SnapshotDiff_Improvemnets .pdf
>
>
> The snapshotDiff command fails if the snapshotDiff report size is larger than
> the configuration value of ipc.maximum.response.length which is by default
> 128 MB.
> Worst case, with all Renames ops in sanpshots each with source and target
> name equal to MAX_PATH_LEN which is 8k characters, this would result in at
> 8192 renames.
>
> SnapshotDiff is currently used by distcp to optimize copy operations and in
> case of the the diff report exceeding the limit , it fails with the below
> exception:
> Test set:
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec
> <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
> testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
> Time elapsed: 111.906 sec <<< ERROR!
> java.io.IOException: Failed on local exception:
> org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length;
> Host Details : local host is: "hw15685.local/10.200.5.230"; destination host
> is: "localhost":59808;
> Attached is the proposal for the changes required.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.006.patch [~szetszwo], for the review comments. patch v6 addresses the review comments. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > HDFS-12594.006.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12794: --- Attachment: HDFS-12794-HDFS-7240.002.patch Thanks [~msingh] for the review comments. Patch v2 addresses the review comments.Please have a look. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch, > HDFS-12794-HDFS-7240.002.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12340) Ozone: C/C++ implementation of ozone client using curl
[ https://issues.apache.org/jira/browse/HDFS-12340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12340: --- Attachment: HDFS-12340-HDFS-7240.004.patch Thanks [~msingh] for the review comments. patch v4 addresses the review comments. Please have a look. > Ozone: C/C++ implementation of ozone client using curl > -- > > Key: HDFS-12340 > URL: https://issues.apache.org/jira/browse/HDFS-12340 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Labels: OzonePostMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12340-HDFS-7240.001.patch, > HDFS-12340-HDFS-7240.002.patch, HDFS-12340-HDFS-7240.003.patch, > HDFS-12340-HDFS-7240.004.patch, main.C, ozoneClient.C, ozoneClient.h > > > This Jira is introduced for implementation of ozone client in C/C++ using > curl library. > All these calls will make use of HTTP protocol and would require libcurl. The > libcurl API are referenced from here: > https://curl.haxx.se/libcurl/ > Additional details would be posted along with the patches. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.004.patch Rebased patch v3. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch, > HDFS-12740-HDFS-7240.004.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259728#comment-16259728
]
Shashikant Banerjee edited comment on HDFS-12794 at 11/20/17 7:43 PM:
--
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
response = response.thenApply(reply -> {
try {
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
} catch (IOException e) {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
was (Author: shashikant):
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
# While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given pint of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
sceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
try {
String requestID =
traceID + chunkIndex + ContainerProtos.Type.WriteChunk.name();
//add the chunk write traceId to the queue
semaphore.acquire();
LOG.warn("calling async");
response =
writeChunkAsync(xceiverClient, chunk, key, data, requestID);
response = response.thenApply(reply -> {
try {
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
} catch (IOException e) {
LOG.info("coming here to throw exception");
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
}
code {
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne
[jira] [Comment Edited] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259728#comment-16259728
]
Shashikant Banerjee edited comment on HDFS-12794 at 11/20/17 7:57 PM:
--
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
{code}
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
{code}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. {code}
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
{code}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
{code}
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
{code}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
{code}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
was (Author: shashikant):
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {}
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
code {}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {}
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
code {}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
code{}
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
code{}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
code{}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
> Ozone: Parallelize ChunkOutputSream
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee updated HDFS-12794:
---
Attachment: HDFS-12794-HDFS-7240.003.patch
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
# While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given pint of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
sceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
try {
String requestID =
traceID + chunkIndex + ContainerProtos.Type.WriteChunk.name();
//add the chunk write traceId to the queue
semaphore.acquire();
LOG.warn("calling async");
response =
writeChunkAsync(xceiverClient, chunk, key, data, requestID);
response = response.thenApply(reply -> {
try {
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
} catch (IOException e) {
LOG.info("coming here to throw exception");
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
}
code {
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
> Ozone: Parallelize ChunkOutputSream Writes to container
> ---
>
> Key: HDFS-12794
> URL: https://issues.apache.org/jira/browse/HDFS-12794
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
> Fix For: HDFS-7240
>
> Attachments: HDFS-12794-HDFS-7240.001.patch,
> HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch
>
>
> The chunkOutPutStream Write are sync in nature .Once one chunk of data gets
> written, the next chunk write is blocked until the previous chunk is written
> to the container.
> The ChunkOutputWrite Stream writes should be made async and Close on the
> OutputStream should ensure flushing of all dirty buffers to the container.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259728#comment-16259728
]
Shashikant Banerjee edited comment on HDFS-12794 at 11/20/17 7:45 PM:
--
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
was (Author: shashikant):
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
response = response.thenApply(reply -> {
try {
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
} catch (IOException e) {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
> Ozone: Parallelize ChunkOutputSream Writes to container
>
[jira] [Comment Edited] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259728#comment-16259728
]
Shashikant Banerjee edited comment on HDFS-12794 at 11/20/17 7:58 PM:
--
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
{code}
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
{code}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. {code}
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
{code}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
{code}
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
{code}
{code}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
{code}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
was (Author: shashikant):
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
{code}
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
{code}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. {code}
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
{code}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
{code}
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
{code}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
{code}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
> Ozone: Parallelize
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.005.patch Thanks [~nandakumar131] for the review comments.Patch v5 addresses the review comments.Please have a look. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch, > HDFS-12740-HDFS-7240.004.patch, HDFS-12740-HDFS-7240.005.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: (was: HDFS-12740-HDFS-7240.005.patch) > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch, > HDFS-12740-HDFS-7240.004.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.005.patch > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch, > HDFS-12740-HDFS-7240.004.patch, HDFS-12740-HDFS-7240.005.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[
https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259728#comment-16259728
]
Shashikant Banerjee edited comment on HDFS-12794 at 11/20/17 7:50 PM:
--
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {}
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
code {}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {}
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
code {}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
code{}
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
code{}
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
code{}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
was (Author: shashikant):
Thanks [~anu] , for the review comments.
As per discussion with [~anu], here are few conclusions:
1)
code {
//make sure all the data in the ChunkoutputStreams is written to the
// container
Preconditions.checkArgument(
semaphore.availablePermits() == getMaxOutstandingChunks());
}
While doing close on the groupOutputStream, we do chunkOutputstream.close,
where we do future.get() on response obtained after the write completes from
the xceiver server which makes sure the response is received from the xceiver
server. While closing the groupStream, semaphorePermiCount should be equal to
no of available permits which is equal to max no of outstanding chunks at any
given point of time.
2. code {
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e);
}
Hardcoding the exception when the writeChunkToConatiner calls completes in the
xceiverServer shows that , the exception is caught in the
chunkoutputGroupStream.close path which is expected.
Code {
response = response.thenApply(reply -> {
try{
throw new IOException("Exception while validating response");
// ContainerProtocolCalls.validateContainerResponse(reply);
// return reply;
}catch (IOException e){
throw new CompletionException(
"Unexpected Storage Container Exception: " + e.toString(), e)
java.io.IOException: Unexpected Storage Container Exception:
java.util.concurrent.ExecutionException: java.io.IOException: Exception while
validating response
at
org.apache.hadoop.scm.storage.ChunkOutputStream.close(ChunkOutputStream.java:174)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream$ChunkOutputStreamEntry.close(ChunkGroupOutputStream.java:468)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:291)
}
This is as expected. Idea was to write a mocktest while
validatingContainerResposne calls which is static method of a final class, and
this requires powerMockrunner which leads to issues while bringing up the
miniOzoneCluster.Will address the unit test to vertify the same later in a
different jira.
Patch v3 addresses the remaining review comments.
[~anu]/others, please have a look.
> Ozone: Parallelize ChunkOutputSream Writes to container
>
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: (was: HDFS-12741-HDFS-7240.001.patch) > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261633#comment-16261633 ] Shashikant Banerjee commented on HDFS-12741: Removing the patch as it does not apply . Will update the patch after rebasing. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261643#comment-16261643 ] Shashikant Banerjee commented on HDFS-12740: SCM --init is handled in HDFS-12739. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch, HDFS-12740-HDFS-7240.003.patch, > HDFS-12740-HDFS-7240.004.patch, HDFS-12740-HDFS-7240.005.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12794: --- Attachment: HDFS-12794-HDFS-7240.004.patch Thanks [~msingh], for the review comments. Please have a look. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch, > HDFS-12794-HDFS-7240.002.patch, HDFS-12794-HDFS-7240.003.patch, > HDFS-12794-HDFS-7240.004.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.007.patch patch v7 addresses the review comments as per discussion with [~szetszwo]. Please have a look. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > HDFS-12594.006.patch, HDFS-12594.007.patch, SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.007.patch The patch contains changes done by [~nandakumar131] and some code reorganization. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch, HDFS-12739-HDFS-7240.005.patch, > HDFS-12739-HDFS-7240.006.patch, HDFS-12739-HDFS-7240.007.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.005.patch Thanks [~nandakumar131] for the review comments. As per our discussion and review comments here, i have updated the patch. Please have a look. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch, HDFS-12739-HDFS-7240.005.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
Shashikant Banerjee created HDFS-12794: -- Summary: Ozone: Parallelize ChunkOutputSream Writes to container Key: HDFS-12794 URL: https://issues.apache.org/jira/browse/HDFS-12794 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 The chunkOutPutStream Write are sync in nature .Once one chunk of data gets written, the next chunk write is blocked until the previous chunk is written to the container. The ChunkOutputWrite Stream writes should be made async and Close on the OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.002.patch Thanks [~linyiqun]], [~nandakumar131], for the review comments. Patch v2 addresses the review comments. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch, > HDFS-12740-HDFS-7240.002.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12796) SCM should not start if Cluster Version file does not exist
[ https://issues.apache.org/jira/browse/HDFS-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12796: --- Attachment: HDFS-12796-HDFS-7240.001.patch Patch v1 implements the logic to fail SCM --start if the cluster version file does not exist in SCM. It also adds a test case to verify the same. [~nandakumar131], please have a look. > SCM should not start if Cluster Version file does not exist > --- > > Key: HDFS-12796 > URL: https://issues.apache.org/jira/browse/HDFS-12796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12796-HDFS-7240.001.patch > > > We have SCM --init command which persist the cluster Version info in the > version file. If SCM gets > started without SCM --init being done even once, it should fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12796) SCM should not start if Cluster Version file does not exist
Shashikant Banerjee created HDFS-12796: -- Summary: SCM should not start if Cluster Version file does not exist Key: HDFS-12796 URL: https://issues.apache.org/jira/browse/HDFS-12796 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Shashikant Banerjee Assignee: Shashikant Banerjee Fix For: HDFS-7240 We have SCM --init command which persist the cluster Version info in the version file. If SCM gets started without SCM --init being done even once, it should fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12794: --- Attachment: HDFS-12794-HDFS-7240.001.patch In patch v1, an arrayBlocking queue of size determined by ozone_max_chunk_data_limit/chunkSize is used which will basically determine the no of outstanding async chunkoutputstream write requests at any point of time. The entry put in the queue is the traceId of the chunk for which the async write is triggered. When the async write completes, its traceid is removed from the queue . Close on the stream ensures the response gets completed from XceiverClient and entry is removed from the queue. > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12796) SCM should not start if Cluster Version file does not exist
[ https://issues.apache.org/jira/browse/HDFS-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12796: --- Status: Patch Available (was: Open) > SCM should not start if Cluster Version file does not exist > --- > > Key: HDFS-12796 > URL: https://issues.apache.org/jira/browse/HDFS-12796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12796-HDFS-7240.001.patch, > HDFS-12796-HDFS-7240.002.patch > > > We have SCM --init command which persist the cluster Version info in the > version file. If SCM gets > started without SCM --init being done even once, it should fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12794) Ozone: Parallelize ChunkOutputSream Writes to container
[ https://issues.apache.org/jira/browse/HDFS-12794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12794: --- Status: Patch Available (was: Open) > Ozone: Parallelize ChunkOutputSream Writes to container > --- > > Key: HDFS-12794 > URL: https://issues.apache.org/jira/browse/HDFS-12794 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12794-HDFS-7240.001.patch > > > The chunkOutPutStream Write are sync in nature .Once one chunk of data gets > written, the next chunk write is blocked until the previous chunk is written > to the container. > The ChunkOutputWrite Stream writes should be made async and Close on the > OutputStream should ensure flushing of all dirty buffers to the container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12796) SCM should not start if Cluster Version file does not exist
[ https://issues.apache.org/jira/browse/HDFS-12796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12796: --- Attachment: HDFS-12796-HDFS-7240.002.patch Thanks [~nandakumar131], for the review comments. patch v2 addresses the same. Please have a look. > SCM should not start if Cluster Version file does not exist > --- > > Key: HDFS-12796 > URL: https://issues.apache.org/jira/browse/HDFS-12796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12796-HDFS-7240.001.patch, > HDFS-12796-HDFS-7240.002.patch > > > We have SCM --init command which persist the cluster Version info in the > version file. If SCM gets > started without SCM --init being done even once, it should fail. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.008.patch Thanks [~msingh] for the review comments. patch v8 addresses the review comments. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch, HDFS-12739-HDFS-7240.005.patch, > HDFS-12739-HDFS-7240.006.patch, HDFS-12739-HDFS-7240.007.patch, > HDFS-12739-HDFS-7240.008.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.009.patch Thanks [~msingh] , for the review comments. Patch v9 addresses the same. Please have a look. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch, HDFS-12739-HDFS-7240.005.patch, > HDFS-12739-HDFS-7240.006.patch, HDFS-12739-HDFS-7240.007.patch, > HDFS-12739-HDFS-7240.008.patch, HDFS-12739-HDFS-7240.009.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12594) SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC response limit
[ https://issues.apache.org/jira/browse/HDFS-12594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12594: --- Attachment: HDFS-12594.005.patch Thanks [~szetszwo], for the review. Patch v5 addresses the review comments. > SnapshotDiff - snapshotDiff fails if the snapshotDiff report exceeds the RPC > response limit > --- > > Key: HDFS-12594 > URL: https://issues.apache.org/jira/browse/HDFS-12594 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12594.001.patch, HDFS-12594.002.patch, > HDFS-12594.003.patch, HDFS-12594.004.patch, HDFS-12594.005.patch, > SnapshotDiff_Improvemnets .pdf > > > The snapshotDiff command fails if the snapshotDiff report size is larger than > the configuration value of ipc.maximum.response.length which is by default > 128 MB. > Worst case, with all Renames ops in sanpshots each with source and target > name equal to MAX_PATH_LEN which is 8k characters, this would result in at > 8192 renames. > > SnapshotDiff is currently used by distcp to optimize copy operations and in > case of the the diff report exceeding the limit , it fails with the below > exception: > Test set: > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > --- > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport > testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport) > Time elapsed: 111.906 sec <<< ERROR! > java.io.IOException: Failed on local exception: > org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length; > Host Details : local host is: "hw15685.local/10.200.5.230"; destination host > is: "localhost":59808; > Attached is the proposal for the changes required. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: (was: HDFS-12740-HDFS-7240.001.patch) > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Attachment: HDFS-12740-HDFS-7240.001.patch Removed the older patch. Patch v1 implements the changes as per HDFS-12739. [~nandakumar131], please have a look. > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12740) SCM should support a RPC to share the cluster Id with KSM and DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12740: --- Status: Patch Available (was: Open) > SCM should support a RPC to share the cluster Id with KSM and DataNodes > --- > > Key: HDFS-12740 > URL: https://issues.apache.org/jira/browse/HDFS-12740 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12740-HDFS-7240.001.patch > > > When the ozone cluster is first Created, SCM --init command will generate > cluster Id as well as SCM Id and persist it locally. The same cluster Id and > the SCM id will be shared with KSM during the KSM initialization and > Datanodes during datanode registration. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.004.patch [~linyiqun], Thanks for the review comments. The patch addresses Review comments . Please have a look. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12739) Add Support for SCM --init command
[ https://issues.apache.org/jira/browse/HDFS-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12739: --- Attachment: HDFS-12739-HDFS-7240.006.patch Addressed checkStyle issues. > Add Support for SCM --init command > -- > > Key: HDFS-12739 > URL: https://issues.apache.org/jira/browse/HDFS-12739 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: HDFS-7240 >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-12739-HDFS-7240.001.patch, > HDFS-12739-HDFS-7240.002.patch, HDFS-12739-HDFS-7240.003.patch, > HDFS-12739-HDFS-7240.004.patch, HDFS-12739-HDFS-7240.005.patch, > HDFS-12739-HDFS-7240.006.patch > > > SCM --init command will generate cluster ID and persist it locally. The same > cluster Id will be shared with KSM and the datanodes. IF the cluster Id is > already available in the locally available version file, it will just read > the cluster Id . -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: HDFS-12741-HDFS-7240.004.patch > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12741: --- Attachment: (was: HDFS-12741-HDFS-7240.004.patch) > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12741) ADD support for KSM --createObjectStore command
[ https://issues.apache.org/jira/browse/HDFS-12741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279085#comment-16279085 ] Shashikant Banerjee edited comment on HDFS-12741 at 12/5/17 7:25 PM: - Thanks [~nandakumar131], for the review comments. Patch v4 addresses the same. Please have a look. was (Author: shashikant): Thanks [~nandakumar131], for the review comments. Patch v4 addresses the same. > ADD support for KSM --createObjectStore command > --- > > Key: HDFS-12741 > URL: https://issues.apache.org/jira/browse/HDFS-12741 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Fix For: HDFS-7240 > > Attachments: HDFS-12741-HDFS-7240.001.patch, > HDFS-12741-HDFS-7240.002.patch, HDFS-12741-HDFS-7240.003.patch, > HDFS-12741-HDFS-7240.004.patch > > > KSM --createObjectStore command reads the ozone configuration information and > creates the KSM version file and reads the SCM version file from the SCM > specified. > > The SCM version file is stored in the KSM metadata directory and before > communicating with an SCM KSM verifies that it is communicating with an SCM > where the relationship has been established via createObjectStore command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
