[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178198#comment-16178198 ] Surendra Singh Lilhore commented on HDFS-12291: --- Thanks [~xiaochen] and [~umamaheswararao] for review.. Attached v5 patch, fixed all above comments > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184749#comment-16184749 ] Surendra Singh Lilhore commented on HDFS-12291: --- --- T E S T S --- Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.262 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 167.82 sec Running org.apache.hadoop.hdfs.TestReconstructStripedFile Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.365 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 100.842 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.732 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.439 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.883 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 97.406 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 120.407 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 191.447 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.02 sec Running org.apache.hadoop.hdfs.TestReadStripedFileWithMissingBlocks Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 121.655 sec Running org.apache.hadoop.hdfs.TestEncryptedTransfer Tests run: 28, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 103.219 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 101.036 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 95.42 sec Running org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.167 sec Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.564 sec Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 98.35 sec Running org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.753 sec > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch, > HDFS-12291-HDFS-10285-08.patch, HDFS-12291-HDFS-10285-09.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[
https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12291:
--
Attachment: HDFS-12291-HDFS-10285-09.patch
Attached v9. Fixed tab warnings..
All the failed test are not related to this patch. Mostly all are passing
locally..
Some time {{TestPersistentStoragePolicySatisfier}} failing. I have raised one
jira to handle this HDFS-12556
> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy
> of all the files under the given dir
> -
>
> Key: HDFS-12291
> URL: https://issues.apache.org/jira/browse/HDFS-12291
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12291-HDFS-10285-01.patch,
> HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch,
> HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch,
> HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch,
> HDFS-12291-HDFS-10285-08.patch, HDFS-12291-HDFS-10285-09.patch
>
>
> For the given source path directory, presently SPS consider only the files
> immediately under the directory(only one level of scanning) for satisfying
> the policy. It WON’T do recursive directory scanning and then schedules SPS
> tasks to satisfy the storage policy of all the files till the leaf node.
> The idea of this jira is to discuss & implement an efficient recursive
> directory iteration mechanism and satisfies storage policy for all the files
> under the given directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Summary: [SPS] : Block movement analysis should be done on read lock.
(was: [SPS] : Fix TestPersistentStoragePolicySatisfier#testWithRestarts )
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Attachment: HDFS-12556-HDFS-10285-01.patch
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Summary: [SPS] : Block movement analysis should be done in read lock.
(was: [SPS] : Block movement analysis should be done on read lock.)
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done on read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Status: Patch Available (was: Open)
> [SPS] : Block movement analysis should be done on read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187347#comment-16187347
]
Surendra Singh Lilhore commented on HDFS-12556:
---
Some time {{TestPersistentStoragePolicySatisfier#testWithRestarts}} failing
because of {{ArrayIndexOutOfBoundsException}}. This is happening because when
block movement analysis is going on, same time NN receive block report from one
DN and it will update the block storage list.
{{BlockManager#getStorages(BlockInfo)}} API create internal array {{storages}}
based on current number of node for a block but later number of node is
increased while filling the array and it cause
{{ArrayIndexOutOfBoundsException}}.
{code}
public DatanodeStorageInfo[] getStorages(BlockInfo block) {
final DatanodeStorageInfo[] storages = new
DatanodeStorageInfo[block.numNodes()];
int i = 0;
for(DatanodeStorageInfo s : blocksMap.getStorages(block)) {
storages[i++] = s;
}
return storages;
}
{code}
Block analysis should be done in read lock, so we can avoid in-between block
location storage update.
Attached initial patch. Please review...
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Attachment: HDFS-12291-HDFS-10285-08.patch > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch, > HDFS-12291-HDFS-10285-08.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[
https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182368#comment-16182368
]
Surendra Singh Lilhore commented on HDFS-12291:
---
Thanks [~umamaheswararao] for review...
Attached updated patch v8. Fixed one test case {{TestHdfsConfigFields}}.
Other failed tests are not related to this patch...
> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy
> of all the files under the given dir
> -
>
> Key: HDFS-12291
> URL: https://issues.apache.org/jira/browse/HDFS-12291
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12291-HDFS-10285-01.patch,
> HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch,
> HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch,
> HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch,
> HDFS-12291-HDFS-10285-08.patch
>
>
> For the given source path directory, presently SPS consider only the files
> immediately under the directory(only one level of scanning) for satisfying
> the policy. It WON’T do recursive directory scanning and then schedules SPS
> tasks to satisfy the storage policy of all the files till the leaf node.
> The idea of this jira is to discuss & implement an efficient recursive
> directory iteration mechanism and satisfies storage policy for all the files
> under the given directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12556) [SPS] : Fix TestPersistentStoragePolicySatisfier#testWithRestarts
Surendra Singh Lilhore created HDFS-12556:
-
Summary: [SPS] : Fix
TestPersistentStoragePolicySatisfier#testWithRestarts
Key: HDFS-12556
URL: https://issues.apache.org/jira/browse/HDFS-12556
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Surendra Singh Lilhore
{noformat}
2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
namenode.StoragePolicySatisfier
(StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
thread received runtime exception. Stopping Storage policy satisfier work
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
at
org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
at
org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12556) [SPS] : Fix TestPersistentStoragePolicySatisfier#testWithRestarts
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore reassigned HDFS-12556:
-
Assignee: Surendra Singh Lilhore
> [SPS] : Fix TestPersistentStoragePolicySatisfier#testWithRestarts
> --
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[
https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189504#comment-16189504
]
Surendra Singh Lilhore commented on HDFS-11968:
---
+1 LGTM.
> ViewFS: StoragePolicies commands fail with HDFS federation
> --
>
> Key: HDFS-11968
> URL: https://issues.apache.org/jira/browse/HDFS-11968
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.1
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch,
> HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch,
> HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch,
> HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch
>
>
> hdfs storagepolicies command fails with HDFS federation.
> For storage policies commands, a given user path should be resolved to a HDFS
> path and
> storage policy command should be applied onto the resolved HDFS path.
> {code}
> static DistributedFileSystem getDFS(Configuration conf)
> throws IOException {
> FileSystem fs = FileSystem.get(conf);
> if (!(fs instanceof DistributedFileSystem)) {
> throw new IllegalArgumentException("FileSystem " + fs.getUri() +
> " is not an HDFS file system");
> }
> return (DistributedFileSystem)fs;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179540#comment-16179540 ] Surendra Singh Lilhore commented on HDFS-12291: --- Thanks [~xiaochen], Attached v7 patch. Fixed failed test case.. > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Attachment: HDFS-12291-HDFS-10285-07.patch > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[
https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178872#comment-16178872
]
Surendra Singh Lilhore commented on HDFS-12291:
---
Hi [~xiaochen],
I have one doubt. In my current patch {{TestReencryption}} is failing and this
is because I am submitting the current batch after releasing read locks.
{code}
+final String parentPath = parent.getFullPathName();
+lockReleased = true;
+readUnlock();
+submitCurrentBatch(startId);
+try {
+ throttle();
{code}
In {{ReencryptionHandler}} before submitting the current batch checking the
read lock. My doubt is why we need lock to submit current batch ?
{code}
@Override
protected void submitCurrentBatch(final long zoneId) throws IOException,
InterruptedException {
assert dir.hasReadLock();
if (currentBatch.isEmpty()) {
return;
}
{code}
> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy
> of all the files under the given dir
> -
>
> Key: HDFS-12291
> URL: https://issues.apache.org/jira/browse/HDFS-12291
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12291-HDFS-10285-01.patch,
> HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch,
> HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch,
> HDFS-12291-HDFS-10285-06.patch
>
>
> For the given source path directory, presently SPS consider only the files
> immediately under the directory(only one level of scanning) for satisfying
> the policy. It WON’T do recursive directory scanning and then schedules SPS
> tasks to satisfy the storage policy of all the files till the leaf node.
> The idea of this jira is to discuss & implement an efficient recursive
> directory iteration mechanism and satisfies storage policy for all the files
> under the given directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Attachment: HDFS-12291-HDFS-10285-06.patch Missed new added file in last patch. Attached updated patch V6.. > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183684#comment-16183684 ] Surendra Singh Lilhore commented on HDFS-12291: --- bq. Not sure if we worry about create/delete inside the base dir during traversal here. This was important for re-encryption, where a bunch of race tests are added in TestReencryption. For SPS, newly created files anyway will use current policy, so they would be automatically satisfied. For Deleted, anyway we need not care about such files. Later while processing deleted inode, it will simply ignore in SPS daemon. bq. Heads up: there's a recent bug/improvement HDFS-12518 for HDFS-10899, will likely create some conflicts here. Issue itself is mostly for re-encryption though. We will take care this while doing rebase... > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch, HDFS-12291-HDFS-10285-03.patch, > HDFS-12291-HDFS-10285-04.patch, HDFS-12291-HDFS-10285-05.patch, > HDFS-12291-HDFS-10285-06.patch, HDFS-12291-HDFS-10285-07.patch, > HDFS-12291-HDFS-10285-08.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136290#comment-16136290
]
Surendra Singh Lilhore edited comment on HDFS-12225 at 8/22/17 5:09 AM:
Thanks [~umamaheswararao] for review..
bq. Why is this required? We will remove Xattr only when queue really becomes
empty right?
This is for standby namenode. SN will load inode's into
{{pendingSPSxAttrInode}} from the edits logs based in SPS xAttr. When SPS work
is done for that INode, again one remove xAttr edit log will come and in that
flow we will remove the inode from the {{pendingSPSxAttrInode}} queue in SN.
was (Author: surendrasingh):
bq. Why is this required? We will remove Xattr only when queue really becomes
empty right?
This is for standby namenode. SN will load inode's into
{{pendingSPSxAttrInode}} from the edits logs based in SPS xAttr. When SPS work
is done for that INode, again one remove xAttr edit log will come and in that
flow we will remove the inode from the {{pendingSPSxAttrInode}} queue.
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136290#comment-16136290
]
Surendra Singh Lilhore commented on HDFS-12225:
---
bq. Why is this required? We will remove Xattr only when queue really becomes
empty right?
This is for standby namenode. SN will load inode's into
{{pendingSPSxAttrInode}} from the edits logs based in SPS xAttr. When SPS work
is done for that INode, again one remove xAttr edit log will come and in that
flow we will remove the inode from the {{pendingSPSxAttrInode}} queue.
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134966#comment-16134966
]
Surendra Singh Lilhore commented on HDFS-12225:
---
{noformat}
hadoop.hdfs.server.namenode.TestStoragePolicySatisfier
hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier
{noformat}
Both the failed tests are passing locally
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-07.patch
Thanks [~rakeshr] for review..
Attached updated patch..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-06.patch
v6:
Fixed checkstyle warnings and some review comment given by Rakesh R.
Please review.
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136641#comment-16136641
]
Surendra Singh Lilhore commented on HDFS-12225:
---
Attached updated patch. Please review..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-08.patch
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139543#comment-16139543
]
Surendra Singh Lilhore commented on HDFS-12225:
---
Thanks [~umamaheswararao] for review and commit.
Thanks [~rakeshr] for review..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch, HDFS-12225-HDFS-10285-05.patch,
> HDFS-12225-HDFS-10285-06.patch, HDFS-12225-HDFS-10285-07.patch,
> HDFS-12225-HDFS-10285-08.patch, HDFS-12225-HDFS-10285-09.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Status: Patch Available (was: Open) > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Attachment: HDFS-12291-HDFS-10285-01.patch Attached first initial patch. It will do the following things. 1. Traverse directory recursively and process all the child files to satisfy the policy. 2. Queued up 1000 item at a time for satisfy the policy. Please review... > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134332#comment-16134332
]
Surendra Singh Lilhore commented on HDFS-12225:
---
bq. Also, need to call SPS#clearQueues() on SPS#disable() function ?
yes
Please review..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12225:
--
Attachment: HDFS-12225-HDFS-10285-04.patch
Thanks [~rakeshr] for review..
Attached updated patch..
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch, HDFS-12225-HDFS-10285-03.patch,
> HDFS-12225-HDFS-10285-04.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12225) [SPS]: Optimize extended attributes for tracking SPS movements
[
https://issues.apache.org/jira/browse/HDFS-12225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131166#comment-16131166
]
Surendra Singh Lilhore commented on HDFS-12225:
---
Thanks [~rakeshr] for reviews.
bq. How about renaming Candidate class, we could use StorageMovementTrackInfo
or SatisfyTrackInfo or any better name. Also, IMHO to use isDir bool flag to
classify dir/file and make it explicit.
I didn't got the 6th comment. In {{Candidate}} class {{childCount}} field is
not there.
> [SPS]: Optimize extended attributes for tracking SPS movements
> --
>
> Key: HDFS-12225
> URL: https://issues.apache.org/jira/browse/HDFS-12225
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12225-HDFS-10285-01.patch,
> HDFS-12225-HDFS-10285-02.patch
>
>
> We have discussed to optimize number extended attributes and asked to report
> separate JIRA while implementing [HDFS-11150 |
> https://issues.apache.org/jira/browse/HDFS-11150?focusedCommentId=15766127=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15766127]
> This is the JIRA to track that work
> For the context, comment copied from HDFS-11150
> {quote}
> [~yuanbo] wrote : I've tried that before. There is an issue here if we only
> mark the directory. When recovering from FsImage, the InodeMap isn't built
> up, so we don't know the sub-inode of a given inode, in the end, We cannot
> add these inodes to movement queue in FSDirectory#addToInodeMap, any
> thoughts?{quote}
> {quote}
> [~umamaheswararao] wrote: I got what you are saying. Ok for simplicity we can
> add for all Inodes now. For this to handle 100%, we may need intermittent
> processing, like first we should add them to some intermittentList while
> loading fsImage, once fully loaded and when starting active services, we
> should process that list and do required stuff. But it would add some
> additional complexity may be. Let's do with all file inodes now and we can
> revisit later if it is really creating issues. How about you raise a JIRA for
> it and think to optimize separately?
> {quote}
> {quote}
> [~andrew.wang] wrote in HDFS-10285 merge time review comment : HDFS-10899
> also the cursor of the iterator in the EZ root xattr to track progress and
> handle restarts. I wonder if we can do something similar here to avoid having
> an xattr-per-file being moved.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[
https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144821#comment-16144821
]
Surendra Singh Lilhore commented on HDFS-11968:
---
Hi [~msingh],
Thanks for the patch. I have one suggestion. HADOOP-13272 implemented storage
policy APIs in ViewFs. I think you can remove DFS check and directly send the
call to give filesystem. If in give filesystem this APIs are not implemented
then command will get UnsupportedOperationException.
> ViewFS: StoragePolicies commands fail with HDFS federation
> --
>
> Key: HDFS-11968
> URL: https://issues.apache.org/jira/browse/HDFS-11968
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.1
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch,
> HDFS-11968.003.patch, HDFS-11968.004.patch
>
>
> hdfs storagepolicies command fails with HDFS federation.
> For storage policies commands, a given user path should be resolved to a HDFS
> path and
> storage policy command should be applied onto the resolved HDFS path.
> {code}
> static DistributedFileSystem getDFS(Configuration conf)
> throws IOException {
> FileSystem fs = FileSystem.get(conf);
> if (!(fs instanceof DistributedFileSystem)) {
> throw new IllegalArgumentException("FileSystem " + fs.getUri() +
> " is not an HDFS file system");
> }
> return (DistributedFileSystem)fs;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[ https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16152380#comment-16152380 ] Surendra Singh Lilhore commented on HDFS-12310: --- Thanks [~umamaheswararao] Proposal LGTM... > [SPS]: Provide an option to track the status of in progress requests > > > Key: HDFS-12310 > URL: https://issues.apache.org/jira/browse/HDFS-12310 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G > > As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA > for tracking about the options how we track the progress of SPS requests. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12291: -- Attachment: HDFS-12291-HDFS-10285-02.patch > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[
https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145158#comment-16145158
]
Surendra Singh Lilhore commented on HDFS-12291:
---
Attached v2 patch. Please review.
Thanks [~umamaheswararao] and [~rakeshr] for offline discussion. Reusing the
EDEKs(HDFS-10899) code to traverse directory recursively.
[~xiaochen], can you look in to v2 patch and give your opinion. I changed some
code in {{ReencryptionHandler}}.
> [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy
> of all the files under the given dir
> -
>
> Key: HDFS-12291
> URL: https://issues.apache.org/jira/browse/HDFS-12291
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12291-HDFS-10285-01.patch,
> HDFS-12291-HDFS-10285-02.patch
>
>
> For the given source path directory, presently SPS consider only the files
> immediately under the directory(only one level of scanning) for satisfying
> the policy. It WON’T do recursive directory scanning and then schedules SPS
> tasks to satisfy the storage policy of all the files till the leaf node.
> The idea of this jira is to discuss & implement an efficient recursive
> directory iteration mechanism and satisfies storage policy for all the files
> under the given directory.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12291) [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir
[ https://issues.apache.org/jira/browse/HDFS-12291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153666#comment-16153666 ] Surendra Singh Lilhore commented on HDFS-12291: --- Thanks [~xiaochen], [~umamaheswararao], [~rakeshr] for review... I will update the patch soon.. > [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy > of all the files under the given dir > - > > Key: HDFS-12291 > URL: https://issues.apache.org/jira/browse/HDFS-12291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12291-HDFS-10285-01.patch, > HDFS-12291-HDFS-10285-02.patch > > > For the given source path directory, presently SPS consider only the files > immediately under the directory(only one level of scanning) for satisfying > the policy. It WON’T do recursive directory scanning and then schedules SPS > tasks to satisfy the storage policy of all the files till the leaf node. > The idea of this jira is to discuss & implement an efficient recursive > directory iteration mechanism and satisfies storage policy for all the files > under the given directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12556:
--
Attachment: HDFS-12556-HDFS-10285-02.patch
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch,
> HDFS-12556-HDFS-10285-02.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12556) [SPS] : Block movement analysis should be done in read lock.
[
https://issues.apache.org/jira/browse/HDFS-12556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16195845#comment-16195845
]
Surendra Singh Lilhore commented on HDFS-12556:
---
Attached v2 patch. This patch fixing two test case.
# TestPersistentStoragePolicySatisfier#testWithRestarts()
# TestPersistentStoragePolicySatisfier#testWithCheckpoint()
First test is failing because of {{ArrayIndexOutOfBoundsException}} and it is
coming because block analysis is done without read lock. Second test is failing
because first test case. First test case timed out without waiting to shutdown
the cluster. This is cause the inconsistency in namedir for second test case.
> [SPS] : Block movement analysis should be done in read lock.
>
>
> Key: HDFS-12556
> URL: https://issues.apache.org/jira/browse/HDFS-12556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12556-HDFS-10285-01.patch,
> HDFS-12556-HDFS-10285-02.patch
>
>
> {noformat}
> 2017-09-27 15:58:32,852 [StoragePolicySatisfier] ERROR
> namenode.StoragePolicySatisfier
> (StoragePolicySatisfier.java:handleException(308)) - StoragePolicySatisfier
> thread received runtime exception. Stopping Storage policy satisfier work
> java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getStorages(BlockManager.java:4130)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.analyseBlocksStorageMovementsAndAssignToDN(StoragePolicySatisfier.java:362)
> at
> org.apache.hadoop.hdfs.server.namenode.StoragePolicySatisfier.run(StoragePolicySatisfier.java:236)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[ https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12310: -- Attachment: HDFS-12310-HDFS-10285-04.patch > [SPS]: Provide an option to track the status of in progress requests > > > Key: HDFS-12310 > URL: https://issues.apache.org/jira/browse/HDFS-12310 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12310-HDFS-10285-01.patch, > HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch, > HDFS-12310-HDFS-10285-04.patch > > > As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA > for tracking about the options how we track the progress of SPS requests. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[
https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216656#comment-16216656
]
Surendra Singh Lilhore commented on HDFS-12310:
---
Attached V4 patch..
bq. Please fix the comments:
Fixed
bq. It should put spsStatusInfo into the map?
Yes, need to put in map. null check I added only for safety, spsStatusInfo
never be null.
bq. The above code is in muitple places, i.e., removeItemTrackInfo, looks
suspicious.
{{removeItemTrackInfo}} handling three cases and trying to clean the status
from map in all the case. Status will be cleand from map only if it is in
SUCCESS status more than 5 minute.
# If trackInfo is directory and inode is null, means start inode is delete but
status still there in statusInfo map. We need to clean it, so marking it
success.
# If trackInfo is directory and inode not null, check pending work for
directory. If all the files processed in directory then mark is success.
# If trackInfo is file then just mark is success becuase all the blocks satisfy
the policy.
bq. Would it make sense to put cleanSpsStatus(); into {{ if
(!namesystem.isInSafeMode()) }}, and avoid run it for every processed inode.
Fixed. Added check to run {{cleanSpsStatus()}} every 5 minute.
bq. StoragePolicySatisfyPathStatusInfo#setSuccessStatus()/setInProgress() /
canRemoveStatus() can be project-private.
Fixed
bq. Could you add comment to DFSClient# checkStoragePolicySatisfyPathStatus()
as well?
Fixed
Please review..
> [SPS]: Provide an option to track the status of in progress requests
>
>
> Key: HDFS-12310
> URL: https://issues.apache.org/jira/browse/HDFS-12310
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12310-HDFS-10285-01.patch,
> HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch,
> HDFS-12310-HDFS-10285-04.patch
>
>
> As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA
> for tracking about the options how we track the progress of SPS requests.
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12857) StoragePolicyAdmin should support schema based path
[
https://issues.apache.org/jira/browse/HDFS-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269249#comment-16269249
]
Surendra Singh Lilhore commented on HDFS-12857:
---
I've committed this. Thanks for the review [~arpitagarwal].
> StoragePolicyAdmin should support schema based path
> ---
>
> Key: HDFS-12857
> URL: https://issues.apache.org/jira/browse/HDFS-12857
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12857.001.patch
>
>
> When we execute storagepolicy admin command with full schema path, then it
> will throw this exception
> {noformat}
> ./hdfs storagepolicies -getstoragepolicy -path
> hdfs://localhost:39133/user1/bar
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost:39133/user1/bar, expected: viewfs://cluster/
> {noformat}
> This is because path schema is not matching with {{fs.defaultFS}}.
> {{fs.defaultFS}} configured with {{viewFs}} and in file path {{hdfs}} is
> used. This is broken because of HDFS-11968
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12857) StoragePolicyAdmin should support schema based path
[
https://issues.apache.org/jira/browse/HDFS-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12857:
--
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> StoragePolicyAdmin should support schema based path
> ---
>
> Key: HDFS-12857
> URL: https://issues.apache.org/jira/browse/HDFS-12857
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: 3.1.0
>
> Attachments: HDFS-12857.001.patch
>
>
> When we execute storagepolicy admin command with full schema path, then it
> will throw this exception
> {noformat}
> ./hdfs storagepolicies -getstoragepolicy -path
> hdfs://localhost:39133/user1/bar
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost:39133/user1/bar, expected: viewfs://cluster/
> {noformat}
> This is because path schema is not matching with {{fs.defaultFS}}.
> {{fs.defaultFS}} configured with {{viewFs}} and in file path {{hdfs}} is
> used. This is broken because of HDFS-11968
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12857) StoragePolicyAdmin should support schema based path
Surendra Singh Lilhore created HDFS-12857:
-
Summary: StoragePolicyAdmin should support schema based path
Key: HDFS-12857
URL: https://issues.apache.org/jira/browse/HDFS-12857
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.1.0
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore
When we execute storagepolicy admin command with full schema path, then it will
throw this exception
{noformat}
./hdfs storagepolicies -getstoragepolicy -path hdfs://localhost:39133/user1/bar
java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:39133/user1/bar,
expected: viewfs://cluster/
{noformat}
This is because path schema is not matching with {{fs.defaultFS}}.
{{fs.defaultFS}} configured with {{viewFs}} and in file path {{hdfs}} is used.
This is broken because of HDFS-11968
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12857) StoragePolicyAdmin should support schema based path
[
https://issues.apache.org/jira/browse/HDFS-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12857:
--
Attachment: HDFS-12857.001.patch
Attaches patch. Please review
> StoragePolicyAdmin should support schema based path
> ---
>
> Key: HDFS-12857
> URL: https://issues.apache.org/jira/browse/HDFS-12857
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12857.001.patch
>
>
> When we execute storagepolicy admin command with full schema path, then it
> will throw this exception
> {noformat}
> ./hdfs storagepolicies -getstoragepolicy -path
> hdfs://localhost:39133/user1/bar
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost:39133/user1/bar, expected: viewfs://cluster/
> {noformat}
> This is because path schema is not matching with {{fs.defaultFS}}.
> {{fs.defaultFS}} configured with {{viewFs}} and in file path {{hdfs}} is
> used. This is broken because of HDFS-11968
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12857) StoragePolicyAdmin should support schema based path
[
https://issues.apache.org/jira/browse/HDFS-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12857:
--
Status: Patch Available (was: Open)
> StoragePolicyAdmin should support schema based path
> ---
>
> Key: HDFS-12857
> URL: https://issues.apache.org/jira/browse/HDFS-12857
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.1.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12857.001.patch
>
>
> When we execute storagepolicy admin command with full schema path, then it
> will throw this exception
> {noformat}
> ./hdfs storagepolicies -getstoragepolicy -path
> hdfs://localhost:39133/user1/bar
> java.lang.IllegalArgumentException: Wrong FS:
> hdfs://localhost:39133/user1/bar, expected: viewfs://cluster/
> {noformat}
> This is because path schema is not matching with {{fs.defaultFS}}.
> {{fs.defaultFS}} configured with {{viewFs}} and in file path {{hdfs}} is
> used. This is broken because of HDFS-11968
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12833) In Distcp, Delete option not having the proper usage message.
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265151#comment-16265151
]
Surendra Singh Lilhore commented on HDFS-12833:
---
Thanks [~peruguusha] for patch.
Minor comment.
Please give the space between source and delete in {{DistCpOptionSwitch.java}}
and same in {{DistCp.md.vm}} for {{enable.Delete}}.
> In Distcp, Delete option not having the proper usage message.
> -
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12833) In Distcp, Delete option not having the proper usage message.
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265647#comment-16265647
]
Surendra Singh Lilhore commented on HDFS-12833:
---
+1 LGTM.. will wait for QA report...
> In Distcp, Delete option not having the proper usage message.
> -
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[
https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273971#comment-16273971
]
Surendra Singh Lilhore commented on HDFS-10285:
---
Thanks [~umamaheswararao], [~rakeshr] and [~anu] for new design. Starting SPS
as independent service is good proposal, really it will reduce the namenode
workload.
I have one question for new design. How other services (like hbase) will
communicate with SPS ? Do they need to create new client for SPS service or
{{DistributedFileSystem}} API only internally redirect call to the SPS service
?
> Storage Policy Satisfier in Namenode
>
>
> Key: HDFS-10285
> URL: https://issues.apache.org/jira/browse/HDFS-10285
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: HDFS-10285
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-10285-consolidated-merge-patch-00.patch,
> HDFS-10285-consolidated-merge-patch-01.patch,
> HDFS-10285-consolidated-merge-patch-02.patch,
> HDFS-10285-consolidated-merge-patch-03.patch,
> HDFS-SPS-TestReport-20170708.pdf,
> Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf,
> Storage-Policy-Satisfier-in-HDFS-May10.pdf,
> Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf
>
>
> Heterogeneous storage in HDFS introduced the concept of storage policy. These
> policies can be set on directory/file to specify the user preference, where
> to store the physical block. When user set the storage policy before writing
> data, then the blocks could take advantage of storage policy preferences and
> stores physical block accordingly.
> If user set the storage policy after writing and completing the file, then
> the blocks would have been written with default storage policy (nothing but
> DISK). User has to run the ‘Mover tool’ explicitly by specifying all such
> file names as a list. In some distributed system scenarios (ex: HBase) it
> would be difficult to collect all the files and run the tool as different
> nodes can write files separately and file can have different paths.
> Another scenarios is, when user rename the files from one effected storage
> policy file (inherited policy from parent directory) to another storage
> policy effected directory, it will not copy inherited storage policy from
> source. So it will take effect from destination file/dir parent storage
> policy. This rename operation is just a metadata change in Namenode. The
> physical blocks still remain with source storage policy.
> So, Tracking all such business logic based file names could be difficult for
> admins from distributed nodes(ex: region servers) and running the Mover tool.
> Here the proposal is to provide an API from Namenode itself for trigger the
> storage policy satisfaction. A Daemon thread inside Namenode should track
> such calls and process to DN as movement commands.
> Will post the detailed design thoughts document soon.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298205#comment-16298205
]
Surendra Singh Lilhore commented on HDFS-9023:
--
Thanks [~xiaochen] for looking in to this issue.
bq. Would you have time to take a look?
Yeah. I will post my review comment in HDFS-12726
bq. I can also post here and close HDFS-12726 as a dup if you don't mind.
Yes, you can attached your patch in this jira. I will assign this to you.
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Critical
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore reassigned HDFS-9023:
Assignee: Xiao Chen (was: Surendra Singh Lilhore)
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Xiao Chen
>Priority: Critical
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12726) BlockPlacementPolicyDefault's debugLoggingBuilder may not be logged
[
https://issues.apache.org/jira/browse/HDFS-12726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298258#comment-16298258
]
Surendra Singh Lilhore commented on HDFS-12726:
---
Thanks [~xiaochen] for patch..
Some review comment for v1 patch.
# {{private static final ThreadLocal chooseRandomReasons}}
Here HashMap is a raw type. it should be parameterized.
e.x :
{code}
private static final ThreadLocal>
chooseRandomReasons = ThreadLocal
.withInitial(() -> new HashMap<>());
{code}
# No need to log failure reason here.
{code}
Map reasonMap = chooseRandomReasons.get();
if (!reasonMap.isEmpty()) {
LOG.info("Not enough replicas was chosen. Reason:{}", reasonMap);
}
throw new NotEnoughReplicasException(detail);
{code}
just append reason in {{detail}} string. {{NotEnoughReplicasException}} message
will be logged in {{chooseTarget(..)}} method.
{code}
if (LOG.isTraceEnabled()) {
LOG.trace(message, e);
} else {
LOG.warn(message + " " + e.getMessage());
}
{code}
# I feel {{NodeNotChosenReason#NO_GOOD_STORAGE}} should be
{{NodeNotChosenReason#NOTE_ENOUGH_STORAGE_SPACE}}. This we are using when
{{(requiredSize > remaining - scheduledSize)}}
> BlockPlacementPolicyDefault's debugLoggingBuilder may not be logged
> ---
>
> Key: HDFS-12726
> URL: https://issues.apache.org/jira/browse/HDFS-12726
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Labels: supportability
> Attachments: HDFS-12726.01.patch
>
>
> During debugging HDFS-12725, {{BlockPlacementPolicyDefault's}} class'
> {{debugLoggingBuilder}} does a lot of {{get}} and {{append}}, but never
> {{toString}} and {{LOG.debug}}'ed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288728#comment-16288728
]
Surendra Singh Lilhore edited comment on HDFS-12833 at 12/13/17 5:29 AM:
-
committed to branch-2, branch-2.9 and branch-2.8. Thanks [~usharani] for
contribution.
was (Author: surendrasingh):
committed to branch-2 and branch-2.8. Thanks [~usharani] for contribution.
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 2.8.0
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4
>
> Attachments: HDFS-12833-branch-2.001.patch,
> HDFS-12833-branch-2.committed.patch, HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12833:
--
Attachment: HDFS-12833-branch-2.committed.patch
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.8.4
>
> Attachments: HDFS-12833-branch-2.001.patch,
> HDFS-12833-branch-2.committed.patch, HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12833:
--
Resolution: Fixed
Fix Version/s: 2.8.4
2.10.0
3.1.0
Status: Resolved (was: Patch Available)
committed to branch-2 and branch-2.8. Thanks [~usharani] for contribution.
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.8.4
>
> Attachments: HDFS-12833-branch-2.001.patch,
> HDFS-12833-branch-2.committed.patch, HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12833:
--
Affects Version/s: (was: 3.0.0-alpha1)
2.8.0
Fix Version/s: 2.9.1
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 2.8.0
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4
>
> Attachments: HDFS-12833-branch-2.001.patch,
> HDFS-12833-branch-2.committed.patch, HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12837) Intermittent failure TestReencryptionWithKMS#testReencryptionKMSDown
Surendra Singh Lilhore created HDFS-12837: - Summary: Intermittent failure TestReencryptionWithKMS#testReencryptionKMSDown Key: HDFS-12837 URL: https://issues.apache.org/jira/browse/HDFS-12837 Project: Hadoop HDFS Issue Type: Bug Components: kms, namenode Affects Versions: 3.0.0-beta1 Reporter: Surendra Singh Lilhore https://builds.apache.org/job/PreCommit-HDFS-Build/22112/testReport/org.apache.hadoop.hdfs.server.namenode/TestReencryptionWithKMS/testReencryptionKMSDown/ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Attachment: HDFS-12106-HDFS-10285-04.patch
Attached v4 patch. Fixed checkstyle and failed tests case.
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch,
> HDFS-12106-HDFS-10285-04.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Attachment: HDFS-12106-HDFS-10285-03.patch
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16247990#comment-16247990
]
Surendra Singh Lilhore commented on HDFS-12106:
---
Thanks [~rakeshr] for review..
Attached V3 patch. please review...
Fixed Comments:
# Comment1 : Fixed
# Comment2 : Fixed
# Comment3 : Agree.Changed to 0.
# Comment4 : Changed to switch case
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12824) ViewFileSystem should support EC.
[
https://issues.apache.org/jira/browse/HDFS-12824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore reassigned HDFS-12824:
-
Assignee: Surendra Singh Lilhore
> ViewFileSystem should support EC.
> -
>
> Key: HDFS-12824
> URL: https://issues.apache.org/jira/browse/HDFS-12824
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, fs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: Surendra Singh Lilhore
>
> Current {{ViewFileSystem}} does not support EC, it will throw
> {{IllegalArgumentException}}.
> {noformat}
> ./hdfs ec -listPolicies
> IllegalArgumentException: FileSystem viewfs://ClusterX/ is not an HDFS file
> system
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12825) After Block Corrupted, FSCK Report printing the Direct configuration.
[
https://issues.apache.org/jira/browse/HDFS-12825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12825:
--
Labels: newbie (was: )
> After Block Corrupted, FSCK Report printing the Direct configuration.
> ---
>
> Key: HDFS-12825
> URL: https://issues.apache.org/jira/browse/HDFS-12825
> Project: Hadoop HDFS
> Issue Type: Wish
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Priority: Minor
> Labels: newbie
> Attachments: error.JPG
>
>
> Scenario:
> Corrupt the Block in any datanode
> Take the *FSCK *Report for that file.
> Actual Output:
> ==
> printing the direct configuration in fsck report
> {{dfs.namenode.replication.min}}
> Expected Output:
>
> it should be {{MINIMAL BLOCK REPLICATION}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12825) After Block Corrupted, FSCK Report printing the Direct configuration.
[
https://issues.apache.org/jira/browse/HDFS-12825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12825:
--
Issue Type: Bug (was: Wish)
> After Block Corrupted, FSCK Report printing the Direct configuration.
> ---
>
> Key: HDFS-12825
> URL: https://issues.apache.org/jira/browse/HDFS-12825
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Priority: Minor
> Labels: newbie
> Attachments: error.JPG
>
>
> Scenario:
> Corrupt the Block in any datanode
> Take the *FSCK *Report for that file.
> Actual Output:
> ==
> printing the direct configuration in fsck report
> {{dfs.namenode.replication.min}}
> Expected Output:
>
> it should be {{MINIMAL BLOCK REPLICATION}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
Thanks [~rakeshr] for review. I have fixed typo and pushed it to branch.
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch,
> HDFS-12106-HDFS-10285-04.patch, HDFS-12106-HDFS-10285-05-Committed.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Attachment: HDFS-12106-HDFS-10285-05-Committed.patch
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Fix For: HDFS-10285
>
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch,
> HDFS-12106-HDFS-10285-04.patch, HDFS-12106-HDFS-10285-05-Committed.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251932#comment-16251932
]
Surendra Singh Lilhore commented on HDFS-12106:
---
Failed tests are unrelated.. Failed SPS tests are passing locally...
{code}
---
T E S T S
---
Running
org.apache.hadoop.hdfs.server.namenode.TestStoragePolicySatisfierWithStripedFile
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.311 sec - in
org.apache.hadoop.hdfs.server.namenode.TestStoragePolicySatisfierWithStripedFile
Running org.apache.hadoop.hdfs.server.namenode.TestStoragePolicySatisfier
Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 364.026 sec -
in org.apache.hadoop.hdfs.server.namenode.TestStoragePolicySatisfier
Results :
Tests run: 36, Failures: 0, Errors: 0, Skipped: 0
{code}
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch, HDFS-12106-HDFS-10285-03.patch,
> HDFS-12106-HDFS-10285-04.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243414#comment-16243414
]
Surendra Singh Lilhore commented on HDFS-12106:
---
Thanks [~rakeshr] for review..
I fixed all above review comments. Please review..
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Attachment: HDFS-12106-HDFS-10285-02.patch
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch,
> HDFS-12106-HDFS-10285-02.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12790) [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits
[ https://issues.apache.org/jira/browse/HDFS-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245642#comment-16245642 ] Surendra Singh Lilhore commented on HDFS-12790: --- Thanks [~rakeshr] for patch. Latest patch looks good to me. Please fix the checkstyle issues. I run the failed SPS test locally, its passed > [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 > commits > - > > Key: HDFS-12790 > URL: https://issues.apache.org/jira/browse/HDFS-12790 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-12790-HDFS-10285-00.patch, > HDFS-12790-HDFS-10285-01.patch, HDFS-12790-HDFS-10285-02.patch > > > This task is a continuation with the periodic HDFS-10285 branch code rebasing > with the trunk code. To make branch code compile with the trunk code, it > needs to be refactored with the latest trunk code changes - HDFS-10467, > HDFS-12599 and HDFS-11968. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12833:
--
Summary: Distcp : Update the usage of delete option for dependency with
update and overwrite option (was: In Distcp, Delete option not having the
proper usage message.)
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12833) Distcp : Update the usage of delete option for dependency with update and overwrite option
[
https://issues.apache.org/jira/browse/HDFS-12833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286419#comment-16286419
]
Surendra Singh Lilhore commented on HDFS-12833:
---
Committed to trunk. [~usharani] Can you attach the patch from branch2 ?.
> Distcp : Update the usage of delete option for dependency with update and
> overwrite option
> --
>
> Key: HDFS-12833
> URL: https://issues.apache.org/jira/browse/HDFS-12833
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp, hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Harshakiran Reddy
>Assignee: usharani
>Priority: Minor
> Attachments: HDFS-12833.001.patch, HDFS-12833.patch
>
>
> Basically Delete option applicable only with update or overwrite options. I
> tried as per usage message am getting the bellow exception.
> {noformat}
> bin:> ./hadoop distcp -delete /Dir1/distcpdir /Dir/distcpdir5
> 2017-11-17 20:48:09,828 ERROR tools.DistCp: Invalid arguments:
> java.lang.IllegalArgumentException: Delete missing is applicable only with
> update or overwrite options
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.validate(DistCpOptions.java:528)
> at
> org.apache.hadoop.tools.DistCpOptions$Builder.build(DistCpOptions.java:487)
> at org.apache.hadoop.tools.OptionsParser.parse(OptionsParser.java:233)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:141)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:432)
> Invalid arguments: Delete missing is applicable only with update or overwrite
> options
> usage: distcp OPTIONS [source_path...]
> OPTIONS
> -append Reuse existing data in target files and
>append new data to them if possible
> -asyncShould distcp execution be blocking
> -atomic Commit all changes or none
> -bandwidth Specify bandwidth per map in MB, accepts
>bandwidth as a fraction.
> -blocksperchunk If set to a positive value, fileswith more
>blocks than this value will be split into
>chunks of blocks to be
>transferred in parallel, and reassembled on
>the destination. By default,
> is 0 and the files will be
>transmitted in their entirety without
>splitting. This switch is only applicable
>when the source file system implements
>getBlockLocations method and the target
>file system implements concat method
> -copybuffersize Size of the copy buffer to use. By default
> is 8192B.
> -delete Delete from target, files missing in source
> -diffUse snapshot diff report to identify the
>difference between source and target
> {noformat}
> Even in Document also it's not updated proper usage.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Status: Patch Available (was: Open)
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12106) [SPS]: Improve storage policy satisfier configurations
[
https://issues.apache.org/jira/browse/HDFS-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12106:
--
Attachment: HDFS-12106-HDFS-10285-01.patch
Attached initial patch...
Please review
> [SPS]: Improve storage policy satisfier configurations
> --
>
> Key: HDFS-12106
> URL: https://issues.apache.org/jira/browse/HDFS-12106
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-12106-HDFS-10285-01.patch
>
>
> Following are the changes doing as part of this task:-
> # Make satisfy policy retry configurable.
> Based on
> [discussion|https://issues.apache.org/jira/browse/HDFS-11965?focusedCommentId=16074338=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16074338]
> in HDFS-11965, we can make satisfy policy retry configurable.
> # Change {{dfs.storage.policy.satisfier.low.max-streams.preference}}'s value
> to {{true}} and modify the default value to true as well. If user wants equal
> share then it should be false, but presently it is true which is not correct.
> Thanks [~umamaheswararao] for pointing out this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[ https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-12310: -- Attachment: HDFS-12310-HDFS-10285-05.patch Attached v5 patch. Fixed failed tests and check-style warnings. > [SPS]: Provide an option to track the status of in progress requests > > > Key: HDFS-12310 > URL: https://issues.apache.org/jira/browse/HDFS-12310 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12310-HDFS-10285-01.patch, > HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch, > HDFS-12310-HDFS-10285-04.patch, HDFS-12310-HDFS-10285-05.patch > > > As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA > for tracking about the options how we track the progress of SPS requests. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12310) [SPS]: Provide an option to track the status of in progress requests
[ https://issues.apache.org/jira/browse/HDFS-12310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213734#comment-16213734 ] Surendra Singh Lilhore commented on HDFS-12310: --- Thanks [~eddyxu] for review.. I will update the patch soon.. > [SPS]: Provide an option to track the status of in progress requests > > > Key: HDFS-12310 > URL: https://issues.apache.org/jira/browse/HDFS-12310 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Uma Maheswara Rao G >Assignee: Surendra Singh Lilhore > Attachments: HDFS-12310-HDFS-10285-01.patch, > HDFS-12310-HDFS-10285-02.patch, HDFS-12310-HDFS-10285-03.patch > > > As per the [~andrew.wang] 's review comments in HDFS-10285, This is the JIRA > for tracking about the options how we track the progress of SPS requests. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13614) DN failed to connect with NN because of NPE in SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HDFS-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488493#comment-16488493
]
Surendra Singh Lilhore commented on HDFS-13614:
---
{noformat}
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
set up IO streams; Host Details : local host is: "XXX"; destination
host is: "XXX;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:799)
at org.apache.hadoop.ipc.Client.call(Client.java:1522)
at org.apache.hadoop.ipc.Client.call(Client.java:1454)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy15.blockReceivedAndDeleted(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:251)
at
org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:180)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:575)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:718)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't set up IO streams
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:834)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:394)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1571)
at org.apache.hadoop.ipc.Client.call(Client.java:1493)
... 8 more
Caused by: java.util.NoSuchElementException
at java.util.LinkedList.removeLast(LinkedList.java:283)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:414)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:652)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757){noformat}
> DN failed to connect with NN because of NPE in SocketIOWithTimeout
> --
>
> Key: HDFS-13614
> URL: https://issues.apache.org/jira/browse/HDFS-13614
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Priority: Major
>
> {\{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
> Because of this socket connections are failing. It may be java bug also..
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023148
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13614) DN failed to connect with NN because of NPE in SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HDFS-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488492#comment-16488492
]
Surendra Singh Lilhore commented on HDFS-13614:
---
{noformat}
java.lang.NullPointerException
at java.util.LinkedList$ListItr.next(LinkedList.java:893)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.trimIdleSelectors(SocketIOWithTimeout.java:447)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.release(SocketIOWithTimeout.java:429)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:373)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2483)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1333)
at java.lang.Thread.run(Thread.java:748)
===
java.lang.NullPointerException
at java.util.LinkedList$ListItr.next(LinkedList.java:893)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.trimIdleSelectors(SocketIOWithTimeout.java:447)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:417)
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:325)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:526)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:938)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:900)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:138)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:265)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> DN failed to connect with NN because of NPE in SocketIOWithTimeout
> --
>
> Key: HDFS-13614
> URL: https://issues.apache.org/jira/browse/HDFS-13614
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Priority: Major
>
> {\{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
> Because of this socket connections are failing. It may be java bug also..
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023148
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13614) DN failed to connect with NN because of NPE in SocketIOWithTimeout
Surendra Singh Lilhore created HDFS-13614:
-
Summary: DN failed to connect with NN because of NPE in
SocketIOWithTimeout
Key: HDFS-13614
URL: https://issues.apache.org/jira/browse/HDFS-13614
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.2
Reporter: Surendra Singh Lilhore
{\{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
Because of this socket connections are failing. It may be java bug also..
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023148
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13614) DN failed to connect with NN because of NPE in SocketIOWithTimeout
[
https://issues.apache.org/jira/browse/HDFS-13614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13614:
--
Description:
{{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
Because of this socket connections are failing. It may be java bug also..
[https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133715]
was:
{\{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
Because of this socket connections are failing. It may be java bug also..
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023148
> DN failed to connect with NN because of NPE in SocketIOWithTimeout
> --
>
> Key: HDFS-13614
> URL: https://issues.apache.org/jira/browse/HDFS-13614
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2
>Reporter: Surendra Singh Lilhore
>Priority: Major
>
> {{LinkedList$ListItr.next()}} is throwing NPE in {{SocketIOWithTimeout}}.
> Because of this socket connections are failing. It may be java bug also..
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8133715]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[ https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16301041#comment-16301041 ] Surendra Singh Lilhore commented on HDFS-12955: --- Thanks [~rakeshr] for clarification. bq. I have named the package namenode.sps, keeping in mind that the current implementation classes are specific to the NN side service. Does this make sense to you? ok.. > [SPS]: Move SPS classes to a separate package > - > > Key: HDFS-12955 > URL: https://issues.apache.org/jira/browse/HDFS-12955 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nn >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R >Priority: Trivial > Attachments: HDFS-12955-HDFS-10285-00.patch, > HDFS-12955-HDFS-10285-01.patch > > > For clean modularization, it would be good if we moved SPS related classes > into its own package. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[
https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300984#comment-16300984
]
Surendra Singh Lilhore commented on HDFS-12955:
---
Thanks [~rakeshr] for patch.
I feel sps package name should be {{org.apache.hadoop.hdfs.server.sps}}. We are
planning to start SPS as a separate service, so it should follow other service
package name
{noformat}
org.apache.hadoop.hdfs.server.datanode
org.apache.hadoop.hdfs.server.namenode
org.apache.hadoop.hdfs.server.mover
{noformat}
> [SPS]: Move SPS classes to a separate package
> -
>
> Key: HDFS-12955
> URL: https://issues.apache.org/jira/browse/HDFS-12955
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nn
>Affects Versions: HDFS-10285
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
>Priority: Trivial
> Attachments: HDFS-12955-HDFS-10285-00.patch,
> HDFS-12955-HDFS-10285-01.patch
>
>
> For clean modularization, it would be good if we moved SPS related classes
> into its own package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9023) When NN is not able to identify DN for replication, reason behind it can be logged
[
https://issues.apache.org/jira/browse/HDFS-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16302920#comment-16302920
]
Surendra Singh Lilhore commented on HDFS-9023:
--
Thanks [~xiaochen] for the patch. V2 patch almost looks good to me.
Some minor comment..
# No need to put {{if (LOG.isDebugEnabled() && builder != null)}} check in else
part. Already {{LOG.isDebugEnabled()}} check done in first {{if}}. So it shoud
be like this.
{code}
if (LOG.isDebugEnabled() && builder != null) {
detail = builder.toString();
if (badTarget) {
builder.setLength(0);
} else {
if (detail.length() > 1) {
// only log if there's more than "[", which is always appended at
// the beginning of this method.
LOG.debug(builder.toString());
}
detail = "";
}
}
{code}
# Just give the HashMap generic paramter here.
{code}+ HashMap reasonMap = CHOOSE_RANDOM_REASONS.get();{code}
# Log message should be {{warn}}
{code}+LOG.info("Not enough replicas was chosen. Reason:{}",
reasonMap);{code}
> When NN is not able to identify DN for replication, reason behind it can be
> logged
> --
>
> Key: HDFS-9023
> URL: https://issues.apache.org/jira/browse/HDFS-9023
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, namenode
>Affects Versions: 2.7.1
>Reporter: Surendra Singh Lilhore
>Assignee: Xiao Chen
>Priority: Critical
> Attachments: HDFS-9023.01.patch, HDFS-9023.02.patch
>
>
> When NN is not able to identify DN for replication, reason behind it can be
> logged (at least critical information why DNs not chosen like disk is full).
> At present it is expected to enable debug log.
> For example the reason for below error looks like all 7 DNs are busy for data
> writes. But at client or NN side no hint is given in the log message.
> {noformat}
> File /tmp/logs/spark/logs/application_1437051383180_0610/xyz-195_26009.tmp
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 7 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1553)
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12987) Document - Disabling the Lazy persist file scrubber.
[
https://issues.apache.org/jira/browse/HDFS-12987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311142#comment-16311142
]
Surendra Singh Lilhore commented on HDFS-12987:
---
+1, pending Jenkins..
> Document - Disabling the Lazy persist file scrubber.
>
>
> Key: HDFS-12987
> URL: https://issues.apache.org/jira/browse/HDFS-12987
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, hdfs
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>Priority: Trivial
> Labels: documentation
> Attachments: HDFS-12987.patch
>
>
> LazyPersistFileScrubber will be disabled if scrubber interval configured to
> zero. But the document was incorrect - "Set it to a negative value to disable
> this behavior".
> Namenode will not start if we set to negative value. We have another strong
> check to prevent it.
> {code:java}
>if (this.lazyPersistFileScrubIntervalSec < 0) {
> throw new IllegalArgumentException(
> DFS_NAMENODE_LAZY_PERSIST_FILE_SCRUB_INTERVAL_SEC
> + " must be zero (for disable) or greater than zero.");
> }
> {code}
> This seems reflected due to
> [HDFS-8276|https://issues.apache.org/jira/browse/HDFS-8276]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313177#comment-16313177
]
Surendra Singh Lilhore commented on HDFS-12982:
---
Thanks [~rakeshr] for patch. I have some review comment.
# I feel {{Context}} object should't know about the SPS running status.
{{Context#isRunning()}} should return only the Namenode running status.
{{StoragePolicySatisfier#isRunning()}} method should give overall status of
service (inclusding namenode).
So {{SPS#isRunning()}} implementation should be like this
{code}
public boolean isRunning() {
return isRunning && ctxt.isRunning();
}
{code}
# Currently submit task action is little confusing. It’s getting task handler
from context and finally task handle call {{context}} API to move the block.
{noformat}
SPS#getBlockMoveTaskHandler()==>Context.getBlockMoveTaskHandler()==>IntraSPSNameNodeBlockMoveTaskHandler#submitMoveTask()===>Context#assignBlockMoveTaskToTargetNode()
{noformat}
It’s better to remove BlockMoveTaskHandler from the context and pass it to the
SPS constructor. When SPS is created from BlockManager then create
{{IntraSPSNameNodeBlockMoveTaskHandler}} object and pass, otherwise create
external task handler and pass.
# In {{IntraSPSNameNodeContext#getFileInfo()}} no need to take the read lock.
{{getFilePath()}} and {{getFileInfo()}} internally taking the read lock.
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12982-HDFS-10285-00.patch,
> HDFS-12982-HDFS-10285-01.patch, HDFS-12982-HDFS-10285-02.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316041#comment-16316041
]
Surendra Singh Lilhore commented on HDFS-12982:
---
+1 LGTM
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12982-HDFS-10285-00.patch,
> HDFS-12982-HDFS-10285-01.patch, HDFS-12982-HDFS-10285-02.patch,
> HDFS-12982-HDFS-10285-03.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-12982:
--
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: HDFS-10285
>
> Attachments: HDFS-12982-HDFS-10285-00.patch,
> HDFS-12982-HDFS-10285-01.patch, HDFS-12982-HDFS-10285-02.patch,
> HDFS-12982-HDFS-10285-03.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12982) [SPS]: Reduce the locking and cleanup the Namesystem access
[
https://issues.apache.org/jira/browse/HDFS-12982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316042#comment-16316042
]
Surendra Singh Lilhore commented on HDFS-12982:
---
I have just pushed it to branch, Thanks [~rakeshr]
> [SPS]: Reduce the locking and cleanup the Namesystem access
> ---
>
> Key: HDFS-12982
> URL: https://issues.apache.org/jira/browse/HDFS-12982
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12982-HDFS-10285-00.patch,
> HDFS-12982-HDFS-10285-01.patch, HDFS-12982-HDFS-10285-02.patch,
> HDFS-12982-HDFS-10285-03.patch
>
>
> This task is to optimize the NS lock usage in SPS and cleanup the Namesystem
> access via {{Context}} interface.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
Surendra Singh Lilhore created HDFS-13050:
-
Summary: [SPS] : Create start/stop script to start external SPS
process.
Key: HDFS-13050
URL: https://issues.apache.org/jira/browse/HDFS-13050
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: HDFS-10285
Reporter: Surendra Singh Lilhore
Assignee: Surendra Singh Lilhore
As part of this Jira we will main class for SPS and modify the
\{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338938#comment-16338938
]
Surendra Singh Lilhore commented on HDFS-13050:
---
Thanks [~umamaheswararao] for comment.
{quote}do we really need to check dependent service running?
{quote}
If context can take for infinite retry then its not required.
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13077) [SPS]: Fix review comments of external storage policy satisfier
[
https://issues.apache.org/jira/browse/HDFS-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13077:
--
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
+1 LGTM
Based on the previous report i am committing to the branch. Locally all sps
tests are passing with this patch.
> [SPS]: Fix review comments of external storage policy satisfier
> ---
>
> Key: HDFS-13077
> URL: https://issues.apache.org/jira/browse/HDFS-13077
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-10285
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13077-HDFS-10285-00.patch,
> HDFS-13077-HDFS-10285-01.patch
>
>
> This task is to address the following Uma's review comments:
> - Implement login with external SPS keytab
> - make SPS outstanding requests Q limit configurable. Configuration could be
> {{dfs.storage.policy.satisfier.max.outstanding.paths}}
> - fix checkstyle warnings
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13077) [SPS]: Fix review comments of external storage policy satisfier
[
https://issues.apache.org/jira/browse/HDFS-13077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343683#comment-16343683
]
Surendra Singh Lilhore commented on HDFS-13077:
---
Thanks [~rakeshr] for patch. I verified patch in secure cluster It is working
fine.
One minor comment, I feel \{{dfs.storage.policy.satisfier.keytab.enabled}}
property is not required.
{{SecurityUtil.login()}} will check internally if security is enable or not.
{code:java}
if(! UserGroupInformation.isSecurityEnabled())
return;{code}
> [SPS]: Fix review comments of external storage policy satisfier
> ---
>
> Key: HDFS-13077
> URL: https://issues.apache.org/jira/browse/HDFS-13077
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-10285
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13077-HDFS-10285-00.patch
>
>
> This task is to address the following Uma's review comments:
> - Implement login with external SPS keytab
> - make SPS outstanding requests Q limit configurable. Configuration could be
> {{dfs.storage.policy.satisfier.max.outstanding.paths}}
> - fix checkstyle warnings
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13050:
--
Attachment: HDFS-13050-HDFS-10285-03.patch
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch,
> HDFS-13050-HDFS-10285-02.patch, HDFS-13050-HDFS-10285-03.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342710#comment-16342710
]
Surendra Singh Lilhore edited comment on HDFS-13050 at 1/28/18 8:01 PM:
Thanks [~rakeshr] for review. Attached v3 patch.
was (Author: surendrasingh):
Attached v3 patch.
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch,
> HDFS-13050-HDFS-10285-02.patch, HDFS-13050-HDFS-10285-03.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342710#comment-16342710
]
Surendra Singh Lilhore commented on HDFS-13050:
---
Attached v3 patch.
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch,
> HDFS-13050-HDFS-10285-02.patch, HDFS-13050-HDFS-10285-03.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13075) [SPS]: Provide External Context implementation.
[ https://issues.apache.org/jira/browse/HDFS-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-13075: -- Attachment: HDFS-13075-HDFS-10285-02.patch > [SPS]: Provide External Context implementation. > --- > > Key: HDFS-13075 > URL: https://issues.apache.org/jira/browse/HDFS-13075 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13075-HDFS-10285-0.patch, > HDFS-13075-HDFS-10285-01.patch, HDFS-13075-HDFS-10285-02.patch, > HDFS-13075-HDFS-10285-1.patch > > > This JIRA to provide initial implementation of External Context. > With HDFS-12995, we improve further retry mechanism etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13050:
--
Attachment: HDFS-13075-HDFS-10285-02.patch
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13050) [SPS] : Create start/stop script to start external SPS process.
[
https://issues.apache.org/jira/browse/HDFS-13050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13050:
--
Attachment: (was: HDFS-13075-HDFS-10285-02.patch)
> [SPS] : Create start/stop script to start external SPS process.
> ---
>
> Key: HDFS-13050
> URL: https://issues.apache.org/jira/browse/HDFS-13050
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Blocker
> Attachments: HDFS-13050-HDFS-10285-01.patch
>
>
> As part of this Jira we will main class for SPS and modify the
> \{{hadoop-daemon.sh}} to start external SPS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13075) [SPS]: Provide External Context implementation.
[ https://issues.apache.org/jira/browse/HDFS-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342537#comment-16342537 ] Surendra Singh Lilhore commented on HDFS-13075: --- Re-based and attached v2 patch > [SPS]: Provide External Context implementation. > --- > > Key: HDFS-13075 > URL: https://issues.apache.org/jira/browse/HDFS-13075 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-13075-HDFS-10285-0.patch, > HDFS-13075-HDFS-10285-01.patch, HDFS-13075-HDFS-10285-02.patch, > HDFS-13075-HDFS-10285-1.patch > > > This JIRA to provide initial implementation of External Context. > With HDFS-12995, we improve further retry mechanism etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347575#comment-16347575 ] Surendra Singh Lilhore commented on HDFS-10285: --- Thanks [~daryn] for reviews. Create Part1 Jira HDFS-13097 to fix few comments > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-10285-consolidated-merge-patch-04.patch, > HDFS-10285-consolidated-merge-patch-05.patch, > HDFS-SPS-TestReport-20170708.pdf, SPS Modularization.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13097) [SPS]: Fix the branch review comments(Part1)
[ https://issues.apache.org/jira/browse/HDFS-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347573#comment-16347573 ] Surendra Singh Lilhore commented on HDFS-13097: --- Attached v1 patch > [SPS]: Fix the branch review comments(Part1) > > > Key: HDFS-13097 > URL: https://issues.apache.org/jira/browse/HDFS-13097 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-13097-HDFS-10285.01.patch > > > Fix the branch review comment. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13097) [SPS]: Fix the branch review comments(Part1)
[
https://issues.apache.org/jira/browse/HDFS-13097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347565#comment-16347565
]
Surendra Singh Lilhore commented on HDFS-13097:
---
Fixing below comments
*Comment-1)*
{quote}BlockManager
Shouldn’t spsMode be volatile? Although I question why it’s here.
{quote}
[Rakesh's reply] Agreed, will do the changes.
*Comment-2)*
{quote}Adding SPS methods to this class implies an unexpected coupling of the
SPS service to the block manager. Please move them out to prove it’s not
tightly coupled.
{quote}
[Rakesh's reply] Agreed. I'm planning to create {{StoragePolicySatisfyManager}}
and keep all the related apis over there.
*Comment-5)*
{quote}DatanodeDescriptor
Why use a synchronized linked list to offer/poll instead of BlockingQueue?
{quote}
[Rakesh's reply] Agreed, will do the changes.
*Comment-8)*
{quote}DFSUtil
DFSUtil.removeOverlapBetweenStorageTypes and {{DFSUtil.getSPSWorkMultiplier
}}. These aren’t generally useful methods so why are they in DFSUtil? Why
aren’t they in the only calling class StoragePolicySatisfier?
{quote}
[Rakesh's reply] Agreed, Will do the changes.
*Comment-11)*
{quote}HdfsServerConstants
The xattr is called user.hdfs.sps.xattr. Why does the xattr name actually
contain the word “xattr”?
{quote}
[Rakesh's reply] Sure, will remove “xattr” word.
*Comment-12)*
{quote}NameNode
Super trivial but using the plural pronoun “we” in this exception message is
odd. Changing the value isn’t a joint activity.
For enabling or disabling storage policy satisfier, we must pass either
none/internal/external string value only
{quote}
[Rakesh's reply] oops, sorry for the mistake. Will change it.
*Comment-16)*
{quote}FSDirStatAndListOp
Not sure why javadoc was changed to add needLocation. It's already present and
now doubled up.{quote}
[Rakesh'r reply] Agreed, will correct it.
*Comment-18)*
{quote}DFS_MOVER_MOVERTHREADS_DEFAULT is 1000 per DN? If the DN is concurrently
doing 1000 moves, it's not in a good state, disk io is probably saturated, and
this will only make it much worse. 10 is probably more than sufficient.{quote}
[Rakesh'r reply] Agreed, will make it to smaller value 10.
> [SPS]: Fix the branch review comments(Part1)
>
>
> Key: HDFS-13097
> URL: https://issues.apache.org/jira/browse/HDFS-13097
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-10285
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
>
> Fix the branch review comment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
