[
https://issues.apache.org/jira/browse/HDDS-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Soumitra Sulav updated HDDS-6700:
---------------------------------
Description:
DistCp fails with NPE with ofs scheme when a user doesn't have the privilege to
read the volume.
{code:java}
[root@nightly-71x-eu-1 ~]# kinit -kt /cdep/keytabs/systest.keytab systest
[root@nightly-71x-eu-1 ~]# /opt/cloudera/parcels/CDH/bin/hadoop distcp -i -pp
-skipcrccheck /tmp/testDir ofs://ozone1/test/test/test
22/05/04 17:15:10 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=true, overwrite=false, append=false, useDiff=false,
useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=true,
blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0,
copyStrategy='uniformsize', preserveStatus=[PERMISSION], atomicWorkPath=null,
logPath=null, sourceFileListing=null, sourcePaths=[/tmp/testDir],
targetPath=ofs://ozone1/test/test/test, filtersFile='null', blocksPerChunk=0,
copyBufferSize=8192, verboseLog=false, directWrite=false},
sourcePaths=[/tmp/testDir], targetPathExists=false, preserveRawXattrsfalse
22/05/04 17:15:11 INFO client.RMProxy: Connecting to ResourceManager at
nightly-71x-eu-2.nightly-71x-eu.root.hwx.site/172.27.11.70:8032
22/05/04 17:15:12 ERROR tools.DistCp: Exception encountered
java.lang.NullPointerException
at
org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:134)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:443) {code}
Even on adding acls for systest as all permissions same NPE was seen.
Works only when the volume owner was changed to the test user i.e. systest
{code:java}
# ozone sh volume update test --user=systest {code}
was:
DistCp fails with NPE with ofs scheme when a user doesn't have the privilege to
read the volume.
{code:java}
[root@nightly-71x-eu-1 ~]# /opt/cloudera/parcels/CDH/bin/hadoop distcp -i -pp
-skipcrccheck /tmp/testDir ofs://ozone1/test/test/test
22/05/04 17:15:10 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=true, overwrite=false, append=false, useDiff=false,
useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=true,
blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0,
copyStrategy='uniformsize', preserveStatus=[PERMISSION], atomicWorkPath=null,
logPath=null, sourceFileListing=null, sourcePaths=[/tmp/testDir],
targetPath=ofs://ozone1/test/test/test, filtersFile='null', blocksPerChunk=0,
copyBufferSize=8192, verboseLog=false, directWrite=false},
sourcePaths=[/tmp/testDir], targetPathExists=false, preserveRawXattrsfalse
22/05/04 17:15:11 INFO client.RMProxy: Connecting to ResourceManager at
nightly-71x-eu-2.nightly-71x-eu.root.hwx.site/172.27.11.70:8032
22/05/04 17:15:12 ERROR tools.DistCp: Exception encountered
java.lang.NullPointerException
at
org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:134)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:443) {code}
Even on adding acls for systest as all permissions same NPE was seen.
Works only when the volume owner was changed to the test user i.e. systest
{code:java}
# ozone sh volume update test --user=systest {code}
> NPE with distcp on ofs with non-owner user
> ------------------------------------------
>
> Key: HDDS-6700
> URL: https://issues.apache.org/jira/browse/HDDS-6700
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 1.3.0
> Reporter: Soumitra Sulav
> Assignee: Saketa Chalamchala
> Priority: Major
> Attachments: Ranger Audit logs for the 2 attempts.png, Ranger
> Permissions.png
>
>
> DistCp fails with NPE with ofs scheme when a user doesn't have the privilege
> to read the volume.
> {code:java}
> [root@nightly-71x-eu-1 ~]# kinit -kt /cdep/keytabs/systest.keytab systest
> [root@nightly-71x-eu-1 ~]# /opt/cloudera/parcels/CDH/bin/hadoop distcp -i
> -pp -skipcrccheck /tmp/testDir ofs://ozone1/test/test/test
> 22/05/04 17:15:10 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=true, overwrite=false, append=false, useDiff=false,
> useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=true,
> blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0,
> copyStrategy='uniformsize', preserveStatus=[PERMISSION], atomicWorkPath=null,
> logPath=null, sourceFileListing=null, sourcePaths=[/tmp/testDir],
> targetPath=ofs://ozone1/test/test/test, filtersFile='null', blocksPerChunk=0,
> copyBufferSize=8192, verboseLog=false, directWrite=false},
> sourcePaths=[/tmp/testDir], targetPathExists=false, preserveRawXattrsfalse
> 22/05/04 17:15:11 INFO client.RMProxy: Connecting to ResourceManager at
> nightly-71x-eu-2.nightly-71x-eu.root.hwx.site/172.27.11.70:8032
> 22/05/04 17:15:12 ERROR tools.DistCp: Exception encountered
> java.lang.NullPointerException
> at
> org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:134)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:85)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:89)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:379)
> at org.apache.hadoop.tools.DistCp.prepareFileListing(DistCp.java:96)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:216)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:193)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:155)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:443) {code}
> Even on adding acls for systest as all permissions same NPE was seen.
> Works only when the volume owner was changed to the test user i.e. systest
> {code:java}
> # ozone sh volume update test --user=systest {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
