[jira] [Commented] (HDFS-15390) client fails forever when namenode ipaddr changed
[
https://issues.apache.org/jira/browse/HDFS-15390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127515#comment-17127515
]
Sean Chow commented on HDFS-15390:
--
Hi [~ayushtkn] , I've tried written a unit test for this but it's not easy :(
Because emulating namenode ipaddr change need the third namenode setup to
connect.
HDFS-4404 is good example, not for this issue.
> client fails forever when namenode ipaddr changed
> -
>
> Key: HDFS-15390
> URL: https://issues.apache.org/jira/browse/HDFS-15390
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 2.10.0, 2.9.2, 3.2.1
>Reporter: Sean Chow
>Priority: Major
> Attachments: HDFS-15390.01.patch
>
>
> For machine replacement, I replace my standby namenode with a new ipaddr and
> keep the same hostname. Also update the client's hosts to make it resolve
> correctly
> When I try to run failover to transite the new namenode(let's say nn2), the
> client will fail to read or write forever until it's restarted.
> That make yarn nodemanager in sick state. Even the new tasks will encounter
> this exception too. Until all nodemanager restart.
>
> {code:java}
> 20/06/02 15:12:25 WARN ipc.Client: Address change detected. Old:
> nn2-192-168-1-100/192.168.1.100:9000 New: nn2-192-168-1-100/192.168.1.200:9000
> 20/06/02 15:12:25 DEBUG ipc.Client: closing ipc connection to
> nn2-192-168-1-100/192.168.1.200:9000: Connection refused
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1517)
> at org.apache.hadoop.ipc.Client.call(Client.java:1440)
> at org.apache.hadoop.ipc.Client.call(Client.java:1401)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:193)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> {code}
>
> We can see the client has {{Address change detected}}, but it still fails. I
> find out that's because when method {{updateAddress()}} return true, the
> {{handleConnectionFailure()}} thow an exception that break the next retry
> with the right ipaddr.
> Client.java: setupConnection()
> {code:java}
> } catch (ConnectTimeoutException toe) {
> /* Check for an address change and update the local reference.
>* Reset the failure counter if the address was changed
>*/
> if (updateAddress()) {
> timeoutFailures = ioFailures = 0;
> }
> handleConnectionTimeout(timeoutFailures++,
> maxRetriesOnSocketTimeouts, toe);
> } catch (IOException ie) {
> if (updateAddress()) {
> timeoutFailures = ioFailures = 0;
> }
> // because the namenode ip changed in updateAddress(), the old namenode
> ipaddress cannot be accessed now
> // handleConnectionFailure will thow an exception, the next retry never have
> a chance to use the right server updated in updateAddress()
> handleConnectionFailure(ioFailures++, ie);
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15346) RBF: DistCpFedBalance implementation
[
https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127489#comment-17127489
]
Yiqun Lin edited comment on HDFS-15346 at 6/7/20, 4:35 AM:
---
Some more detailed review comments:
*HdfsConstants.java*
Can we rename DOT_SNAPSHOT_SEPARATOR_DIR to the more readable name
DOT_SNAPSHOT_DIR_SEPARATOR?
*DistCpFedBalance.java*
# It would good to print the fed context that created from input options, so
that we will know final options that we passed in.
{noformat}
+. // --> print fed balancer context
+ // Construct the balance job.
+ BalanceJob.Builder builder = new
BalanceJob.Builder<>();
+ DistCpProcedure dcp =
+ new DistCpProcedure(DISTCP_PROCEDURE, null, delayDuration, context);
+ builder.nextProcedure(dcp);
{noformat}
# We can replace this system out in LOG instance,
{noformat}
+for (BalanceJob job : jobs) {
+ if (!job.isJobDone()) {
+unfinished++;
+ }
+ System.out.println(job);
+}
{noformat}
*DistCpProcedure.java*
# The message in IOException(src + " doesn't exist.") not correctly described,
should be 'src + " should be the directory."'
# For each stage change, can we add aN additional output log, like this:
{noformat}
+if (srcFs.exists(new Path(src, HdfsConstants.DOT_SNAPSHOT_DIR))) {
+ throw new IOException(src + " shouldn't enable snapshot.");
+}
LOG.info("Stage updated from {} to {}.", stage.name(),
Stage.INIT_DISTCP.name())
+stage = Stage.INIT_DISTCP;
+ }
{noformat}
# Here we reset permission to 0, that means no any operation is allowed? Is
this expected, why not is 400 (only allow read)? The comment said that
'cancelling the x permission of the source path.' makes me confused.
{noformat}
srcFs.setPermission(src, FsPermission.createImmutable((short) 0));
{noformat}
# I prefer to throw IOException rather than doing delete operation in
cleanUpBeforeInitDistcp. cleanUpBeforeInitDistcp is expected to be the final
pre-check function before submitting ditcp job. And let admin users to check
and do delete operation manually by themself.
{noformat}
+ private void initialCheckBeforeInitDistcp() throws IOException {
+if (dstFs.exists(dst)) {
+ throw IOException();
+}
+srcFs.allowSnapshot(src);
+if (srcFs.exists(new Path(src,
+HdfsConstants.DOT_SNAPSHOT_SEPARATOR_DIR + CURRENT_SNAPSHOT_NAME))) {
throw IOException();
+}
{noformat}
*FedBalanceConfigs.java*
Can we move all keys from BalanceProcedureConfigKeys to this class? We don't
need two duplicated Config class. One follow-up task I am thinking that we can
have a separated config file something named fedbalance-default.xml for
fedbalance tool, like ditcp-default.xml for distcp tool now. I don't prefer to
add all tool config settings into hdfs-default.xml.
*FedBalanceContext.java*
Override the toString method in FedBalanceContext to help us know the input
options that actually be used.
*MountTableProcedure.java*
The for loop can just break once we find the first source path that matched.
{noformat}
+for (MountTable result : results) {
+ if (mount.equals(result.getSourcePath())) {
+ existingEntry = result;
break;
+ }
+}
{noformat}
*TrashProcedure.java*
{noformat}
+ /**
+ * Delete source path to trash.
+ */
+ void moveToTrash() throws IOException {
+Path src = context.getSrc();
+if (srcFs.exists(src)) {
+ switch (context.getTrashOpt()) {
+ case TRASH:
+conf.setFloat(FS_TRASH_INTERVAL_KEY, 1);
+if (!Trash.moveToAppropriateTrash(srcFs, src, conf)) {
+ throw new IOException("Failed move " + src + " to trash.");
+}
+break;
+ case DELETE:
+if (!srcFs.delete(src, true)) {
+ throw new IOException("Failed delete " + src);
+}
+LOG.info("{} is deleted.", src);
+break;
+ default:
+break;
+ }
+}
+ }
{noformat}
For above lines, two review comments:
# Can we add SKIP option check as well and throw unexpected option error?
{noformat}
case SKIP:
break;
+ default:
+ throw new IOException("Unexpected trash option=" +
context.getTrashOpt());
+ }
{noformat}
# FS_TRASH_INTERVAL_KEY defined with 1 is too small, that means we the trash
will be deleted after 1 minute. Can you increased this to 60? Also please add
necessary comment in trash option description to say the default trash behavior
when trash is disabled in server side and client side value will be used.
was (Author: linyiqun):
Some more detailed review comments:
*HdfsConstants.java*
Can we rename DOT_SNAPSHOT_SEPARATOR_DIR to the more readable name
DOT_SNAPSHOT_DIR_SEPARATOR?
*DistCpFedBalance.java*
# It would good to print the fed context that created from input options, so
that we will
[jira] [Commented] (HDFS-15346) RBF: DistCpFedBalance implementation
[
https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127498#comment-17127498
]
Yiqun Lin commented on HDFS-15346:
--
Review comments for unit tests:
*TestDistCpProcedure.java*
# Use CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY to replace
'fs.defaultFS'.
# In {{TestDistCpProcedure#testSuccessfulDistCpProcedure}}, can we add
additional file length check between src file and dst file?
# Please complete the javadoc comment for method executeProcedure and
createFiles.
# Method sede can update to a more readable name serializeProcedure.
# I think we missing a corner case test case that disable writer behavior in
non-RBF mode.
# The test need a little long time to execute the whole test.
>From Jenkins test result:
{noformat}
testDiffDistCp 1 min 18 secPassed
testInitDistCp 22 sec Passed
testRecoveryByStage 55 sec Passed
testShutdown8.9 sec Passed
testStageFinalDistCp47 sec Passed
testStageFinish 0.22 secPassed
testSuccessfulDistCpProcedure 38 sec Passed
{noformat}
Can we look into why some ut spend so many time? Increasing timeout value is a
quick-fix way but not the best way.
*TestMountTableProcedure.java*
Please update testSeDe to testSeDeserialize
*TestTrashProcedure.java*
Can we also add a test method testSeDeserialize like TestMountTableProcedure
does?
> RBF: DistCpFedBalance implementation
>
>
> Key: HDFS-15346
> URL: https://issues.apache.org/jira/browse/HDFS-15346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15346.001.patch, HDFS-15346.002.patch,
> HDFS-15346.003.patch, HDFS-15346.004.patch, HDFS-15346.005.patch,
> HDFS-15346.006.patch, HDFS-15346.007.patch
>
>
> Patch in HDFS-15294 is too big to review so we split it into 2 patches. This
> is the second one. Detail can be found at HDFS-15294.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15346) RBF: DistCpFedBalance implementation
[
https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127489#comment-17127489
]
Yiqun Lin commented on HDFS-15346:
--
Some more detailed review comments:
*HdfsConstants.java*
Can we rename DOT_SNAPSHOT_SEPARATOR_DIR to the more readable name
DOT_SNAPSHOT_DIR_SEPARATOR?
*DistCpFedBalance.java*
# It would good to print the fed context that created from input options, so
that we will know final options that we passed in.
{noformat}
+. // --> print fed balancer context
+ // Construct the balance job.
+ BalanceJob.Builder builder = new
BalanceJob.Builder<>();
+ DistCpProcedure dcp =
+ new DistCpProcedure(DISTCP_PROCEDURE, null, delayDuration, context);
+ builder.nextProcedure(dcp);
{noformat}
# We can replace this system out in LOG instance,
{noformat}
+for (BalanceJob job : jobs) {
+ if (!job.isJobDone()) {
+unfinished++;
+ }
+ System.out.println(job);
+}
{noformat}
*DistCpProcedure.java*
# The message in IOException(src + " doesn't exist.") not correctly described,
should be 'src + " should be the directory."'
# For each stage change, can we add aN additional output log, like this:
{noformat}
+if (srcFs.exists(new Path(src, HdfsConstants.DOT_SNAPSHOT_DIR))) {
+ throw new IOException(src + " shouldn't enable snapshot.");
+}
LOG.info("Stage updated from {} to {}.", stage.name(),
Stage.INIT_DISTCP.name())
+stage = Stage.INIT_DISTCP;
+ }
{noformat}
# Here we reset permission to 0, that means no any operation is allowed? Is
this expected, why not is 400 (only allow read)? The comment said that
'cancelling the x permission of the source path.' makes me confused.
{noformat}
srcFs.setPermission(src, FsPermission.createImmutable((short) 0));
{noformat}
# I prefer to throw IOException rather than doing delete operation in
cleanUpBeforeInitDistcp. cleanUpBeforeInitDistcp is expected to be the final
pre-check function before submit ditcp job.
{noformat}
+ private void initialCheckBeforeInitDistcp() throws IOException {
+if (dstFs.exists(dst)) {
+ throw IOException();
+}
+srcFs.allowSnapshot(src);
+if (srcFs.exists(new Path(src,
+HdfsConstants.DOT_SNAPSHOT_SEPARATOR_DIR + CURRENT_SNAPSHOT_NAME))) {
throw IOException();
+}
{noformat}
*FedBalanceConfigs.java*
Can we move all keys from BalanceProcedureConfigKeys to this class? We don't
need two duplicated Config class. One follow-up task I am thinking that we can
have a separated config file something named fedbalance-default.xml for
fedbalance tool, like ditcp-default.xml for distcp tool now. I don't prefer to
add all tool config settings into hdfs-default.xml.
*FedBalanceContext.java*
Override the toString method in FedBalanceContext to help us know the input
options that actually be used.
*MountTableProcedure.java*
The for loop can just break once we find the first source path that matched.
{noformat}
+for (MountTable result : results) {
+ if (mount.equals(result.getSourcePath())) {
+ existingEntry = result;
break;
+ }
+}
{noformat}
*TrashProcedure.java*
{noformat}
+ /**
+ * Delete source path to trash.
+ */
+ void moveToTrash() throws IOException {
+Path src = context.getSrc();
+if (srcFs.exists(src)) {
+ switch (context.getTrashOpt()) {
+ case TRASH:
+conf.setFloat(FS_TRASH_INTERVAL_KEY, 1);
+if (!Trash.moveToAppropriateTrash(srcFs, src, conf)) {
+ throw new IOException("Failed move " + src + " to trash.");
+}
+break;
+ case DELETE:
+if (!srcFs.delete(src, true)) {
+ throw new IOException("Failed delete " + src);
+}
+LOG.info("{} is deleted.", src);
+break;
+ default:
+break;
+ }
+}
+ }
{noformat}
For above lines, two review comments:
# Can we add SKIP option check as well and throw unexpected option error?
{noformat}
case SKIP:
break;
+ default:
+ throw new IOException("Unexpected trash option=" +
context.getTrashOpt());
+ }
{noformat}
# FS_TRASH_INTERVAL_KEY defined with 1 is too small, that means we the trash
will be deleted after 1 minute. Can you increased this to 60? Also please add
necessary comment in trash option description to say the default trash behavior
when trash is disabled in server side and client side value will be used.
> RBF: DistCpFedBalance implementation
>
>
> Key: HDFS-15346
> URL: https://issues.apache.org/jira/browse/HDFS-15346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-15346.001.patch, HDFS-15346.002.patch,
>
[jira] [Updated] (HDFS-15394) Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.
[ https://issues.apache.org/jira/browse/HDFS-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15394: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thanks [~ayushtkn] for reviews! I have committed this to trunk. > Add all available fs.viewfs.overload.scheme.target..impl classes in > core-default.xml bydefault. > --- > > Key: HDFS-15394 > URL: https://issues.apache.org/jira/browse/HDFS-15394 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Fix For: 3.4.0 > > > This proposes to add all available > fs.viewfs.overload.scheme.target..impl classes in core-default.xml. > So, that users need not configure them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15372) Files in snapshots no longer see attribute provider permissions
[ https://issues.apache.org/jira/browse/HDFS-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127418#comment-17127418 ] Stephen O'Donnell edited comment on HDFS-15372 at 6/6/20, 6:46 PM: --- This Jira is only relevant when an attribute provider like Sentry is in place. For normal ACLs add by "setfacl" etc, the snapshot ACLs are correct. To be clear, there are two type of permissions / acls. Those stored in HDFS and those provided by something like Sentry which are never persisted into the inode object. They are overlayed on top of it, hiding the underlying HDFS permissions / ACLs. Take this path for example "/data/tab1" and a snasphot of it "/data/.snapshot/snap1/tab1". Pre Hadoop 3.0 (I think, as I am comparing trunk with CDH 5), the attribute provider actually received an inode. Now on trunk the attribute provider receives a list of path components, which is basically a list of each directory in the path as a string. Pre Hadoop 3.0, the attribute provider simply called "inode.getFullPath" on the inode it received. If you ask for the permissions of the components of the snapshot path above, these are the paths the attribute provider sees: /data -> provider sees /data /data/.snapshot -> provider does not see this and an dummy permission object is returned. /data/.snapshot/snap1 -> provider sees /data/.snapshot/snap1 /data/.snapshot/snap1/tab1 -> provider sees /data/tab1, as calling getFullPath() on the inode returns this value. Now on trunk, if you take this same set of paths, this is what the provider sees: /data -> provider see [/data] /data/.snapshot -> provider does not see this and an dummy permission object is returned. /data/.snapshot/snap1 -> provider see [/data, .snapshot/snap1] - note that the .snapshot dir is not separated from the snapshot name here. /data/.snapshot/snap1/tab1 -> provider sees [/data, .snapshot/snap1, tab1] This means that if something like Sentry provides ACLs on /data/tab1, then when Sentry checks the snapshot path it sees /data/.snapshot/snap1/tab1 and it does not give the ACLs to it. In CDH 5, the snapshot path looks the same as the "live" path to the provider, and Sentry returns the same ACLs on the snapshot as for the live path. I believe this change (what the provider sees) is a regression and not an intentional change. The idea of this patch is to ensure the provider sees the same as it did before. The patch I have posted, for the examples above would see: /data -> provider see [/data] /data/.snapshot -> provider does not see this and an dummy permission object is returned. /data/.snapshot/snap1 -> provider sees [/data, snap1] - This is a problem with my patch, it should see [/data, .snapshot/snap1]. I need to figure out how to fix this. /data/.snapshot/snap1/tab1 -> provider sees [[/data, tab1] was (Author: sodonnell): This Jira is only relevant when an attribute provider like Sentry is in place. For normal ACLs add by "setfacl" etc, the snapshot ACLs are correct. Take this path for example "/data/tab1" and a snasphot of it "/data/.snapshot/snap1/tab1". Pre Hadoop 3.0 (I think, as I am comparing trunk with CDH 5), the attribute provider actually received an inode. Now on trunk the attribute provider receives a list of path components, which is basically a list of each directory in the path as a string. Pre Hadoop 3.0, the attribute provider simply called "inode.getFullPath" on the inode it received. If you ask for the permissions of the components of the snapshot path above, these are the paths the attribute provider sees: /data -> provider sees /data /data/.snapshot -> provider does not see this and an dummy permission object is returned. /data/.snapshot/snap1 -> provider sees /data/.snapshot/snap1 /data/.snapshot/snap1/tab1 -> provider sees /data/tab1, as calling getFullPath() on the inode returns this value. Now on trunk, if you take this same set of paths, this is what the provider sees: /data -> provider see [/data] /data/.snapshot -> provider does not see this and an dummy permission object is returned. /data/.snapshot/snap1 -> provider see [/data, .snapshot/snap1] - note that the .snapshot dir is not separated from the snapshot name here. /data/.snapshot/snap1/tab1 -> provider sees [/data, .snapshot/snap1, tab1] This means that if something like Sentry provides ACLs on /data/tab1, then when Sentry checks the snapshot path it sees /data/.snapshot/snap1/tab1 and it does not give the ACLs to it. In CDH 5, the snapshot path looks the same as the "live" path to the provider, and Sentry returns the same ACLs on the snapshot as for the live path. I believe this change (what the provider sees) is a regression and not an intentional change. The idea of this patch is to ensure the provider sees the same as it did before. The patch I have posted, for the examples above would see: /data -> provider see
[jira] [Commented] (HDFS-15372) Files in snapshots no longer see attribute provider permissions
[
https://issues.apache.org/jira/browse/HDFS-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127418#comment-17127418
]
Stephen O'Donnell commented on HDFS-15372:
--
This Jira is only relevant when an attribute provider like Sentry is in place.
For normal ACLs add by "setfacl" etc, the snapshot ACLs are correct.
Take this path for example "/data/tab1" and a snasphot of it
"/data/.snapshot/snap1/tab1".
Pre Hadoop 3.0 (I think, as I am comparing trunk with CDH 5), the attribute
provider actually received an inode. Now on trunk the attribute provider
receives a list of path components, which is basically a list of each directory
in the path as a string.
Pre Hadoop 3.0, the attribute provider simply called "inode.getFullPath" on the
inode it received. If you ask for the permissions of the components of the
snapshot path above, these are the paths the attribute provider sees:
/data -> provider sees /data
/data/.snapshot -> provider does not see this and an dummy permission object is
returned.
/data/.snapshot/snap1 -> provider sees /data/.snapshot/snap1
/data/.snapshot/snap1/tab1 -> provider sees /data/tab1, as calling
getFullPath() on the inode returns this value.
Now on trunk, if you take this same set of paths, this is what the provider
sees:
/data -> provider see [/data]
/data/.snapshot -> provider does not see this and an dummy permission object is
returned.
/data/.snapshot/snap1 -> provider see [/data, .snapshot/snap1] - note that the
.snapshot dir is not separated from the snapshot name here.
/data/.snapshot/snap1/tab1 -> provider sees [/data, .snapshot/snap1, tab1]
This means that if something like Sentry provides ACLs on /data/tab1, then when
Sentry checks the snapshot path it sees /data/.snapshot/snap1/tab1 and it does
not give the ACLs to it. In CDH 5, the snapshot path looks the same as the
"live" path to the provider, and Sentry returns the same ACLs on the snapshot
as for the live path.
I believe this change (what the provider sees) is a regression and not an
intentional change. The idea of this patch is to ensure the provider sees the
same as it did before.
The patch I have posted, for the examples above would see:
/data -> provider see [/data]
/data/.snapshot -> provider does not see this and an dummy permission object is
returned.
/data/.snapshot/snap1 -> provider sees [/data, snap1] - This is a problem with
my patch, it should see [/data, .snapshot/snap1]. I need to figure out how to
fix this.
/data/.snapshot/snap1/tab1 -> provider sees [[/data, tab1]
> Files in snapshots no longer see attribute provider permissions
> ---
>
> Key: HDFS-15372
> URL: https://issues.apache.org/jira/browse/HDFS-15372
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15372.001.patch
>
>
> Given a cluster with an authorization provider configured (eg Sentry) and the
> paths covered by the provider are snapshotable, there was a change in
> behaviour in how the provider permissions and ACLs are applied to files in
> snapshots between the 2.x branch and Hadoop 3.0.
> Eg, if we have the snapshotable path /data, which is Sentry managed. The ACLs
> below are provided by Sentry:
> {code}
> hadoop fs -getfacl -R /data
> # file: /data
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::---
> group:flume:rwx
> user:hive:rwx
> group:hive:rwx
> group:testgroup:rwx
> mask::rwx
> other::--x
> /data/tab1
> {code}
> After taking a snapshot, the files in the snapshot do not see the provider
> permissions:
> {code}
> hadoop fs -getfacl -R /data/.snapshot
> # file: /data/.snapshot
> # owner:
> # group:
> user::rwx
> group::rwx
> other::rwx
> # file: /data/.snapshot/snap1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/.snapshot/snap1/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> {code}
> However pre-Hadoop 3.0 (when the attribute provider etc was extensively
> refactored) snapshots did get the provider permissions.
> The reason is this code in FSDirectory.java which ultimately calls the
> attribute provider and passes the path we want permissions for:
> {code}
> INodeAttributes getAttributes(INodesInPath iip)
> throws IOException {
> INode node = FSDirectory.resolveLastINode(iip);
> int snapshot = iip.getPathSnapshotId();
> INodeAttributes nodeAttrs = node.getSnapshotINode(snapshot);
> UserGroupInformation ugi = NameNode.getRemoteUser();
> INodeAttributeProvider ap = this.getUserFilteredAttributeProvider(ugi);
> if (ap != null) {
> // permission checking sends the full components
[jira] [Commented] (HDFS-15351) Blocks Scheduled Count was wrong on Truncate
[ https://issues.apache.org/jira/browse/HDFS-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127413#comment-17127413 ] hemanthboyina commented on HDFS-15351: -- thanks [~belugabehr] for the review > Blocks Scheduled Count was wrong on Truncate > - > > Key: HDFS-15351 > URL: https://issues.apache.org/jira/browse/HDFS-15351 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15351.001.patch, HDFS-15351.002.patch, > HDFS-15351.003.patch > > > On truncate and append we remove the blocks from Reconstruction Queue > On removing the blocks from pending reconstruction , we need to decrement > Blocks Scheduled -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15372) Files in snapshots no longer see attribute provider permissions
[
https://issues.apache.org/jira/browse/HDFS-15372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127412#comment-17127412
]
hemanthboyina commented on HDFS-15372:
--
thanks for good analysis [~sodonnell]
{quote}With the 001 patch in place, if you try to list
/data/.snapshot/snapshot_1, the path seen by the attribute provider is:
/user/snapshot_1
Before, it was:
/user/.snapshot/snapshot1
When checking a path like /data/.snapshot/snap1 the provider will see
/data/snap1, but on the branch-2, it would have seen /data/.snapshot/snap1.
{quote}
is the path seen by the attribute provider for branch and trunk was same ? it
was bit confusing , can you add all in one comment with an example for a
snapshot path
If we try list for a path , the path will be resolved as Inodes from
InodeInPath , and the same inodes components will be used by the provider right
? and INodesInPath handles .snapshot part of a path
While creating a snapshot we add the inode directory as the root to snapshot
{code:java}
DirectorySnapshottableFeature#createSnaphot
public Snapshot addSnapshot(INodeDirectory snapshotRoot, int id, String name,
final Snapshot s = new Snapshot(id, name, snapshotRoot); {code}
While getting inodesInPath for a file in snapshot we use the root of snapshot
to get the file , IMO that means the if the file has an acl the file under
snapshot root should have acl
{code:java}
if (isDotSnapshotDir(childName) && dir.isSnapshottable()) {
final Snapshot s = dir.getSnapshot(components[count + 1]);
else {
curNode = s.getRoot();
snapshotId = s.getId();
} {code}
please correct me if am missing some thing here
> Files in snapshots no longer see attribute provider permissions
> ---
>
> Key: HDFS-15372
> URL: https://issues.apache.org/jira/browse/HDFS-15372
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
> Attachments: HDFS-15372.001.patch
>
>
> Given a cluster with an authorization provider configured (eg Sentry) and the
> paths covered by the provider are snapshotable, there was a change in
> behaviour in how the provider permissions and ACLs are applied to files in
> snapshots between the 2.x branch and Hadoop 3.0.
> Eg, if we have the snapshotable path /data, which is Sentry managed. The ACLs
> below are provided by Sentry:
> {code}
> hadoop fs -getfacl -R /data
> # file: /data
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::---
> group:flume:rwx
> user:hive:rwx
> group:hive:rwx
> group:testgroup:rwx
> mask::rwx
> other::--x
> /data/tab1
> {code}
> After taking a snapshot, the files in the snapshot do not see the provider
> permissions:
> {code}
> hadoop fs -getfacl -R /data/.snapshot
> # file: /data/.snapshot
> # owner:
> # group:
> user::rwx
> group::rwx
> other::rwx
> # file: /data/.snapshot/snap1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> # file: /data/.snapshot/snap1/tab1
> # owner: hive
> # group: hive
> user::rwx
> group::rwx
> other::--x
> {code}
> However pre-Hadoop 3.0 (when the attribute provider etc was extensively
> refactored) snapshots did get the provider permissions.
> The reason is this code in FSDirectory.java which ultimately calls the
> attribute provider and passes the path we want permissions for:
> {code}
> INodeAttributes getAttributes(INodesInPath iip)
> throws IOException {
> INode node = FSDirectory.resolveLastINode(iip);
> int snapshot = iip.getPathSnapshotId();
> INodeAttributes nodeAttrs = node.getSnapshotINode(snapshot);
> UserGroupInformation ugi = NameNode.getRemoteUser();
> INodeAttributeProvider ap = this.getUserFilteredAttributeProvider(ugi);
> if (ap != null) {
> // permission checking sends the full components array including the
> // first empty component for the root. however file status
> // related calls are expected to strip out the root component according
> // to TestINodeAttributeProvider.
> byte[][] components = iip.getPathComponents();
> components = Arrays.copyOfRange(components, 1, components.length);
> nodeAttrs = ap.getAttributes(components, nodeAttrs);
> }
> return nodeAttrs;
> }
> {code}
> The line:
> {code}
> INode node = FSDirectory.resolveLastINode(iip);
> {code}
> Picks the last resolved Inode and if you then call node.getPathComponents,
> for a path like '/data/.snapshot/snap1/tab1' it will return /data/tab1. It
> resolves the snapshot path to its original location, but its still the
> snapshot inode.
> However the logic passes 'iip.getPathComponents' which returns
> "/user/.snapshot/snap1/tab" to the provider.
>
[jira] [Commented] (HDFS-12453) TestDataNodeHotSwapVolumes fails in trunk Jenkins runs
[ https://issues.apache.org/jira/browse/HDFS-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127125#comment-17127125 ] Jim Brennan commented on HDFS-12453: [~eddyxu], [~arp], [~kihwal] we have seen this failure in our internal branch-2.10 builds. The original patch applies cleanly, but there were a few issues that I needed to change for branch-2.10 * Convert the stream() to a StringBuffer for building the list of newDirs (java7) * Don't use a lambda for the thread. (java7) * Change volume.getStorageLocation() to volume.getBasePath(). I've put up a patch for branch-2.10 with these changes. Please let me know if it would be better to open a new Jira to pull this back to branch-2.10. > TestDataNodeHotSwapVolumes fails in trunk Jenkins runs > -- > > Key: HDFS-12453 > URL: https://issues.apache.org/jira/browse/HDFS-12453 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Arpit Agarwal >Assignee: Lei (Eddy) Xu >Priority: Critical > Labels: flaky-test > Fix For: 3.0.0 > > Attachments: HDFS-12453-branch-2.10.001.patch, HDFS-12453.00.patch, > HDFS-12453.01.patch, TestLogs.txt > > > TestDataNodeHotSwapVolumes fails occasionally with the following error (see > comment). Ran it ~10 times locally and it passed every time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15391) Standby NameNode due loads the corruption edit log, the service exits and cannot be restarted
[
https://issues.apache.org/jira/browse/HDFS-15391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huhaiyang updated HDFS-15391:
-
Summary: Standby NameNode due loads the corruption edit log, the service
exits and cannot be restarted (was: Standby NameNode cannot load the edit log
correctly due to edit log corruption, resulting in the service exiting
abnormally and unable to restart)
> Standby NameNode due loads the corruption edit log, the service exits and
> cannot be restarted
> -
>
> Key: HDFS-15391
> URL: https://issues.apache.org/jira/browse/HDFS-15391
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0
>Reporter: huhaiyang
>Priority: Critical
>
> In the cluster version 3.2.0 production environment，
> We found that due to edit log corruption, Standby NameNode could not
> properly load the Ediltog log, result in abnormal exit of the service and
> failure to restart
> {noformat}
> The specific scenario is that Flink writes to HDFS(replication file), and in
> the case of an exception to the write file, the following operations are
> performed :
> 1.close file
> 2.open file
> 3.truncate file
> 4.append file
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[ https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15289: --- Attachment: ViewFSOverloadScheme.png > Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table > - > > Key: HDFS-15289 > URL: https://issues.apache.org/jira/browse/HDFS-15289 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 3.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png > > > ViewFS provides flexibility to mount different filesystem types with mount > points configuration table. Additionally viewFS provides flexibility to > configure any fs (not only HDFS) scheme in mount table mapping. This approach > is solving the scalability problems, but users need to reconfigure the > filesystem to ViewFS and to its scheme. This will be problematic in the case > of paths persisted in meta stores, ex: Hive. In systems like Hive, it will > store uris in meta store. So, changing the file system scheme will create a > burden to upgrade/recreate meta stores. In our experience many users are not > ready to change that. > Router based federation is another implementation to provide coordinated > mount points for HDFS federation clusters. Even though this provides > flexibility to handle mount points easily, this will not allow > other(non-HDFS) file systems to mount. So, this does not solve the purpose > when users want to mount external(non-HDFS) filesystems. > So, the problem here is: Even though many users want to adapt to the scalable > fs options available, technical challenges of changing schemes (ex: in meta > stores) in deployments are obstructing them. > So, we propose to allow hdfs scheme in ViewFS like client side mount system > and provision user to create mount links without changing URI paths. > I will upload detailed design doc shortly. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12453) TestDataNodeHotSwapVolumes fails in trunk Jenkins runs
[ https://issues.apache.org/jira/browse/HDFS-12453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated HDFS-12453: --- Attachment: HDFS-12453-branch-2.10.001.patch > TestDataNodeHotSwapVolumes fails in trunk Jenkins runs > -- > > Key: HDFS-12453 > URL: https://issues.apache.org/jira/browse/HDFS-12453 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Reporter: Arpit Agarwal >Assignee: Lei (Eddy) Xu >Priority: Critical > Labels: flaky-test > Fix For: 3.0.0 > > Attachments: HDFS-12453-branch-2.10.001.patch, HDFS-12453.00.patch, > HDFS-12453.01.patch, TestLogs.txt > > > TestDataNodeHotSwapVolumes fails occasionally with the following error (see > comment). Ran it ~10 times locally and it passed every time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-15184) Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned
[
https://issues.apache.org/jira/browse/HDFS-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
任建亭 resolved HDFS-15184.
Resolution: Not A Problem
> Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
> (make) on project hadoop-hdfs-native-client: An Ant BuildException has
> occured: exec returned: 1
>
>
> Key: HDFS-15184
> URL: https://issues.apache.org/jira/browse/HDFS-15184
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.1
> Environment: windows 10
> JDK 1.8
> maven3.6.1
> ProtocolBuffer 2.5.0
> CMake 3.1.3
> git 2.25.0
> zlib 1.2.5
> Visual Studio 2010 Professional
>Reporter: 任建亭
>Priority: Major
> Fix For: 3.2.1
>
>
> When I build hadoop 3.2.1 on windows10, it failed. My command is 'mvn clean
> package -Pdist,native-win -DskipTests -Dtar'.
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part ... dir="D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
> executable="msbuild">... @ 9:122 in
> D:\h3s\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15394) Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.
[ https://issues.apache.org/jira/browse/HDFS-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-15394: --- Status: Patch Available (was: Open) > Add all available fs.viewfs.overload.scheme.target..impl classes in > core-default.xml bydefault. > --- > > Key: HDFS-15394 > URL: https://issues.apache.org/jira/browse/HDFS-15394 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This proposes to add all available > fs.viewfs.overload.scheme.target..impl classes in core-default.xml. > So, that users need not configure them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15395) The exception message of "DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY" is not precise
Yuanliang Zhang created HDFS-15395:
--
Summary: The exception message of
"DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY" is not precise
Key: HDFS-15395
URL: https://issues.apache.org/jira/browse/HDFS-15395
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Yuanliang Zhang
The exception message of "DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY" in DFSUtil.java
may not be precise.
The current message is: "Incorrect configuration: namenode address
{color:#FF}DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY *or*
DFS_NAMENODE_RPC_ADDRESS_KEY is not configured{color}"
{code:java}
public static Map>
getNNServiceRpcAddresses(
Configuration conf) throws IOException {
...
Map> addressList =
DFSUtilClient.getAddresses(conf, defaultAddress,
DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY,
DFS_NAMENODE_RPC_ADDRESS_KEY);
if (addressList.isEmpty()) {
throw new IOException("Incorrect configuration: namenode address "
+ DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY + " or "
+ DFS_NAMENODE_RPC_ADDRESS_KEY
+ " is not configured.");
}
return addressList;
}
{code}
However, from the doc:
{quote}If the value of this property (dfs.namenode.servicerpc-address) is unset
the value of dfs.namenode.rpc-address will be used as the default.
{quote}
The code in NameNode.java also confirm this logic. So I think this message may
need to refined to be consistent with the doc and code logic that the
dfs.namenode.rpc-address should always be set.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15394) Add all available fs.viewfs.overload.scheme.target..impl classes in core-default.xml bydefault.
[ https://issues.apache.org/jira/browse/HDFS-15394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127375#comment-17127375 ] Hudson commented on HDFS-15394: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18335 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18335/]) HDFS-15394. Add all available (github: rev 3ca15292c5584ec220b3eeaf76da85d228bcbd8b) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemOverloadSchemeWithHdfsScheme.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml > Add all available fs.viewfs.overload.scheme.target..impl classes in > core-default.xml bydefault. > --- > > Key: HDFS-15394 > URL: https://issues.apache.org/jira/browse/HDFS-15394 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: configuration, viewfs, viewfsOverloadScheme >Affects Versions: 3.2.1 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > > This proposes to add all available > fs.viewfs.overload.scheme.target..impl classes in core-default.xml. > So, that users need not configure them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15389) DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15389:
---
Affects Version/s: 3.2.1
> DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should
> work with ViewFSOverloadScheme
> --
>
> Key: HDFS-15389
> URL: https://issues.apache.org/jira/browse/HDFS-15389
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.2.1
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15389-01.patch
>
>
> Two Issues Here :
> Firstly Prior to HDFS-15321, When DFSAdmin was closed the FileSystem
> associated with it was closed as part of close method, But post HDFS-15321,
> the {{FileSystem}} isn't stored as part of {{FsShell}}, hence during close,
> the FileSystem still stays and isn't close.
> * This is the reason for failure of TestDFSHAAdmin
> Second : {{DfsAdmin -setBalancerBandwidth}} doesn't work with
> {{ViewFSOverloadScheme}} since the setBalancerBandwidth calls {{getFS()}}
> rather than {{getDFS()}} which resolves the scheme in {{HDFS-15321}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15389) DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15389:
---
Fix Version/s: 3.4.0
> DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should
> work with ViewFSOverloadScheme
> --
>
> Key: HDFS-15389
> URL: https://issues.apache.org/jira/browse/HDFS-15389
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.2.1
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15389-01.patch
>
>
> Two Issues Here :
> Firstly Prior to HDFS-15321, When DFSAdmin was closed the FileSystem
> associated with it was closed as part of close method, But post HDFS-15321,
> the {{FileSystem}} isn't stored as part of {{FsShell}}, hence during close,
> the FileSystem still stays and isn't close.
> * This is the reason for failure of TestDFSHAAdmin
> Second : {{DfsAdmin -setBalancerBandwidth}} doesn't work with
> {{ViewFSOverloadScheme}} since the setBalancerBandwidth calls {{getFS()}}
> rather than {{getDFS()}} which resolves the scheme in {{HDFS-15321}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15396) Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
[
https://issues.apache.org/jira/browse/HDFS-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127347#comment-17127347
]
Ayush Saxena commented on HDFS-15396:
-
Raised PR : https://github.com/apache/hadoop/pull/2059
As of now modified the test to change the mount entry. A possible alternative
could be to be handle empty string in {{listStatus}} and resolve it to "/" or
home directory but am not sure, doing that would be good...
> Fix
> TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
>
>
> Key: HDFS-15396
> URL: https://issues.apache.org/jira/browse/HDFS-15396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> Exception :
> {code:java}
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:172)
> at org.apache.hadoop.fs.Path.(Path.java:184)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.listStatus(ViewFileSystem.java:1207)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:514)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.assertListStatusFinds(FileSystemContractBaseTest.java:867)
> at
> org.apache.hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract.testListStatusRootDir(TestViewFileSystemOverloadSchemeHdfsFileSystemContract.java:119)
> {code}
> The reason for failure being, the mount destination for /user and /append in
> the test is just the URI, with no further path.
> Thus while listing, in order to fetch the permissions, the destination URI is
> used to get the path, which resolves being empty. Hence the failure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15389) DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15389:
---
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thank you [~ayushtkn] for your contribution! I have committed it to trunk.
> DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should
> work with ViewFSOverloadScheme
> --
>
> Key: HDFS-15389
> URL: https://issues.apache.org/jira/browse/HDFS-15389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfsadmin, viewfsOverloadScheme
>Affects Versions: 3.2.1
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15389-01.patch
>
>
> Two Issues Here :
> Firstly Prior to HDFS-15321, When DFSAdmin was closed the FileSystem
> associated with it was closed as part of close method, But post HDFS-15321,
> the {{FileSystem}} isn't stored as part of {{FsShell}}, hence during close,
> the FileSystem still stays and isn't close.
> * This is the reason for failure of TestDFSHAAdmin
> Second : {{DfsAdmin -setBalancerBandwidth}} doesn't work with
> {{ViewFSOverloadScheme}} since the setBalancerBandwidth calls {{getFS()}}
> rather than {{getDFS()}} which resolves the scheme in {{HDFS-15321}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15396) Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
[
https://issues.apache.org/jira/browse/HDFS-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15396:
Status: Patch Available (was: Open)
> Fix
> TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
>
>
> Key: HDFS-15396
> URL: https://issues.apache.org/jira/browse/HDFS-15396
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>
> Exception :
> {code:java}
> java.lang.IllegalArgumentException: Can not create a Path from an empty string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:172)
> at org.apache.hadoop.fs.Path.(Path.java:184)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.listStatus(ViewFileSystem.java:1207)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:514)
> at
> org.apache.hadoop.fs.FileSystemContractBaseTest.assertListStatusFinds(FileSystemContractBaseTest.java:867)
> at
> org.apache.hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract.testListStatusRootDir(TestViewFileSystemOverloadSchemeHdfsFileSystemContract.java:119)
> {code}
> The reason for failure being, the mount destination for /user and /append in
> the test is just the URI, with no further path.
> Thus while listing, in order to fetch the permissions, the destination URI is
> used to get the path, which resolves being empty. Hence the failure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15389) DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15389:
---
Component/s: viewfsOverloadScheme
dfsadmin
> DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should
> work with ViewFSOverloadScheme
> --
>
> Key: HDFS-15389
> URL: https://issues.apache.org/jira/browse/HDFS-15389
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfsadmin, viewfsOverloadScheme
>Affects Versions: 3.2.1
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15389-01.patch
>
>
> Two Issues Here :
> Firstly Prior to HDFS-15321, When DFSAdmin was closed the FileSystem
> associated with it was closed as part of close method, But post HDFS-15321,
> the {{FileSystem}} isn't stored as part of {{FsShell}}, hence during close,
> the FileSystem still stays and isn't close.
> * This is the reason for failure of TestDFSHAAdmin
> Second : {{DfsAdmin -setBalancerBandwidth}} doesn't work with
> {{ViewFSOverloadScheme}} since the setBalancerBandwidth calls {{getFS()}}
> rather than {{getDFS()}} which resolves the scheme in {{HDFS-15321}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15389) DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme
[
https://issues.apache.org/jira/browse/HDFS-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127241#comment-17127241
]
Hudson commented on HDFS-15389:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18334 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18334/])
HDFS-15389. DFSAdmin should close filesystem and dfsadmin (github: rev
cc671b16f7b0b7c1ed7b41b96171653dc43cf670)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFileSystemOverloadSchemeWithDFSAdmin.java
> DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should
> work with ViewFSOverloadScheme
> --
>
> Key: HDFS-15389
> URL: https://issues.apache.org/jira/browse/HDFS-15389
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15389-01.patch
>
>
> Two Issues Here :
> Firstly Prior to HDFS-15321, When DFSAdmin was closed the FileSystem
> associated with it was closed as part of close method, But post HDFS-15321,
> the {{FileSystem}} isn't stored as part of {{FsShell}}, hence during close,
> the FileSystem still stays and isn't close.
> * This is the reason for failure of TestDFSHAAdmin
> Second : {{DfsAdmin -setBalancerBandwidth}} doesn't work with
> {{ViewFSOverloadScheme}} since the setBalancerBandwidth calls {{getFS()}}
> rather than {{getDFS()}} which resolves the scheme in {{HDFS-15321}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15396) Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
Ayush Saxena created HDFS-15396:
---
Summary: Fix
TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir
Key: HDFS-15396
URL: https://issues.apache.org/jira/browse/HDFS-15396
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena
Exception :
{code:java}
java.lang.IllegalArgumentException: Can not create a Path from an empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:172)
at org.apache.hadoop.fs.Path.(Path.java:184)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.listStatus(ViewFileSystem.java:1207)
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:514)
at
org.apache.hadoop.fs.FileSystemContractBaseTest.assertListStatusFinds(FileSystemContractBaseTest.java:867)
at
org.apache.hadoop.fs.viewfs.TestViewFileSystemOverloadSchemeHdfsFileSystemContract.testListStatusRootDir(TestViewFileSystemOverloadSchemeHdfsFileSystemContract.java:119)
{code}
The reason for failure being, the mount destination for /user and /append in
the test is just the URI, with no further path.
Thus while listing, in order to fetch the permissions, the destination URI is
used to get the path, which resolves being empty. Hence the failure
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
