[jira] [Commented] (HADOOP-18117) Add an option to preserve root directory permissions
[ https://issues.apache.org/jira/browse/HADOOP-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489432#comment-17489432 ] Hongbing Wang commented on HADOOP-18117: Thanks for considering this issue. In fact, we also faced this problem and solved it in our internal version as -HADOOP-15211- pointed out. Some discussion: 1. Is it better not to add extra parameters? 2. If add parameters, `updateRootDirectoryAttributes` may be too complicated. > Add an option to preserve root directory permissions > > > Key: HADOOP-18117 > URL: https://issues.apache.org/jira/browse/HADOOP-18117 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Mohanad Elsafty >Priority: Minor > > As mentioned in https://issues.apache.org/jira/browse/HADOOP-15211 > > If *-update* or *-overwrite* is being passed when *distcp* used, the root > directory will be skipped in two occasions (CopyListing#doBuildListing & > CopyCommitter#preserveFileAttributesForDirectories), which will ignore root > directory's attributes. > > We face the same issue when distcp huge data between clusters and it takes > too much effort to update root directories attributes manually. > > From the earlier ticket it's obvious why this behaviour is there, but > sometime we need to enforce root directory update hence I will add a new > option for distcp to enable someone (who understands the need of this and > know what they are doing) to enforce the update of root directory's > attributes (permissions, ownership, ...) > > It should be simple one, something like this > {code:java} > $ hadoop distcp -p -update -updateRootDirectoryAttributes /a/b/c /a/b/d {code} > This behaviour is optional and will be *false* by default. (it should not > affect existing *distcp* users). -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17087) Add EC flag to stat commands
[ https://issues.apache.org/jira/browse/HADOOP-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143785#comment-17143785 ] Hongbing Wang commented on HADOOP-17087: ok, I understand. I agree that no change is best. Thanks [~ayushtkn] for the patient guidance. > Add EC flag to stat commands > > > Key: HADOOP-17087 > URL: https://issues.apache.org/jira/browse/HADOOP-17087 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-17087.001.patch > > > We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can > do but shows too much information. Neither {{du}} nor {{ls}} can accurately > judge the ec file. > So I added ec flag to stat cli. > old result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > regular file > $ hadoop fs -stat "%F" /user/rep/rep.txt > regular file > {code} > new result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > erasure coding file > $ hadoop fs -stat "%F" /user/rep/rep.txt > replica file > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17087) Add EC flag to stat commands
[ https://issues.apache.org/jira/browse/HADOOP-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143771#comment-17143771 ] Hongbing Wang commented on HADOOP-17087: Stat is a comprehensive description of the file. Maybe the ec flag should also be added in stat, I think. Do you [~ayushtkn] think it's necessary? > Add EC flag to stat commands > > > Key: HADOOP-17087 > URL: https://issues.apache.org/jira/browse/HADOOP-17087 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-17087.001.patch > > > We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can > do but shows too much information. Neither {{du}} nor {{ls}} can accurately > judge the ec file. > So I added ec flag to stat cli. > old result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > regular file > $ hadoop fs -stat "%F" /user/rep/rep.txt > regular file > {code} > new result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > erasure coding file > $ hadoop fs -stat "%F" /user/rep/rep.txt > replica file > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17087) Add EC flag to stat commands
[ https://issues.apache.org/jira/browse/HADOOP-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143769#comment-17143769 ] Hongbing Wang commented on HADOOP-17087: {quote} Ls with -e option {quote} Sorry i didn't notice this way before. Yahh~ it's good. Thanks [~ayushtkn] > Add EC flag to stat commands > > > Key: HADOOP-17087 > URL: https://issues.apache.org/jira/browse/HADOOP-17087 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-17087.001.patch > > > We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can > do but shows too much information. Neither {{du}} nor {{ls}} can accurately > judge the ec file. > So I added ec flag to stat cli. > old result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > regular file > $ hadoop fs -stat "%F" /user/rep/rep.txt > regular file > {code} > new result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > erasure coding file > $ hadoop fs -stat "%F" /user/rep/rep.txt > replica file > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17087) Add EC flag to stat commands
[ https://issues.apache.org/jira/browse/HADOOP-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-17087: --- Attachment: HADOOP-17087.001.patch > Add EC flag to stat commands > > > Key: HADOOP-17087 > URL: https://issues.apache.org/jira/browse/HADOOP-17087 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-17087.001.patch > > > We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can > do but shows too much information. Neither {{du}} nor {{ls}} can accurately > judge the ec file. > So I added ec flag to stat cli. > old result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > regular file > $ hadoop fs -stat "%F" /user/rep/rep.txt > regular file > {code} > new result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > erasure coding file > $ hadoop fs -stat "%F" /user/rep/rep.txt > replica file > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17087) Add EC flag to stat commands
[ https://issues.apache.org/jira/browse/HADOOP-17087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-17087: --- Description: We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can do but shows too much information. Neither {{du}} nor {{ls}} can accurately judge the ec file. So I added ec flag to stat cli. old result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt regular file $ hadoop fs -stat "%F" /user/rep/rep.txt regular file {code} new result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt erasure coding file $ hadoop fs -stat "%F" /user/rep/rep.txt replica file {code} was: We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can do but shows too much information. Neither {{du}} nor {{ls}} can accurately judge the ec file. So I added ec flag to stat cli. old result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt regular file $ hadoop fs -stat "%F" /user/rep/rep.txt regular file {code} new result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt erasure coding file $ hadoop fs -stat "%F" /user/rep/rep.txt replica file {code} > Add EC flag to stat commands > > > Key: HADOOP-17087 > URL: https://issues.apache.org/jira/browse/HADOOP-17087 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Hongbing Wang >Priority: Major > > We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can > do but shows too much information. Neither {{du}} nor {{ls}} can accurately > judge the ec file. > So I added ec flag to stat cli. > old result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > regular file > $ hadoop fs -stat "%F" /user/rep/rep.txt > regular file > {code} > new result: > {code:java} > $ hadoop fs -stat "%F" /user/ec/ec.txt > erasure coding file > $ hadoop fs -stat "%F" /user/rep/rep.txt > replica file > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17087) Add EC flag to stat commands
Hongbing Wang created HADOOP-17087: -- Summary: Add EC flag to stat commands Key: HADOOP-17087 URL: https://issues.apache.org/jira/browse/HADOOP-17087 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Hongbing Wang We currently do not have a brief way to judge an ec file. {{hdfs fsck}} can do but shows too much information. Neither {{du}} nor {{ls}} can accurately judge the ec file. So I added ec flag to stat cli. old result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt regular file $ hadoop fs -stat "%F" /user/rep/rep.txt regular file {code} new result: {code:java} $ hadoop fs -stat "%F" /user/ec/ec.txt erasure coding file $ hadoop fs -stat "%F" /user/rep/rep.txt replica file {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088738#comment-17088738 ] Hongbing Wang commented on HADOOP-16977: We have disabled kerberos. The username can be passed to YARN by setting the env var HADOOP_USER_NAME in the submission service (tomcat) and it takes effect. But if we use multiple HADOOP_USER_NAME in the submission service to submit the corresponding distcp, the username set with `System.setProperty(HADOOP_USER_NAME, value)` will override the previous settings. So, this may be a problem. > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-16977.001.patch, HADOOP-16977.002.patch > > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf, the conf values different from the > configuration files will not take effect in UserGroupInformation. E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086367#comment-17086367 ] Hongbing Wang commented on HADOOP-16977: Thank [~ste...@apache.org] , thank you for your detailed answers and suggestions. I think it should be fixed because our internal version uses `hadoop.user.name` for authentication, but it cannot be passed to distcp.execute (). We can choose `System.setProperty(k,v)` to solve the problem, but system variables will affect Global. Now, I try to modify from the business code. Thank you once again! > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-16977.001.patch, HADOOP-16977.002.patch > > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf, the conf values different from the > configuration files will not take effect in UserGroupInformation. E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-16977: --- Attachment: HADOOP-16977.002.patch > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-16977.001.patch, HADOOP-16977.002.patch > > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf, the conf values different from the > configuration files will not take effect in UserGroupInformation. E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-16977: --- Attachment: HADOOP-16977.001.patch > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > Attachments: HADOOP-16977.001.patch > > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf, the conf values different from the > configuration files will not take effect in UserGroupInformation. E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-16977: --- Description: org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always get the configure from the configuration files. Like below: {code:java} private static void ensureInitialized() { if (conf == null) { synchronized(UserGroupInformation.class) { if (conf == null) { // someone might have beat us initialize(new Configuration(), false); } } } }{code} So that, if FileSystem is created through FileSystem#get or FileSystem#newInstance with conf, the conf values different from the configuration files will not take effect in UserGroupInformation. E.g: {code:java} Configuration conf = new Configuration(); conf.set("k1","v1"); conf.set("k2","v2"); FileSystem fs = FileSystem.get(uri, conf);{code} "k1" or "k2" will not work in UserGroupInformation. was: org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always get the configure from the configuration files. Like below: {code:java} private static void ensureInitialized() { if (conf == null) { synchronized(UserGroupInformation.class) { if (conf == null) { // someone might have beat us initialize(new Configuration(), false); } } } }{code} So that, if FileSystem is created through FileSystem#get or FileSystem#newInstance with conf*, the conf values different from the configuration files will not take effect in UserGroupInformation.* E.g: {code:java} Configuration conf = new Configuration(); conf.set("k1","v1"); conf.set("k2","v2"); FileSystem fs = FileSystem.get(uri, conf);{code} "k1" or "k2" will not work in UserGroupInformation. > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf, the conf values different from the > configuration files will not take effect in UserGroupInformation. E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
[ https://issues.apache.org/jira/browse/HADOOP-16977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hongbing Wang updated HADOOP-16977: --- Description: org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always get the configure from the configuration files. Like below: {code:java} private static void ensureInitialized() { if (conf == null) { synchronized(UserGroupInformation.class) { if (conf == null) { // someone might have beat us initialize(new Configuration(), false); } } } }{code} So that, if FileSystem is created through FileSystem#get or FileSystem#newInstance with conf*, the conf values different from the configuration files will not take effect in UserGroupInformation.* E.g: {code:java} Configuration conf = new Configuration(); conf.set("k1","v1"); conf.set("k2","v2"); FileSystem fs = FileSystem.get(uri, conf);{code} "k1" or "k2" will not work in UserGroupInformation. > in javaApi, UGI params should be overidden through FileSystem conf > -- > > Key: HADOOP-16977 > URL: https://issues.apache.org/jira/browse/HADOOP-16977 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 2.7.2, 3.2.0 >Reporter: Hongbing Wang >Priority: Major > > org.apache.hadoop.security.UserGroupInformation#ensureInitialized,will always > get the configure from the configuration files. Like below: > {code:java} > private static void ensureInitialized() { > if (conf == null) { > synchronized(UserGroupInformation.class) { > if (conf == null) { // someone might have beat us > initialize(new Configuration(), false); > } > } > } > }{code} > So that, if FileSystem is created through FileSystem#get or > FileSystem#newInstance with conf*, the conf values different from the > configuration files will not take effect in UserGroupInformation.* E.g: > {code:java} > Configuration conf = new Configuration(); > conf.set("k1","v1"); > conf.set("k2","v2"); > FileSystem fs = FileSystem.get(uri, conf);{code} > "k1" or "k2" will not work in UserGroupInformation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16977) in javaApi, UGI params should be overidden through FileSystem conf
Hongbing Wang created HADOOP-16977: -- Summary: in javaApi, UGI params should be overidden through FileSystem conf Key: HADOOP-16977 URL: https://issues.apache.org/jira/browse/HADOOP-16977 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.2.0, 2.7.2 Reporter: Hongbing Wang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org