[jira] [Updated] (HADOOP-18563) Misleading AWS SDK S3 timeout configuration comment

[Oleksandr Shevchenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-18563:
--
Status: Patch Available  (was: In Progress)

> Misleading AWS SDK S3 timeout configuration comment
> ---
>
> Key: HADOOP-18563
> URL: https://issues.apache.org/jira/browse/HADOOP-18563
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Trivial
>  Labels: pull-request-available
>
> [https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
> {code:java}
> // seconds until we give up trying to establish a connection to s3
>   public static final String ESTABLISH_TIMEOUT =
>   "fs.s3a.connection.establish.timeout";
>   public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;
>   // seconds until we give up on a connection to s3
>   public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
>   public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
>  
> {code:java}
> public static void initConnectionSettings(Configuration conf,
>   ClientConfiguration awsConf) throws IOException {
> awsConf.setMaxConnections(intOption(conf, MAXIMUM_CONNECTIONS,
> DEFAULT_MAXIMUM_CONNECTIONS, 1));
> initProtocolSettings(conf, awsConf);
> awsConf.setMaxErrorRetry(intOption(conf, MAX_ERROR_RETRIES,
> DEFAULT_MAX_ERROR_RETRIES, 0));
> awsConf.setConnectionTimeout(intOption(conf, ESTABLISH_TIMEOUT,
> DEFAULT_ESTABLISH_TIMEOUT, 0));
> awsConf.setSocketTimeout(intOption(conf, SOCKET_TIMEOUT,
> DEFAULT_SOCKET_TIMEOUT, 0));
> int sockSendBuffer = intOption(conf, SOCKET_SEND_BUFFER,
> {code}
> AWS SDK S3 client actually consumes timeout in milliseconds
> [https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18563) Misleading AWS SDK S3 timeout configuration comment

[Oleksandr Shevchenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-18563:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Misleading AWS SDK S3 timeout configuration comment
> ---
>
> Key: HADOOP-18563
> URL: https://issues.apache.org/jira/browse/HADOOP-18563
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Trivial
>  Labels: pull-request-available
>
> [https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
> {code:java}
> // seconds until we give up trying to establish a connection to s3
>   public static final String ESTABLISH_TIMEOUT =
>   "fs.s3a.connection.establish.timeout";
>   public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;
>   // seconds until we give up on a connection to s3
>   public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
>   public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
>  
> {code:java}
> public static void initConnectionSettings(Configuration conf,
>   ClientConfiguration awsConf) throws IOException {
> awsConf.setMaxConnections(intOption(conf, MAXIMUM_CONNECTIONS,
> DEFAULT_MAXIMUM_CONNECTIONS, 1));
> initProtocolSettings(conf, awsConf);
> awsConf.setMaxErrorRetry(intOption(conf, MAX_ERROR_RETRIES,
> DEFAULT_MAX_ERROR_RETRIES, 0));
> awsConf.setConnectionTimeout(intOption(conf, ESTABLISH_TIMEOUT,
> DEFAULT_ESTABLISH_TIMEOUT, 0));
> awsConf.setSocketTimeout(intOption(conf, SOCKET_TIMEOUT,
> DEFAULT_SOCKET_TIMEOUT, 0));
> int sockSendBuffer = intOption(conf, SOCKET_SEND_BUFFER,
> {code}
> AWS SDK S3 client actually consumes timeout in milliseconds
> [https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-18563) Misleading AWS SDK S3 timeout configuration comment

[Oleksandr Shevchenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-18563:
--
Description: 
[https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
{code:java}
// seconds until we give up trying to establish a connection to s3
  public static final String ESTABLISH_TIMEOUT =
  "fs.s3a.connection.establish.timeout";
  public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;

  // seconds until we give up on a connection to s3
  public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
  public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
 
{code:java}
public static void initConnectionSettings(Configuration conf,
  ClientConfiguration awsConf) throws IOException {
awsConf.setMaxConnections(intOption(conf, MAXIMUM_CONNECTIONS,
DEFAULT_MAXIMUM_CONNECTIONS, 1));
initProtocolSettings(conf, awsConf);
awsConf.setMaxErrorRetry(intOption(conf, MAX_ERROR_RETRIES,
DEFAULT_MAX_ERROR_RETRIES, 0));
awsConf.setConnectionTimeout(intOption(conf, ESTABLISH_TIMEOUT,
DEFAULT_ESTABLISH_TIMEOUT, 0));
awsConf.setSocketTimeout(intOption(conf, SOCKET_TIMEOUT,
DEFAULT_SOCKET_TIMEOUT, 0));
int sockSendBuffer = intOption(conf, SOCKET_SEND_BUFFER,
{code}
AWS SDK S3 client actually consumes timeout in milliseconds
[https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185]

  was:
[https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
{code:java}
// seconds until we give up trying to establish a connection to s3
  public static final String ESTABLISH_TIMEOUT =
  "fs.s3a.connection.establish.timeout";
  public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;

  // seconds until we give up on a connection to s3
  public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
  public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
AWS SDK S3 client actually consumes timeout in milliseconds
https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185


> Misleading AWS SDK S3 timeout configuration comment
> ---
>
> Key: HADOOP-18563
> URL: https://issues.apache.org/jira/browse/HADOOP-18563
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Trivial
>  Labels: pull-request-available
>
> [https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
> {code:java}
> // seconds until we give up trying to establish a connection to s3
>   public static final String ESTABLISH_TIMEOUT =
>   "fs.s3a.connection.establish.timeout";
>   public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;
>   // seconds until we give up on a connection to s3
>   public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
>   public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
>  
> {code:java}
> public static void initConnectionSettings(Configuration conf,
>   ClientConfiguration awsConf) throws IOException {
> awsConf.setMaxConnections(intOption(conf, MAXIMUM_CONNECTIONS,
> DEFAULT_MAXIMUM_CONNECTIONS, 1));
> initProtocolSettings(conf, awsConf);
> awsConf.setMaxErrorRetry(intOption(conf, MAX_ERROR_RETRIES,
> DEFAULT_MAX_ERROR_RETRIES, 0));
> awsConf.setConnectionTimeout(intOption(conf, ESTABLISH_TIMEOUT,
> DEFAULT_ESTABLISH_TIMEOUT, 0));
> awsConf.setSocketTimeout(intOption(conf, SOCKET_TIMEOUT,
> DEFAULT_SOCKET_TIMEOUT, 0));
> int sockSendBuffer = intOption(conf, SOCKET_SEND_BUFFER,
> {code}
> AWS SDK S3 client actually consumes timeout in milliseconds
> [https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-18563) Misleading AWS SDK S3 timeout configuration comment

[Oleksandr Shevchenko (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17644802#comment-17644802
 ] 

Oleksandr Shevchenko commented on HADOOP-18563:
---

Opened PR: https://github.com/apache/hadoop/pull/5197

> Misleading AWS SDK S3 timeout configuration comment
> ---
>
> Key: HADOOP-18563
> URL: https://issues.apache.org/jira/browse/HADOOP-18563
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.4
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Trivial
>  Labels: pull-request-available
>
> [https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
> {code:java}
> // seconds until we give up trying to establish a connection to s3
>   public static final String ESTABLISH_TIMEOUT =
>   "fs.s3a.connection.establish.timeout";
>   public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;
>   // seconds until we give up on a connection to s3
>   public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
>   public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
> AWS SDK S3 client actually consumes timeout in milliseconds
> https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-18563) Misleading AWS SDK S3 timeout configuration comment

[Oleksandr Shevchenko (Jira)]

Oleksandr Shevchenko created HADOOP-18563:
-

 Summary: Misleading AWS SDK S3 timeout configuration comment
 Key: HADOOP-18563
 URL: https://issues.apache.org/jira/browse/HADOOP-18563
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.3.4
Reporter: Oleksandr Shevchenko
Assignee: Oleksandr Shevchenko


[https://github.com/apache/hadoop/blob/branch-3.3.5/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java#L254]
{code:java}
// seconds until we give up trying to establish a connection to s3
  public static final String ESTABLISH_TIMEOUT =
  "fs.s3a.connection.establish.timeout";
  public static final int DEFAULT_ESTABLISH_TIMEOUT = 5;

  // seconds until we give up on a connection to s3
  public static final String SOCKET_TIMEOUT = "fs.s3a.connection.timeout";
  public static final int DEFAULT_SOCKET_TIMEOUT = 20;{code}
AWS SDK S3 client actually consumes timeout in milliseconds
https://github.com/aws/aws-sdk-java/blob/1.11.600/aws-java-sdk-core/src/main/java/com/amazonaws/ClientConfiguration.java#L185



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899240#comment-16899240
 ] 

Oleksandr Shevchenko commented on HADOOP-15908:
---

Any objections about the current approach?
Can I get a review for the last patch?
Thanks!

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch, HADOOP-15908.002.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691758#comment-16691758
 ] 

Oleksandr Shevchenko commented on HADOOP-15865:
---

Could someone kindly review the patch?
Thanks!

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko reassigned HADOOP-15908:
-

Assignee: Oleksandr Shevchenko

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch, HADOOP-15908.002.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1668#comment-1668
 ] 

Oleksandr Shevchenko edited comment on HADOOP-15908 at 11/8/18 4:48 PM:


I misunderstood functionality of "runOnlyAtExecutionRoot" parameter.

But the approach is the same as before. We need to run 
"maven-remote-resources-plugin" to add "LICENSE.txt" and "NOTICE.txt" to the 
jars only when we build the whole project from project root directory.

As far as I see in HADOOP-13297 was added the workaround for project building:
1. Added dependency on hadoop-build-tools
2. hadoop-build-tools pom and jar should be always uploaded to the repository

In this case, we can't build the project (or needed submodule) if 
hadoop-build-tools doesn't present in the repository.

I have attached the new patch with the following changes:
1. Added "resource-bundle" profile which is activated only when we build the 
whole project from project root directory (check if LICENSE.txt is present in 
the current working directory where the build was triggered).
2. Removed dependency on hadoop-build-tools since this is not needed anymore.
3. Moved "configuration" section to the "execution" section just to fit 
official Maven plugin documentation.

I checked that "LICENSE.txt" and "NOTICE.txt" are present in jars when we build 
the whole project from project root directory.
{code:java}
jar tf hadoop-common-3.2.0-test.jar | grep LICENSE.txt
META-INF/LICENSE.txt
jar tf hadoop-common-3.2.0-test.jar | grep NOTICE.txt
META-INF/NOTICE.txt
{code}
These files are not present if we build only some submodule or run tests 
("maven-remote-resources-plugin" not run).

[~ajisakaa] Could you please review the new patch and evaluate the approach? 
Please correct me if I missed something.
Thanks!


was (Author: oshevchenko):
I misunderstood functionality of "runOnlyAtExecutionRoot" parameter.

But the approach is the same as before. We need to run 
"maven-remote-resources-plugin" to add "LICENSE.txt" and "NOTICE.txt" to the 
jars only when we build the whole project from project root directory.

As far as I see in HADOOP-13297 was added the workaround for project building:
1. Added dependency on hadoop-build-tools
2. hadoop-build-tools pom and jar should be always uploaded to the repository

In this case, we can't build the project (or needed submodule) if 
hadoop-build-tools doesn't present in the repository.

I have attached the new patch with the following changes:
1. Added "resource-bundle" profile with is activated only when we build the 
whole project from project root directory (check if LICENSE.txt is present in 
the current working directory where the build was triggered).
2. Removed dependency on hadoop-build-tools since this is not needed anymore.
3. Moved "configuration" section to the "execution" section just to fit 
official Maven plugin documentation.

I checked that "LICENSE.txt" and "NOTICE.txt" are present in jars when we build 
the whole project from project root directory.
{code}
jar tf hadoop-common-3.2.0-test.jar | grep LICENSE.txt
META-INF/LICENSE.txt
jar tf hadoop-common-3.2.0-test.jar | grep NOTICE.txt
META-INF/NOTICE.txt
{code}

These files are not present if we build only some submodule or run tests 
("maven-remote-resources-plugin" not run).

[~ajisakaa] Could you please review the new patch and evaluate the approach? 
Please correct me if I missed something.
Thanks!

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch, HADOOP-15908.002.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> 

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: HADOOP-15908.002.patch
Status: Patch Available  (was: Open)

I misunderstood functionality of "runOnlyAtExecutionRoot" parameter.

But the approach is the same as before. We need to run 
"maven-remote-resources-plugin" to add "LICENSE.txt" and "NOTICE.txt" to the 
jars only when we build the whole project from project root directory.

As far as I see in HADOOP-13297 was added the workaround for project building:
1. Added dependency on hadoop-build-tools
2. hadoop-build-tools pom and jar should be always uploaded to the repository

In this case, we can't build the project (or needed submodule) if 
hadoop-build-tools doesn't present in the repository.

I have attached the new patch with the following changes:
1. Added "resource-bundle" profile with is activated only when we build the 
whole project from project root directory (check if LICENSE.txt is present in 
the current working directory where the build was triggered).
2. Removed dependency on hadoop-build-tools since this is not needed anymore.
3. Moved "configuration" section to the "execution" section just to fit 
official Maven plugin documentation.

I checked that "LICENSE.txt" and "NOTICE.txt" are present in jars when we build 
the whole project from project root directory.
{code}
jar tf hadoop-common-3.2.0-test.jar | grep LICENSE.txt
META-INF/LICENSE.txt
jar tf hadoop-common-3.2.0-test.jar | grep NOTICE.txt
META-INF/NOTICE.txt
{code}

These files are not present if we build only some submodule or run tests 
("maven-remote-resources-plugin" not run).

[~ajisakaa] Could you please review the new patch and evaluate the approach? 
Please correct me if I missed something.
Thanks!

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch, HADOOP-15908.002.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute 

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Status: Open  (was: Patch Available)

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch, HADOOP-15908.002.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677910#comment-16677910
 ] 

Oleksandr Shevchenko commented on HADOOP-15908:
---

Thanks [~ajisakaa] for the review. I missed this. I'll recheck this and submit 
a new patch.

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676790#comment-16676790
 ] 

Oleksandr Shevchenko commented on HADOOP-15865:
---

[~ajisakaa] Thanks a lot for the review! [~andrew.wang]  Could you please 
double-check this?

Thanks.

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: (was: HADOOP-15865.001.patch)

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: HADOOP-15908.001.patch
Status: Patch Available  (was: Open)

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15908.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Status: Open  (was: Patch Available)

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15865.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15908:
--
Attachment: HADOOP-15865.001.patch
Status: Patch Available  (was: Open)

Patch is attached.

Could someone review the proposed changes?
Thanks.

> hadoop-build-tools jar is downloaded from remote repository instead of using 
> from local
> ---
>
> Key: HADOOP-15908
> URL: https://issues.apache.org/jira/browse/HADOOP-15908
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Minor
> Attachments: HADOOP-15865.001.patch
>
>
> HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml 
> to verify LICENSE.txt and NOTICE.txt files which includes 
> "hadoop-build-tools" remote resource bundles. 
> {code}
> 
>  org.apache.maven.plugins
>  maven-remote-resources-plugin
>  ${maven-remote-resources-plugin.version}
>  
>  
>  
> org.apache.hadoop:hadoop-build-tools:${hadoop.version}
>  
>  
>  
>  
>  org.apache.hadoop
>  hadoop-build-tools
>  ${hadoop.version}
>  
>  
>  
>  
>  
>  process
>  
>  
>  
>  
> {code}
> If we build only some module we always download " hadoop-build-tools" from 
> maven repository.
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
> hadoop-annotations ---
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 684 B/s)
> Downloading from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots: 
> http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
>  (609 B at 547 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
>  (791 B at 343 B/s)
> Downloading from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
> Downloaded from apache.snapshots.https: 
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
>  (0 B at 0 B/s)
> {noformat}
> If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
> try to build new version locally before repository will be created ) we can't 
> build some module:
> For example run:
> cd hadoop-common-project/
> mvn test
> Then we will get the following output:
> {noformat}
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) 
> on project hadoop-annotations: Execution default of goal 
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
> Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of 
> its dependencies could not be resolved: Failure to find 
> org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
> https://repo.maven.apache.org/maven2 was cached in the local repository, 
> resolution will not be reattempted until the update interval of central has 
> elapsed or updates are forced -> [Help 1]
> {noformat}
> Therefore, we need to limit execution of the Remote Resources Plugin only in 
> the root directory in which the build was run.
> To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
> From maven documentation 
> http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15908) hadoop-build-tools jar is downloaded from remote repository instead of using from local

[Oleksandr Shevchenko (JIRA)]

Oleksandr Shevchenko created HADOOP-15908:
-

 Summary: hadoop-build-tools jar is downloaded from remote 
repository instead of using from local
 Key: HADOOP-15908
 URL: https://issues.apache.org/jira/browse/HADOOP-15908
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Oleksandr Shevchenko


HADOOP-12893 added "maven-remote-resources-plugin" to hadoop-project/pom.xml to 
verify LICENSE.txt and NOTICE.txt files which includes "hadoop-build-tools" 
remote resource bundles. 
{code}

 org.apache.maven.plugins
 maven-remote-resources-plugin
 ${maven-remote-resources-plugin.version}
 
 
 
org.apache.hadoop:hadoop-build-tools:${hadoop.version}
 
 
 
 
 org.apache.hadoop
 hadoop-build-tools
 ${hadoop.version}
 
 
 
 
 
 process
 
 
 
 
{code}

If we build only some module we always download " hadoop-build-tools" from 
maven repository.

For example run:
cd hadoop-common-project/
mvn test
Then we will get the following output:
{noformat}
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
hadoop-annotations ---
Downloading from apache.snapshots: 
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: 
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
 (791 B at 684 B/s)
Downloading from apache.snapshots: 
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: 
http://repository.apache.org/snapshots/org/apache/hadoop/hadoop-main/3.3.0-SNAPSHOT/maven-metadata.xml
 (609 B at 547 B/s)
Downloading from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/maven-metadata.xml
 (791 B at 343 B/s)
Downloading from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
Downloaded from apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-build-tools/3.3.0-SNAPSHOT/hadoop-build-tools-3.3.0-20181022.232020-179.jar
 (0 B at 0 B/s)
{noformat}

If "hadoop-build-tools" jar doesn't exist in maven repository (for example we 
try to build new version locally before repository will be created ) we can't 
build some module:
For example run:
cd hadoop-common-project/
mvn test
Then we will get the following output:
{noformat}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on 
project hadoop-annotations: Execution default of goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed: 
Plugin org.apache.maven.plugins:maven-remote-resources-plugin:1.5 or one of its 
dependencies could not be resolved: Failure to find 
org.apache.hadoop:hadoop-build-tools:jar:3.2.0 in 
https://repo.maven.apache.org/maven2 was cached in the local repository, 
resolution will not be reattempted until the update interval of central has 
elapsed or updates are forced -> [Help 1]
{noformat}

Therefore, we need to limit execution of the Remote Resources Plugin only in 
the root directory in which the build was run.
To accomplish this, we can use the "runOnlyAtExecutionRoot parameter"
>From maven documentation 
>http://maven.apache.org/plugins/maven-remote-resources-plugin/usage.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667410#comment-16667410
 ] 

Oleksandr Shevchenko commented on HADOOP-15865:
---

Could someone review the attached changes?

Thanks.

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656643#comment-16656643
 ] 

Oleksandr Shevchenko commented on HADOOP-15865:
---

Thanks [~lqjacklee] for your comment.
Yes, HADOOP-15418 has the same root cause of the problem. Using not thread-safe 
iterators. 
The different is HADOOP-15418 fix the problem related to using iterator of 
Configuration object instead of the iterator of Properties object. When we 
iterate across Properties ConcurrentModificationException can't be thrown since 
we work with the local variable.

In the case described in this ticket, we use iterator of Properties object 
which is the parameter of overlay() method. As the result, we can get 
ConcurrentModificationException.

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15865:
--
Attachment: HADOOP-15865.001.patch
Status: Patch Available  (was: Open)

Could someone review the attached patch?

Thanks!

> ConcurrentModificationException in Configuration.overlay() method
> -
>
> Key: HADOOP-15865
> URL: https://issues.apache.org/jira/browse/HADOOP-15865
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Assignee: Oleksandr Shevchenko
>Priority: Major
> Attachments: HADOOP-15865.001.patch
>
>
> Configuration.overlay() is not thread-safe and can be the cause of 
> ConcurrentModificationException since we use iteration over Properties 
> object. 
> {code}
> private void overlay(Properties to, Properties from) {
>  for (Entry entry: from.entrySet()) {
>  to.put(entry.getKey(), entry.getValue());
>  }
>  }
> {code}
> Properties class is thread-safe but iterator is not. We should manually 
> synchronize on the returned set of entries which we use for iteration.
> We faced with ResourceManger fails during recovery caused by 
> ConcurrentModificationException:
> {noformat}
> 2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: 
> Service ResourceManager failed in state STARTED; cause: 
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
>  at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
>  at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
>  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
>  at 
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
>  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
>  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
>  at 
> org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
>  at 
> org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
>  at 
> org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
>  at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
>  at 
> org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
>  at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
>  at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>  at 
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
>  removing RMDelegation token with sequence number: 3489914
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
> RMDelegationToken and SequenceNumber
> 2018-10-12 08:00:56,968 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore:
>  Removing RMDelegationToken_3489914
> 2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
> 8032
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15865) ConcurrentModificationException in Configuration.overlay() method

[Oleksandr Shevchenko (JIRA)]

Oleksandr Shevchenko created HADOOP-15865:
-

 Summary: ConcurrentModificationException in 
Configuration.overlay() method
 Key: HADOOP-15865
 URL: https://issues.apache.org/jira/browse/HADOOP-15865
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Oleksandr Shevchenko
Assignee: Oleksandr Shevchenko


Configuration.overlay() is not thread-safe and can be the cause of 
ConcurrentModificationException since we use iteration over Properties object. 
{code}
private void overlay(Properties to, Properties from) {
 for (Entry entry: from.entrySet()) {
 to.put(entry.getKey(), entry.getValue());
 }
 }
{code}
Properties class is thread-safe but iterator is not. We should manually 
synchronize on the returned set of entries which we use for iteration.

We faced with ResourceManger fails during recovery caused by 
ConcurrentModificationException:
{noformat}
2018-10-12 08:00:56,968 INFO org.apache.hadoop.service.AbstractService: Service 
ResourceManager failed in state STARTED; cause: 
java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
 at java.util.Hashtable$Enumerator.next(Hashtable.java:1383)
 at org.apache.hadoop.conf.Configuration.overlay(Configuration.java:2801)
 at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2696)
 at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2632)
 at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2528)
 at org.apache.hadoop.conf.Configuration.get(Configuration.java:1062)
 at 
org.apache.hadoop.conf.Configuration.getStringCollection(Configuration.java:1914)
 at 
org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:53)
 at 
org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2043)
 at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2023)
 at 
org.apache.hadoop.yarn.webapp.util.WebAppUtils.getPassword(WebAppUtils.java:452)
 at 
org.apache.hadoop.yarn.webapp.util.WebAppUtils.loadSslConfiguration(WebAppUtils.java:428)
 at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:293)
 at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1017)
 at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1117)
 at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
 at 
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1251)
2018-10-12 08:00:56,968 INFO 
org.apache.hadoop.yarn.server.resourcemanager.security.RMDelegationTokenSecretManager:
 removing RMDelegation token with sequence number: 3489914
2018-10-12 08:00:56,968 INFO 
org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Removing 
RMDelegationToken and SequenceNumber
2018-10-12 08:00:56,968 INFO 
org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore: 
Removing RMDelegationToken_3489914
2018-10-12 08:00:56,969 INFO org.apache.hadoop.ipc.Server: Stopping server on 
8032
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609671#comment-16609671
 ] 

Oleksandr Shevchenko commented on HADOOP-15725:
---

Thanks all for participating in the discussion. Appreciate your help. 
Closing this ticket as Invalid.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user permissions before mark a file to delete. 
>  Could someone evaluate this? And if no one objects I would like to start 
> working on this.
>  Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko resolved HADOOP-15725.
---
Resolution: Invalid

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user permissions before mark a file to delete. 
>  Could someone evaluate this? And if no one objects I would like to start 
> working on this.
>  Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609606#comment-16609606
 ] 

Oleksandr Shevchenko commented on HADOOP-15725:
---

Thank you [~vinayrpet] for the example. You are right we can delete a file by 
delete() method the same as by deleteOnExit(). I expected a different behavior. 
But if you think that is correct behavior and this is safe for users we can 
close this ticket as Invalid.

[~szetszwo] I have checked one more time with permissions 700 for the directory 
and got the same result.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user permissions before mark a file to delete. 
>  Could someone evaluate this? And if no one objects I would like to start 
> working on this.
>  Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609368#comment-16609368
 ] 

Oleksandr Shevchenko commented on HADOOP-15738:
---

Thank you a lot [~jlowe]. Appreciate your help.

> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: HADOOP-15738
> URL: https://issues.apache.org/jira/browse/HADOOP-15738
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Reporter: Oleksandr Shevchenko
>Priority: Minor
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following 
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at 
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application 
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
>  at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be 
> finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609329#comment-16609329
 ] 

Oleksandr Shevchenko commented on HADOOP-15738:
---

Thank you [~jlowe]. "Move" option is not available for me for now since I'm not 
added to the project.

> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: HADOOP-15738
> URL: https://issues.apache.org/jira/browse/HADOOP-15738
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Reporter: Oleksandr Shevchenko
>Priority: Minor
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following 
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at 
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application 
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
>  at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be 
> finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko resolved HADOOP-15738.
---
Resolution: Fixed

> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: HADOOP-15738
> URL: https://issues.apache.org/jira/browse/HADOOP-15738
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Reporter: Oleksandr Shevchenko
>Priority: Minor
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following 
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at 
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application 
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
>  at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be 
> finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15738:
--
Description: 
MRAppBenchmark.benchmark1() fails with NullPointerException:
1. We do not set any queue for this test. As the result we got the following 
exception:
{noformat}
2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
(RMCommunicator.java:register(177)) - Exception while registering
java.lang.NullPointerException
at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
at org.apache.avro.util.Utf8.(Utf8.java:39)
at 
org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)

{noformat}
2. We override createSchedulerProxy method and do not set application priority 
that was added later by MAPREDUCE-6515. We got the following error:
{noformat}
java.lang.NullPointerException
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
 at java.lang.Thread.run(Thread.java:748)
{noformat}
In both cases, the job never will be run and the test stuck and will not be 
finished.

  was:
MRAppBenchmark.benchmark1() fails with NullPointerException:
1. We do not set any queue for this test. As the result we got the following 
exception:

{noformat}
2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
(RMCommunicator.java:register(177)) - Exception while registering
java.lang.NullPointerException
at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
at org.apache.avro.util.Utf8.(Utf8.java:39)
at 
org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)

{noformat}

2. We override createSchedulerProxy method and do not set application priority 
that was added later by MAPREDUCE-6515. We got the following error:

{noformat}

java.lang.NullPointerException
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
 at java.lang.Thread.run(Thread.java:748)
{noformat}

In both cases, the job never will be run and the test stuck and will not be 
finished.


> MRAppBenchmark.benchmark1() 

[jira] [Commented] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609272#comment-16609272
 ] 

Oleksandr Shevchenko commented on HADOOP-15738:
---

Sorry, incorrect component. Should be MAPREDUCE.

> MRAppBenchmark.benchmark1() fails with NullPointerException
> ---
>
> Key: HADOOP-15738
> URL: https://issues.apache.org/jira/browse/HADOOP-15738
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: test
>Reporter: Oleksandr Shevchenko
>Priority: Minor
>
> MRAppBenchmark.benchmark1() fails with NullPointerException:
> 1. We do not set any queue for this test. As the result we got the following 
> exception:
> {noformat}
> 2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
> (RMCommunicator.java:register(177)) - Exception while registering
> java.lang.NullPointerException
> at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
> at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
> at org.apache.avro.util.Utf8.(Utf8.java:39)
> at 
> org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
> at 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
> at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at 
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
> at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
> at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
> at 
> org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)
> {noformat}
> 2. We override createSchedulerProxy method and do not set application 
> priority that was added later by MAPREDUCE-6515. We got the following error:
> {noformat}
> java.lang.NullPointerException
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
>  at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
>  at java.lang.Thread.run(Thread.java:748)
> {noformat}
> In both cases, the job never will be run and the test stuck and will not be 
> finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15738) MRAppBenchmark.benchmark1() fails with NullPointerException

[Oleksandr Shevchenko (JIRA)]

Oleksandr Shevchenko created HADOOP-15738:
-

 Summary: MRAppBenchmark.benchmark1() fails with 
NullPointerException
 Key: HADOOP-15738
 URL: https://issues.apache.org/jira/browse/HADOOP-15738
 Project: Hadoop Common
  Issue Type: Bug
  Components: test
Reporter: Oleksandr Shevchenko


MRAppBenchmark.benchmark1() fails with NullPointerException:
1. We do not set any queue for this test. As the result we got the following 
exception:

{noformat}
2018-09-10 17:04:23,486 ERROR [Thread-0] rm.RMCommunicator 
(RMCommunicator.java:register(177)) - Exception while registering
java.lang.NullPointerException
at org.apache.avro.util.Utf8$2.toUtf8(Utf8.java:123)
at org.apache.avro.util.Utf8.getBytesFor(Utf8.java:172)
at org.apache.avro.util.Utf8.(Utf8.java:39)
at 
org.apache.hadoop.mapreduce.jobhistory.JobQueueChangeEvent.(JobQueueChangeEvent.java:35)
at 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.setQueueName(JobImpl.java:1167)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:174)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:301)
at org.apache.hadoop.mapreduce.v2.app.MRApp.submit(MRApp.java:285)
at org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.run(MRAppBenchmark.java:72)
at 
org.apache.hadoop.mapreduce.v2.app.MRAppBenchmark.benchmark1(MRAppBenchmark.java:194)

{noformat}

2. We override createSchedulerProxy method and do not set application priority 
that was added later by MAPREDUCE-6515. We got the following error:

{noformat}

java.lang.NullPointerException
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.handleJobPriorityChange(RMContainerAllocator.java:1025)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:880)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:286)
 at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:280)
 at java.lang.Thread.run(Thread.java:748)
{noformat}

In both cases, the job never will be run and the test stuck and will not be 
finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16609199#comment-16609199
 ] 

Oleksandr Shevchenko commented on HADOOP-15725:
---

Thank you [~ste...@apache.org] .

{quote}
1. FS.deleteOnExist() Calls FileSystem.delete(), so has exactly the same 
permissions as the delete() call for that FS instance.
{quote}
In this case not exactly the same. Let me explain.
When we call FileSystem.delete() we check permissions for a user who call 
delete() method.
When we call FS.deleteOnExist() from user Alice we do not check Alice's 
permissions for this file. Later when we call FS.close() (or JVM is shutdown) 
by user Bob we call delete for all files in deleteOnExit list and check Bob's 
permissions for each file (but Alice marked this files for deletion).

{quote}
2. therefore you cannot do more in deleteOnExit that you can from exactly the 
same FS instance through FileSystem.delete()
{quote}
It's a problem. We can do more that FileSystem.delete() call from another user 
if we marked files for deletion.

I think it's not correct behavior:
1. Create FS behalf of user Bob.
2. Mark some file for delete behalf of user Alice (Alice do not have enough 
permissions to delete this file)
3. Close FS behalf of user Bob.
4. The file will be deleted behalf of user Bob.

Correct me if I missed something. Or if something is not clear, please tell me 
and I will provide better clarification or some another example.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user permissions before mark a file to delete. 
>  Could someone evaluate this? And if no one objects I would like to start 
> working on this.
>  Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608859#comment-16608859
 ] 

Oleksandr Shevchenko commented on HADOOP-15725:
---

[~eyang] Thank you for your attention to this issue. I took into account and 
checked your comments.

The problem is not related to HDFS admin user or kerberos/non-kerberos cluster. 
I able to reproduce the same issue with a not-admin user on both secure and 
insecure cluster.
{noformat}
1) Run program behalf of "hive" user and delete file with "700 hive:hive" 
permissions behalf of "hive2" user
hive@node:~$ hadoop fs -ls /user/hive
Found 2 items
-rwx-- 1 hive hive 0 2018-09-10 07:02 /user/hive/testfile
drwxrwxr-x - root supergroup 0 2018-09-10 06:57 /user/hive/warehouse
hive@node:~$ java -jar testDeleteOther.jar 
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1314892313_1, ugi=hive 
(auth:SIMPLE)]]
/user/hive/testfile
hive2 (auth:SIMPLE)
hive@node:~$ hadoop fs -ls /user/hive
Found 1 items
drwxrwxr-x - root supergroup 0 2018-09-10 06:57 /user/hive/warehouse
hive@node:~$ id
uid=1000(hive) gid=1000(hive) groups=1000(hive)
hive@node:~$

2) Run program behalf of "hive" user and delete file with "700 hive:hive" 
permissions behalf of "hive2" user
[hive@node ~]$ hadoop fs -ls /hive
Found 1 items
-rwx-- 1 hive hive 0 2018-09-10 07:56 /hive/testfile
[hive@node ~]$  java -jar testDeleteOther.jar
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-1980008782_1, 
ugi=h...@example.com (auth:KERBEROS)]]
/hive/testfile
hive2 (auth:KERBEROS)
[hive@node ~]$ hadoop fs -ls /hive
[hive@node ~]$
{noformat}

The main cause of the problem is we allow to add any file by any user to delete 
list without checking permissions of a file for a user if we have a link on 
file system object which was created by another user.

I agree with you that not easy to exploit this vulnerability as we need to have 
a link on a file system object. Especially on the secured cluster (user should 
have kerberos ticket). But I think this is still a problem. Even if some user 
does not want to exploit this bug, we still have incorrect behavior that can 
lead to accidental deletion of files.
I propose to add permissions check during adding files to deleteOnExit list and 
throw AccessControlException in a case when a user who adds a file doesn't have 
enough permissions to delete a file.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user 

[jira] [Comment Edited] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606889#comment-16606889
 ] 

Oleksandr Shevchenko edited comment on HADOOP-15725 at 9/7/18 9:48 AM:
---

Thank you [~eyang] for your comment.
The same behavior on a kerberos cluster:
{noformat}
bash-4.1# hadoop fs -put testfile /tmp/
bash-4.1# hadoop fs -chmod 700 /tmp/testfile
bash-4.1# hadoop fs -ls /tmp
Found 2 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
-rwx-- 1 root root 0 2018-09-07 09:16 /tmp/testfile
bash-4.1# java -jar testDeleteOther.jar
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_2134263987_1, 
ugi=r...@example.com (auth:KERBEROS)]]
/tmp/testfile
hive (auth:KERBEROS)
bash-4.1# hadoop fs -ls /tmp
Found 1 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
bash-4.1#

{noformat}
{code:java}
final FileSystem fs;
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.namenode.kerberos.principal.pattern", "*");
conf.set("hadoop.rpc.protection", "privacy");
conf.set("fs.default.name", "hdfs://hadoop.docker.com:9000");
conf.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
);
UserGroupInformation.setConfiguration(conf);
fs = FileSystem.get(conf);
System.out.println(fs);

Path f = new Path("/tmp/testfile");
System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive", 
SaslRpcServer.AuthMethod.KERBEROS);
System.out.println(hive);

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));

fs.close();


{code}
Do you think this is correct behavior for non-Kerberos cluster? I believe this 
is not acceptable for insecure cluster too since we do not honor file 
permissions. Please, correct me if I missed something.
Appreciate your help.


was (Author: oshevchenko):
Thank you [~eyang] for your comment.
The same behavior on a kerberos cluster:
{noformat}

bash-4.1# hadoop fs -put testfile /tmp/
bash-4.1# hadoop fs -chmod 700 /tmp/testfile
bash-4.1# hadoop fs -ls /tmp
Found 2 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
-rwx-- 1 root root 0 2018-09-07 09:16 /tmp/testfile
bash-4.1# java -jar testDeleteOther.jar
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_2134263987_1, 
ugi=r...@example.com (auth:KERBEROS)]]
/tmp/testfile
hive (auth:KERBEROS)
bash-4.1# hadoop fs -ls /tmp
Found 1 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
bash-4.1#

{noformat}

{code}

final FileSystem fs;
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.namenode.kerberos.principal.pattern", "*");
conf.set("hadoop.rpc.protection", "privacy");
conf.set("fs.default.name", "hdfs://hadoop.docker.com:9000");
conf.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
);
UserGroupInformation.setConfiguration(conf);
fs = FileSystem.get(conf);
System.out.println(fs);

Path f = new Path("/tmp/testfile");
System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive", 
SaslRpcServer.AuthMethod.KERBEROS);
System.out.println(hive);

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));

fs.close();


{code}

Are you think this is correct behavior for non-Kerberos cluster? I believe this 
is not acceptable for insecure cluster too since we do not honor file 
permissions. Please, correct me if I missed something.
Appreciate your help.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  

[jira] [Commented] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606889#comment-16606889
 ] 

Oleksandr Shevchenko commented on HADOOP-15725:
---

Thank you [~eyang] for your comment.
The same behavior on a kerberos cluster:
{noformat}

bash-4.1# hadoop fs -put testfile /tmp/
bash-4.1# hadoop fs -chmod 700 /tmp/testfile
bash-4.1# hadoop fs -ls /tmp
Found 2 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
-rwx-- 1 root root 0 2018-09-07 09:16 /tmp/testfile
bash-4.1# java -jar testDeleteOther.jar
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_2134263987_1, 
ugi=r...@example.com (auth:KERBEROS)]]
/tmp/testfile
hive (auth:KERBEROS)
bash-4.1# hadoop fs -ls /tmp
Found 1 items
drwxrwx--- - root root 0 2018-09-07 06:16 /tmp/hadoop-yarn
bash-4.1#

{noformat}

{code}

final FileSystem fs;
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.namenode.kerberos.principal.pattern", "*");
conf.set("hadoop.rpc.protection", "privacy");
conf.set("fs.default.name", "hdfs://hadoop.docker.com:9000");
conf.set("fs.hdfs.impl",
org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
);
UserGroupInformation.setConfiguration(conf);
fs = FileSystem.get(conf);
System.out.println(fs);

Path f = new Path("/tmp/testfile");
System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive", 
SaslRpcServer.AuthMethod.KERBEROS);
System.out.println(hive);

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));

fs.close();


{code}

Are you think this is correct behavior for non-Kerberos cluster? I believe this 
is not acceptable for insecure cluster too since we do not honor file 
permissions. Please, correct me if I missed something.
Appreciate your help.

> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
>  Labels: Security
> Attachments: deleteOnExitReproduce
>
>
> For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
> security problems. Some user (Intruder) can get file system instance which 
> was created by another user (Owner) and mark any files to delete even if 
> "Intruder" doesn't have any access to this files. Later when "Owner" invoke 
> close method (or JVM is shut down since we have ShutdownHook which able to 
> close all file systems) marked files will be deleted successfully since 
> deleting was do behalf of "Owner" (or behalf of a user who ran a program).
> I attached the patch [^deleteOnExitReproduce] which reproduces this 
> possibility and also I able to reproduce it on a cluster with both Local and 
> Distributed file systems:
> {code:java}
> public class Main {
> public static void main(String[] args) throws Exception {
> final FileSystem fs;
>  Configuration conf = new Configuration();
>  conf.set("fs.default.name", "hdfs://node:9000");
>  conf.set("fs.hdfs.impl",
>  org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
>  );
>  fs = FileSystem.get(conf);
>  System.out.println(fs);
> Path f = new Path("/user/root/testfile");
>  System.out.println(f);
> UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");
> hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
> fs.close();
>  }
> {code}
> Result:
> {noformat}
> root@node:/# hadoop fs -put testfile /user/root
> root@node:/# hadoop fs -chmod 700 /user/root/testfile
> root@node:/# hadoop fs -ls /user/root
> Found 1 items
> -rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
> root@node:/# java -jar testDeleteOther.jar 
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.conf.Configuration.deprecation).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
> (auth:SIMPLE)]]
> /user/root/testfile
> []
> root@node:/# hadoop fs -ls /user/root
> root@node:/# 
> {noformat}
> We should add a check user permissions before mark a file to delete. 
>  Could someone evaluate this? And if no one objects I would like to start 
> working on this.
>  Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15725:
--
Description: 
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code:java}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));

fs.close();
 }
{code}
Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chmod 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
root@node:/# 
{noformat}
We should add a check user permissions before mark a file to delete. 
 Could someone evaluate this? And if no one objects I would like to start 
working on this.
 Thanks a lot for any comments.

  was:
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code:java}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}
Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chmod 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
root@node:/# 
{noformat}
We should add a check user permissions before mark a file to delete. 
 Could someone evaluate this? And if no one objects I would like to start 
working on this.
 Thanks a lot for any comments.


> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: 

[jira] [Updated] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15725:
--
Description: 
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code:java}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}
Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chown 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw-r--r-- 1 700 supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
root@node:/# 
{noformat}
We should add a check user permissions before mark a file to delete. 
 Could someone evaluate this? And if no one objects I would like to start 
working on this.
 Thanks a lot for any comments.

  was:
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}

Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chown 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw-r--r-- 1 700 supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
{noformat}

We should add a check user permissions before mark a file to delete. 
Could someone evaluate this? And if no one objects I would like to start 
working on this.
Thanks a lot for any comments.


> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: Hadoop Common
> 

[jira] [Updated] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Shevchenko updated HADOOP-15725:
--
Description: 
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code:java}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}
Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chmod 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw--- 1 root supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
root@node:/# 
{noformat}
We should add a check user permissions before mark a file to delete. 
 Could someone evaluate this? And if no one objects I would like to start 
working on this.
 Thanks a lot for any comments.

  was:
For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code:java}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}
Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chown 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw-r--r-- 1 700 supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
root@node:/# 
{noformat}
We should add a check user permissions before mark a file to delete. 
 Could someone evaluate this? And if no one objects I would like to start 
working on this.
 Thanks a lot for any comments.


> FileSystem.deleteOnExit should check user permissions
> -
>
> Key: HADOOP-15725
> URL: https://issues.apache.org/jira/browse/HADOOP-15725
> Project: 

[jira] [Created] (HADOOP-15725) FileSystem.deleteOnExit should check user permissions

[Oleksandr Shevchenko (JIRA)]

Oleksandr Shevchenko created HADOOP-15725:
-

 Summary: FileSystem.deleteOnExit should check user permissions
 Key: HADOOP-15725
 URL: https://issues.apache.org/jira/browse/HADOOP-15725
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Oleksandr Shevchenko
 Attachments: deleteOnExitReproduce

For now, we able to add any file to FileSystem deleteOnExit list. It leads to 
security problems. Some user (Intruder) can get file system instance which was 
created by another user (Owner) and mark any files to delete even if "Intruder" 
doesn't have any access to this files. Later when "Owner" invoke close method 
(or JVM is shut down since we have ShutdownHook which able to close all file 
systems) marked files will be deleted successfully since deleting was do behalf 
of "Owner" (or behalf of a user who ran a program).

I attached the patch [^deleteOnExitReproduce] which reproduces this possibility 
and also I able to reproduce it on a cluster with both Local and Distributed 
file systems:
{code}
public class Main {

public static void main(String[] args) throws Exception {

final FileSystem fs;
 Configuration conf = new Configuration();
 conf.set("fs.default.name", "hdfs://node:9000");
 conf.set("fs.hdfs.impl",
 org.apache.hadoop.hdfs.DistributedFileSystem.class.getName()
 );
 fs = FileSystem.get(conf);
 System.out.println(fs);

Path f = new Path("/user/root/testfile");
 System.out.println(f);

UserGroupInformation hive = UserGroupInformation.createRemoteUser("hive");

hive.doAs((PrivilegedExceptionAction) () -> fs.deleteOnExit(f));
 System.out.println(Arrays.asList(hive.getGroupNames()).toString());

fs.close();
 }
{code}

Result:
{noformat}
root@node:/# hadoop fs -put testfile /user/root
root@node:/# hadoop fs -chown 700 /user/root/testfile
root@node:/# hadoop fs -ls /user/root
Found 1 items
-rw-r--r-- 1 700 supergroup 0 2018-09-06 18:07 /user/root/testfile
root@node:/# java -jar testDeleteOther.jar 
log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.conf.Configuration.deprecation).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
info.
DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_309539034_1, ugi=root 
(auth:SIMPLE)]]
/user/root/testfile
[]
root@node:/# hadoop fs -ls /user/root
{noformat}

We should add a check user permissions before mark a file to delete. 
Could someone evaluate this? And if no one objects I would like to start 
working on this.
Thanks a lot for any comments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org