[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381249#comment-16381249
 ] 

Chris Douglas commented on HADOOP-13972:


Backported to branch-2.8 and branch-2.9

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374004#comment-16374004
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
This PR is merged as [this 
commit](https://github.com/apache/hadoop/commit/481d79fedc48942654dab08e23e71e80c8eb2aca).
 Therefore, closing it.


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374005#comment-16374005
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker closed the pull request at:

https://github.com/apache/hadoop/pull/339


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373283#comment-16373283
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@ssonker can you close this PR now it's been merged in? thanks


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366130#comment-16366130
 ] 

John Zhuge commented on HADOOP-13972:
-

Thanks [~ste...@apache.org] !

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Sharad Sonker (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365985#comment-16365985
 ] 

Sharad Sonker commented on HADOOP-13972:


Thank [~ste...@apache.org] for committing this and also for helping me out with 
the process.

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365900#comment-16365900
 ] 

Hudson commented on HADOOP-13972:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13664 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13664/])
HADOOP-13972. ADLS to support per-store configuration. Contributed by (stevel: 
rev 481d79fedc48942654dab08e23e71e80c8eb2aca)
* (edit) 
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/common/Parallelized.java
* (edit) hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md
* (edit) 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlConfKeys.java
* (edit) 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
* (edit) 
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365879#comment-16365879
 ] 

Steve Loughran commented on HADOOP-13972:
-

cherry picked into branches 3.0.x and branch-2; reran all the tests in branch-2 
before pushing it up. 

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365548#comment-16365548
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran Please see 
[HADOOP-15238](https://issues.apache.org/jira/browse/HADOOP-15238) for the 
Yetus patch checker. Yetus successfully verified the patch.



> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365475#comment-16365475
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
2-3 hours, normally. It can get a bit confused about PRs, which is why I've 
stopped doing it, even though github review is nice. 

if nothing happens, create a new JIRA, same title + "Yetus patch checker", 
mark as a duplicate of the original, and do a `git diff trunk...HEAD > 
HADOOP-13972-001.patch` call to build that first patch, attach it to the JIRA 
and submit. here's [an 
example](https://issues.apache.org/jira/browse/HADOOP-15003)


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365206#comment-16365206
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran Yeah, I read that in [_How to contribute_ 
wiki](https://wiki.apache.org/hadoop/HowToContribute#Contributing_your_work), 
but I wanted to have a look at jenkins as well. BTW, how long does it take 
after submitting the patch?


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364496#comment-16364496
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
the results should be added as a JIRA comment once completed


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364419#comment-16364419
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran Where can I find the build status of the patch. I was 
looking at 
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/ but 
couldn't locate this patch. Thanks.


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363859#comment-16363859
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
apart from the comments inline, I've got nothing else to add...lets see 
what yetus says. Good to see from the cloudstore test that the option 
provenance is being set, as it will help debugging in a world where things get 
more complex


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363858#comment-16363858
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r168154574
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
 ---
@@ -985,4 +995,65 @@ public void setUserGroupRepresentationAsUPN(boolean 
enableUPN) {
 oidOrUpn = enableUPN ? UserGroupRepresentation.UPN :
 UserGroupRepresentation.OID;
   }
+
+  /**
+   * Gets ADL account name from ADL FQDN
+   * @param accountFQDN ADL account fqdn
+   * @return ADL account name
+   */
+  public static String getAccountNameFromFQDN(String accountFQDN) {
+return accountFQDN.contains(".")
+? accountFQDN.substring(0, accountFQDN.indexOf("."))
+: accountFQDN;
+  }
+
+  /**
+   * Propagates account-specific settings into generic ADL configuration 
keys.
+   * This is done by propagating the values of the form
+   * {@code fs.adl.account.${account_name}.key} to
+   * {@code fs.adl.key}, for all values of "key"
+   *
+   * The source of the updated property is set to the key name of the 
account
+   * property, to aid in diagnostics of where things came from.
+   *
+   * Returns a new configuration. Why the clone?
+   * You can use the same conf for different filesystems, and the original
+   * values are not updated.
+   *
+   *
+   * @param source Source Configuration object
+   * @param accountName account name. Must not be empty
+   * @return a (potentially) patched clone of the original
+   * @throws IOException
+   */
+  public static Configuration propagateAccountOptions(
+  Configuration source, String accountName) throws IOException {
+if (StringUtils.isEmpty(accountName)) {
+  throw new IOException("Account name cannot be empty");
--- End diff --

We're generally happy with {{Precondition.checkArgument()}} here, which 
raises IllegalArgumentException. Best just to use that, unless you have tests 
which look for the specific exception


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363857#comment-16363857
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r168154261
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
 ---
@@ -68,12 +69,16 @@
 
 import static org.apache.hadoop.fs.adl.AdlConfKeys.*;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
--- End diff --

statics at the bottom; these should go into the same block as com.*, 
ideally in order. 


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363854#comment-16363854
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
OK. Go the Hadoop JIRA, hit "submit patch" and see what it says. I Can see 
it'll be complaining about the javadocs, but let's see what else.


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363527#comment-16363527
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran I have to make some changes in cloudstore (specifically in 
```ADLDiagnosticsInfo.java```) so that it can get the patched/propagated 
properties from ```AdlFileSystem```. Now when I run cloudstore, I get:
```
Hadoop information
==

  Hadoop 3.2.0-SNAPSHOT
  Compiled by ssonker on 2018-02-14T04:08Z
  Compiled with protoc 2.5.0
  From source with checksum b89064fa998ad944e87adfda36171356

Diagnostics for filesystem adl://tmphadooptest.azuredatalakestore.net/
==

Azure Datalake connector
ASF Filesystem Connector to Microsoft Azure Datalake
https://hadoop.apache.org/docs/current/hadoop-azure-datalake/index.html

System Properties
=



Selected and Sanitized Configuration Options


fs.adl.oauth2.client.id = "6**1" 
[fs.adl.account.tmphadooptest.oauth2.client.id via [core-site.xml]]
fs.adl.oauth2.credential = "9**=" 
[fs.adl.account.tmphadooptest.oauth2.credential via [core-site.xml]]
fs.adl.oauth2.access.token.provider.type = "ClientCredential" 
[core-default.xml]
fs.adl.oauth2.refresh.token = (unset)
fs.adl.oauth2.devicecode.clientapp.id = (unset)
fs.adl.oauth2.msi.port = (unset)
fs.adl.oauth2.refresh.url = 
"https://login.microsoftonline.com/fae1/oauth2/token; 
[fs.adl.account.tmphadooptest.oauth2.refresh.url via [core-site.xml]]
adl.feature.client.cache.readahead = (unset)
adl.feature.client.cache.drop.behind.writes = (unset)
adl.debug.override.localuserasfileowner = (unset)

Classes
===

class org.apache.hadoop.fs.adl.AdlFileSystem was found in 
file:/Users/ssonker/hadoop/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/share/hadoop/tools/lib/hadoop-azure-datalake-3.2.0-SNAPSHOT.jar
class com.microsoft.azure.datalake.store.ADLStoreClient was found in 
file:/Users/ssonker/hadoop/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/share/hadoop/tools/lib/azure-data-lake-store-sdk-2.2.5.jar

Endpoint: https://login.microsoftonline.com/fae1/oauth2/token:

==

Canonical hostname 52.172.144.29
  IP address 52.172.144.29
Connecting to https://login.microsoftonline.com/fae1/oauth2/token
Response: 200 : OK
HTTP response 200 from 
https://login.microsoftonline.com/fae1/oauth2/token: OK
Using proxy: false 


```


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363513#comment-16363513
 ] 

John Zhuge commented on HADOOP-13972:
-

Thanks [~ssonker] for the PR. Added you as a contributor to Hadoop Common and 
assigned the Jira to you.

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Sharad Sonker
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362850#comment-16362850
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran The account I'm using is 
adl://tmphadooptest.azuredatalakestore.net which is located in Central US.
I'll update with the cloudstore dump!


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362745#comment-16362745
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
OK. For absolute strictness, and this is the same rules we apply 
everywhere: which azure endpoint (e.g ireland, US, etc). (this declaration 
forces max honesty)

BTW, have a look at 
https://github.com/steveloughran/cloudstore/releases/tag/tag_2018-02-13-release-2
 and see what it does when given an ADL fs URI with this patch: it should list 
the origin of the patched properties 
https://github.com/steveloughran/cloudstore/releases/tag/tag_2018-02-13-release-2


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362708#comment-16362708
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran I have a personal ADL account against which I ran the test 
suite. I also tried it with per account settings with no default values, and 
all the test cases passed.


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362702#comment-16362702
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran I've incorporated the review comments, please have a look.


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362316#comment-16362316
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on the issue:

https://github.com/apache/hadoop/pull/339
  
As usual: which endpoint have you run the entire ADL test suite against? 
Have you also tried it with per account settings and no default values?


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362313#comment-16362313
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r167862833
  
--- Diff: hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md ---
@@ -32,6 +32,7 @@ This support comes via the JAR file 
`azure-datalake-store.jar`.
 * Tested for scale.
 * API `setOwner()`, `setAcl`, `removeAclEntries()`, `modifyAclEntries()` 
accepts UPN or OID
   (Object ID) as user and group names.
+* Supports per-account configuration
--- End diff --

add "."


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362312#comment-16362312
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r167862774
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
 ---
@@ -985,4 +994,65 @@ public void setUserGroupRepresentationAsUPN(boolean 
enableUPN) {
 oidOrUpn = enableUPN ? UserGroupRepresentation.UPN :
 UserGroupRepresentation.OID;
   }
+
+  /**
+   * Gets ADL account name from ADL FQDN
+   * @param accountFQDN ADL account fqdn
+   * @return ADL account name
+ */
+  public static String getAccountNameFromFQDN(String accountFQDN) {
+return accountFQDN.contains(".")
+? accountFQDN.substring(0, accountFQDN.indexOf("."))
+: accountFQDN;
+  }
+
+  /**
+   * Propagates account-specific settings into generic ADL configuration 
keys.
+   * This is done by propagating the values of the form
+   * {@code fs.adl.account.${account_name}.key} to
+   * {@code fs.adl.key}, for all values of "key"
+   *
+   * The source of the updated property is set to the key name of the 
account
+   * property, to aid in diagnostics of where things came from.
+   *
+   * Returns a new configuration. Why the clone?
+   * You can use the same conf for different filesystems, and the original
+   * values are not updated.
+   *
+   *
+   * @param source Source Configuration object
+   * @param accountName account name. Must not be empty
+   * @return a (potentially) patched clone of the original
+   * @throws IOException
+ */
+  public static Configuration propagateAccountOptions(
+  Configuration source, String accountName) throws IOException {
+if (StringUtils.isEmpty(accountName)) {
+  throw new IOException("Account name cannot be empty");
+}
+
+final String accountPrefix = AZURE_AD_ACCOUNT_PREFIX + accountName 
+'.';
+LOG.debug("Propagating entries under {}", accountPrefix);
+final Configuration dest = new Configuration(source);
+for (Map.Entry entry : source) {
+  final String key = entry.getKey();
+  // get the (unexpanded) value.
+  final String value = entry.getValue();
+  if (!key.startsWith(accountPrefix) || accountPrefix.equals(key)) {
+continue;
+  }
+  // there's a account prefix, so strip it
+  final String stripped = key.substring(accountPrefix.length());
+
+  // propagate the value, building a new origin field.
+  // to track overwrites, the generic key is overwritten even if
+  // already matches the new one.
+  String origin = "[" + StringUtils.join(
+  source.getPropertySources(key), ", ") +"]";
+  final String generic = AZURE_AD_PREFIX + stripped;
+  LOG.debug("Updating {} from {}", generic, origin);
+  dest.set(generic, value, key + " via " + origin);
--- End diff --

nice!


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362310#comment-16362310
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r167862653
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
 ---
@@ -985,4 +994,65 @@ public void setUserGroupRepresentationAsUPN(boolean 
enableUPN) {
 oidOrUpn = enableUPN ? UserGroupRepresentation.UPN :
 UserGroupRepresentation.OID;
   }
+
+  /**
+   * Gets ADL account name from ADL FQDN
+   * @param accountFQDN ADL account fqdn
+   * @return ADL account name
+ */
--- End diff --

indentation


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362309#comment-16362309
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r167862530
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java
 ---
@@ -65,6 +66,8 @@
 import org.apache.hadoop.util.Progressable;
 import org.apache.hadoop.util.ReflectionUtils;
 import org.apache.hadoop.util.VersionInfo;
+import org.slf4j.Logger;
--- End diff --

should be ordered according to the hadoop import ordering. I know we get 
this wrong, and once its wrong we don't like to fix (it breaks all patches and 
cherry picking), but it's good to try


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362307#comment-16362307
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user steveloughran commented on a diff in the pull request:

https://github.com/apache/hadoop/pull/339#discussion_r167862195
  
--- Diff: 
hadoop-tools/hadoop-azure-datalake/src/test/java/org/apache/hadoop/fs/adl/TestValidateConfiguration.java
 ---
@@ -152,6 +152,95 @@ public void testLoadDeprecatedKeys()
 assertDeprecatedKeys(conf);
   }
 
+  @Test
+  public void testGetAccountNameFromFQDN() {
+Assert.assertEquals("dummy", AdlFileSystem.
+getAccountNameFromFQDN("dummy.azuredatalakestore.net"));
+Assert.assertEquals("localhost", AdlFileSystem.
+getAccountNameFromFQDN("localhost"));
+  }
+
+  @Test
+  public void testPropagateAccountOptionsDefault() throws IOException {
+Configuration conf = new Configuration(false);
+conf.set("fs.adl.oauth2.client.id", "defaultClientId");
+conf.set("fs.adl.oauth2.credential", "defaultCredential");
+conf.set("some.other.config", "someValue");
+Configuration propagatedConf =
+AdlFileSystem.propagateAccountOptions(conf, "dummy");
+Assert.assertEquals("defaultClientId",
+propagatedConf.get(AZURE_AD_CLIENT_ID_KEY));
+Assert.assertEquals("defaultCredential",
+propagatedConf.get(AZURE_AD_CLIENT_SECRET_KEY));
+Assert.assertEquals("someValue",
+propagatedConf.get("some.other.config"));
+  }
+
+  @Test
+  public void testPropagateAccountOptionsSpecified() throws IOException {
+Configuration conf = new Configuration(false);
+conf.set("fs.adl.account.dummy.oauth2.client.id", "dummyClientId");
+conf.set("fs.adl.account.dummy.oauth2.credential", "dummyCredential");
+conf.set("some.other.config", "someValue");
+
+Configuration propagatedConf =
+AdlFileSystem.propagateAccountOptions(conf, "dummy");
+Assert.assertEquals("dummyClientId",
--- End diff --

just statically import Assert, or have the test case extend Assert


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362140#comment-16362140
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

Github user ssonker commented on the issue:

https://github.com/apache/hadoop/pull/339
  
@steveloughran @jzhuge Can you please have a look at this?


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362135#comment-16362135
 ] 

ASF GitHub Bot commented on HADOOP-13972:
-

GitHub user ssonker opened a pull request:

https://github.com/apache/hadoop/pull/339

HADOOP-13972 Supporting per-account configuration for ADL



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssonker/hadoop trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #339


commit 6f401a301d56a459af5d90c10d7a32e480d36915
Author: Sharad Sonker 
Date:   2018-02-13T10:43:34Z

HADOOP-13972 Supporting per-account configuration for ADL




> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Omkar Aradhya K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362074#comment-16362074
 ] 

Omkar Aradhya K S commented on HADOOP-13972:


Hi [~jzhuge] do you have any further information from where you left off on 
this feature?

> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Priority: Major
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276919#comment-16276919
 ] 

Steve Loughran commented on HADOOP-13972:
-

The S3A code should be pulled up to hadoop common for this; we don't want 
different semantics of per4-bucket options

including the (as yet uncommitted) work on jceks support, where getPassword() 
looks for the bucket keys ahead of any more general ones: HADOOP-14507


> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both 
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names, 
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted 
> first, then fall back to the global keys.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org