[jira] [Commented] (HADOOP-16381) The JSON License is included in binary tarball via azure-documentdb:1.16.2

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882654#comment-16882654
 ] 

Hudson commented on HADOOP-16381:
-

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16887 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16887/])
HADOOP-16381. The JSON License is included in binary tarball via (aajisaka: rev 
ccaa99c9235e0c6922e56f93254583eca49a12e1)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/main/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/DocumentStoreUtils.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/writer/cosmosdb/TestCosmosDBDocumentStoreWriter.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/reader/cosmosdb/TestCosmosDBDocumentStoreReader.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/main/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/writer/cosmosdb/CosmosDBDocumentStoreWriter.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/main/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/collection/document/entity/TimelineEventSubDoc.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/main/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/collection/document/entity/TimelineMetricSubDoc.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/main/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/reader/cosmosdb/CosmosDBDocumentStoreReader.java
* (add) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/src/test/java/org/apache/hadoop/yarn/server/timelineservice/documentstore/writer/cosmosdb/MockedCosmosDBDocumentStoreWriter.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-documentstore/pom.xml


> The JSON License is included in binary tarball via azure-documentdb:1.16.2
> --
>
> Key: HADOOP-16381
> URL: https://issues.apache.org/jira/browse/HADOOP-16381
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Sushil Ks
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16381.001.patch, HADOOP-16381.002.patch
>
>
> {noformat}
> $ mvn dependency:tree
> (snip)
> [INFO] +- com.microsoft.azure:azure-documentdb:jar:1.16.2:compile
> [INFO] |  +- com.fasterxml.uuid:java-uuid-generator:jar:3.1.4:compile
> [INFO] |  +- org.json:json:jar:20140107:compile
> [INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
> [INFO] |  \- joda-time:joda-time:jar:2.9.9:compile
> {noformat}
> org.json:json is JSON Licensed and it must be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16381) The JSON License is included in binary tarball via azure-documentdb:1.16.2

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HADOOP-16381:
---
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
   Status: Resolved  (was: Patch Available)

Committed this to trunk. Thanks [~Sushil-K-S] and [~ste...@apache.org].

> The JSON License is included in binary tarball via azure-documentdb:1.16.2
> --
>
> Key: HADOOP-16381
> URL: https://issues.apache.org/jira/browse/HADOOP-16381
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Sushil Ks
>Priority: Blocker
> Fix For: 3.3.0
>
> Attachments: HADOOP-16381.001.patch, HADOOP-16381.002.patch
>
>
> {noformat}
> $ mvn dependency:tree
> (snip)
> [INFO] +- com.microsoft.azure:azure-documentdb:jar:1.16.2:compile
> [INFO] |  +- com.fasterxml.uuid:java-uuid-generator:jar:3.1.4:compile
> [INFO] |  +- org.json:json:jar:20140107:compile
> [INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
> [INFO] |  \- joda-time:joda-time:jar:2.9.9:compile
> {noformat}
> org.json:json is JSON Licensed and it must be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16381) The JSON License is included in binary tarball via azure-documentdb:1.16.2

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882651#comment-16882651
 ] 

Akira Ajisaka commented on HADOOP-16381:


+1, committing this.

> The JSON License is included in binary tarball via azure-documentdb:1.16.2
> --
>
> Key: HADOOP-16381
> URL: https://issues.apache.org/jira/browse/HADOOP-16381
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Sushil Ks
>Priority: Blocker
> Attachments: HADOOP-16381.001.patch, HADOOP-16381.002.patch
>
>
> {noformat}
> $ mvn dependency:tree
> (snip)
> [INFO] +- com.microsoft.azure:azure-documentdb:jar:1.16.2:compile
> [INFO] |  +- com.fasterxml.uuid:java-uuid-generator:jar:3.1.4:compile
> [INFO] |  +- org.json:json:jar:20140107:compile
> [INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.10:compile
> [INFO] |  \- joda-time:joda-time:jar:2.9.9:compile
> {noformat}
> org.json:json is JSON Licensed and it must be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16401) ABFS: port Azure doc to 3.2 branch

[Masatake Iwasaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Masatake Iwasaki updated HADOOP-16401:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
   Fix Version/s: 3.2.1
Target Version/s: 3.2.1  (was: 3.2.0)
  Status: Resolved  (was: Patch Available)

Committed this to branch-3.2. Thanks, [~DanielZhou] and [~ste...@apache.org].

> ABFS: port Azure doc to 3.2 branch
> --
>
> Key: HADOOP-16401
> URL: https://issues.apache.org/jira/browse/HADOOP-16401
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Da Zhou
>Assignee: Masatake Iwasaki
>Priority: Major
> Fix For: 3.2.1
>
> Attachments: HADOOP-16401-branch-3.2.001.patch
>
>
> Need to port the latest Azure markdown docs from trunk to 3.2.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] mukul1987 opened a new pull request #1076: HDDS-1782. Add an option to MiniOzoneChaosCluster to read files multiple times. Contributed by Mukul Kumar Singh.

[GitBox]

mukul1987 opened a new pull request #1076: HDDS-1782. Add an option to 
MiniOzoneChaosCluster to read files multiple times. Contributed by Mukul Kumar 
Singh.
URL: https://github.com/apache/hadoop/pull/1076
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16384) ITestS3AContractRootDir failing.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882556#comment-16882556
 ] 

Hadoop QA commented on HADOOP-16384:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 16 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
17m  6s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue}  1m  
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching 
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
11s{color} | {color:green} trunk passed {color} |
| {color:orange}-0{color} | {color:orange} patch {color} | {color:orange}  1m 
50s{color} | {color:orange} Used diff version of patch file. Binary files and 
potentially other changes not applied. Please rebase and squash commits if 
necessary. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
49s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 
20s{color} | {color:green} root generated 0 new + 1475 unchanged - 2 fixed = 
1475 total (was 1477) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 12s{color} | {color:orange} root: The patch generated 10 new + 66 unchanged 
- 2 fixed = 76 total (was 68) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 16s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
43s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
43s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 28s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| 

[GitHub] [hadoop] hadoop-yetus commented on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

hadoop-yetus commented on issue #1003: HADOOP-16384: Avoid inconsistencies 
between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510285103
 
 
   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 33 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 1 | No case conflicting files found. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 16 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 71 | Maven dependency ordering for branch |
   | +1 | mvninstall | 1072 | trunk passed |
   | +1 | compile | 1042 | trunk passed |
   | +1 | checkstyle | 148 | trunk passed |
   | +1 | mvnsite | 131 | trunk passed |
   | +1 | shadedclient | 1026 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 127 | trunk passed |
   | 0 | spotbugs | 69 | Used deprecated FindBugs config; considering switching 
to SpotBugs. |
   | +1 | findbugs | 191 | trunk passed |
   | -0 | patch | 110 | Used diff version of patch file. Binary files and 
potentially other changes not applied. Please rebase and squash commits if 
necessary. |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 49 | Maven dependency ordering for patch |
   | +1 | mvninstall | 77 | the patch passed |
   | +1 | compile | 980 | the patch passed |
   | +1 | javac | 980 | root generated 0 new + 1475 unchanged - 2 fixed = 1475 
total (was 1477) |
   | -0 | checkstyle | 132 | root: The patch generated 10 new + 66 unchanged - 
2 fixed = 76 total (was 68) |
   | +1 | mvnsite | 105 | the patch passed |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | xml | 1 | The patch has no ill-formed XML file. |
   | +1 | shadedclient | 676 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 105 | the patch passed |
   | +1 | findbugs | 204 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 527 | hadoop-common in the patch passed. |
   | +1 | unit | 283 | hadoop-aws in the patch passed. |
   | +1 | asflicense | 43 | The patch does not generate ASF License warnings. |
   | | | 7048 | |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=17.05.0-ce Server=17.05.0-ce base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/15/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1003 |
   | JIRA Issue | HADOOP-16384 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient findbugs checkstyle xml |
   | uname | Linux e26139f99392 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / 555dabf |
   | Default Java | 1.8.0_212 |
   | checkstyle | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/15/artifact/out/diff-checkstyle-root.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/15/testReport/ |
   | Max. process+thread count | 1398 (vs. ulimit of 5500) |
   | modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws 
U: . |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1003/15/console |
   | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16417) abfs can't access storage account without password

[Da Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882533#comment-16882533
 ] 

Da Zhou commented on HADOOP-16417:
--

I am not sure if ABFS support anonymous connections, [~snvijaya], could you 
verify this when you have time?

> abfs can't access storage account without password
> --
>
> Key: HADOOP-16417
> URL: https://issues.apache.org/jira/browse/HADOOP-16417
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Jose Luis Pedrosa
>Assignee: Masatake Iwasaki
>Priority: Minor
>
> It does not seem possible to access storage accounts without passwords using 
> abfs, but it is possible using wasb.
>  
> This sample code (Spark based) to illustrate, the following code using 
> abfs_path with throw an exception
> {noformat}
> Exception in thread "main" java.lang.IllegalArgumentException: Invalid 
> account key.
> at 
> org.apache.hadoop.fs.azurebfs.services.SharedKeyCredentials.(SharedKeyCredentials.java:70)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:812)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:149)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:108)
> at 
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at 
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> {noformat}
>   While using the wasb_path will work normally,
> {code:java}
> import org.apache.spark.api.java.function.FilterFunction;
> import org.apache.spark.sql.RuntimeConfig;
> import org.apache.spark.sql.SparkSession;
> import org.apache.spark.sql.Dataset;
> import org.apache.spark.sql.Row;
> public class SimpleApp {
> static String blob_account_name = "azureopendatastorage";
> static String blob_container_name = "gfsweatherdatacontainer";
> static String blob_relative_path = "GFSWeather/GFSProcessed";
> static String blob_sas_token = "";
> static String abfs_path = 
> "abfs://"+blob_container_name+"@"+blob_account_name+".dfs.core.windows.net/"+blob_relative_path;
> static String wasbs_path = "wasbs://"+blob_container_name + 
> "@"+blob_account_name+".blob.core.windows.net/" + blob_relative_path;
> public static void main(String[] args) {
>
> SparkSession spark = SparkSession.builder().appName("NOAAGFS 
> Run").getOrCreate();
> configureAzureHadoopConnetor(spark);
> RuntimeConfig conf = spark.conf();
> 
> conf.set("fs.azure.account.key."+blob_account_name+".dfs.core.windows.net", 
> blob_sas_token);
> 
> conf.set("fs.azure.account.key."+blob_account_name+".blob.core.windows.net", 
> blob_sas_token);
> System.out.println("Creating parquet dataset");
> Dataset logData = spark.read().parquet(abfs_path);
> System.out.println("Creating temp view");
> logData.createOrReplaceTempView("source");
> System.out.println("SQL");
> spark.sql("SELECT * FROM source LIMIT 10").show();
> spark.stop();
> }
> public static void configureAzureHadoopConnetor(SparkSession session) {
> RuntimeConfig conf = session.conf();
> 
> conf.set("fs.AbstractFileSystem.wasb.impl","org.apache.hadoop.fs.azure.Wasb");
> 
> conf.set("fs.AbstractFileSystem.wasbs.impl","org.apache.hadoop.fs.azure.Wasbs");
> 
> conf.set("fs.wasb.impl","org.apache.hadoop.fs.azure.NativeAzureFileSystem");
> 
> conf.set("fs.wasbs.impl","org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure");
> conf.set("fs.azure.secure.mode", false);
> conf.set("fs.abfs.impl",  
> "org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem");
> conf.set("fs.abfss.impl", 
> "org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem");
> 
> conf.set("fs.AbstractFileSystem.abfs.impl","org.apache.hadoop.fs.azurebfs.Abfs");
> 
> conf.set("fs.AbstractFileSystem.abfss.impl","org.apache.hadoop.fs.azurebfs.Abfss");
> // Works in conjuction with fs.azure.secure.mode. Setting this config 
> to true
> //results in fs.azure.NativeAzureFileSystem using the local SAS 
> key generation
> //where the SAS keys are generating in the same process as 
> fs.azure.NativeAzureFileSystem.
> //If fs.azure.secure.mode flag is set to false, 

[GitHub] [hadoop] steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies 
between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510263195
 
 
   Status of latest patch
   
   * run the local and s3guard tests
   * disabled FS caching to stop the partition test failing *after* those 
tests; kicked off a new set
   
   Although it stops machines overloading, I'm now unsure about moving the MR 
tests down into the sequential phase; its one of the big sources of delay. I'm 
going to reinstate here and later put some effort into faster test execution in 
general. Not yet done it in this iteration


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882494#comment-16882494
 ] 

Hadoop QA commented on HADOOP-16418:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m  
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  0s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
34s{color} | {color:red} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra in trunk has 5 extant 
Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
30s{color} | {color:red} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen in trunk has 1 
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
13s{color} | {color:green} hadoop-tools/hadoop-dynamometer: The patch generated 
0 new + 0 unchanged - 8 fixed = 0 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 36s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
37s{color} | {color:green} hadoop-dynamometer-workload in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
42s{color} | {color:green} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra generated 0 new + 0 
unchanged - 5 fixed = 0 total (was 5) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
36s{color} | {color:green} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen generated 0 new + 0 
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
54s{color} | {color:green} hadoop-dynamometer-workload in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 28s{color} 
| {color:red} hadoop-dynamometer-infra in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
37s{color} | {color:green} hadoop-dynamometer-blockgen in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The 

[GitHub] [hadoop] justinuang closed pull request #645: HADOOP-16132 Support multipart download in S3AFileSystem

[GitBox]

justinuang closed pull request #645: HADOOP-16132 Support multipart download in 
S3AFileSystem
URL: https://github.com/apache/hadoop/pull/645
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] justinuang commented on issue #645: HADOOP-16132 Support multipart download in S3AFileSystem

[GitBox]

justinuang commented on issue #645: HADOOP-16132 Support multipart download in 
S3AFileSystem
URL: https://github.com/apache/hadoop/pull/645#issuecomment-510240463
 
 
   Sorry, I am not anymore. We can close it.
   
   On Wed, Jul 10, 2019 at 5:09 PM bolkedebruin 
   wrote:
   
   > @justinuang  are you still working on this?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bolkedebruin commented on issue #645: HADOOP-16132 Support multipart download in S3AFileSystem

[GitBox]

bolkedebruin commented on issue #645: HADOOP-16132 Support multipart download 
in S3AFileSystem
URL: https://github.com/apache/hadoop/pull/645#issuecomment-510229523
 
 
   @justinuang are you still working on this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882448#comment-16882448
 ] 

Erik Krogen commented on HADOOP-16418:
--

Fix those last 2 checkstyle warnings in v004

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch, HADOOP-16418.003.patch, HADOOP-16418.004.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HADOOP-16418:
-
Attachment: HADOOP-16418.004.patch

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch, HADOOP-16418.003.patch, HADOOP-16418.004.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16401) ABFS: port Azure doc to 3.2 branch

[Da Zhou (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882445#comment-16882445
 ] 

Da Zhou commented on HADOOP-16401:
--

[~ste...@apache.org], got it.

+1 to this patch.

> ABFS: port Azure doc to 3.2 branch
> --
>
> Key: HADOOP-16401
> URL: https://issues.apache.org/jira/browse/HADOOP-16401
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.2.0
>Reporter: Da Zhou
>Assignee: Masatake Iwasaki
>Priority: Major
> Attachments: HADOOP-16401-branch-3.2.001.patch
>
>
> Need to port the latest Azure markdown docs from trunk to 3.2.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882410#comment-16882410
 ] 

Hadoop QA commented on HADOOP-15729:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 21s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 19s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1 
new + 10 unchanged - 0 fixed = 11 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 24s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  4m 
49s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=18.09.7 Server=18.09.5 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HADOOP-15729 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12974256/HADOOP-15729.002.patch
 |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  |
| uname | Linux ec94c2650f2c 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6872efc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/16378/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HADOOP-Build/16378/testReport/ |
| Max. process+thread count | 363 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output | 

[jira] [Resolved] (HADOOP-16393) S3Guard init command uses global settings, not those of target bucket

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-16393.
-
   Resolution: Fixed
 Assignee: Steve Loughran
Fix Version/s: 3.3.0

> S3Guard init command uses global settings, not those of target bucket
> -
>
> Key: HADOOP-16393
> URL: https://issues.apache.org/jira/browse/HADOOP-16393
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
>
> If you call {{s3guard init s3a://name/}} then the custom bucket options of 
> fs.s3a.bucket.name are not picked up, instead the global value is used.
> Fix: take the name of the bucket and use that to eval properties and patch 
> the config used for the init command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16393) S3Guard init command uses global settings, not those of target bucket

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882407#comment-16882407
 ] 

Hudson commented on HADOOP-16393:
-

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #16885 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/16885/])
HADOOP-16393. S3Guard init command uses global settings, not those of (stevel: 
rev c7b5f858a00ece19e6ee22e9cb09080dfd419a56)
* (edit) 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardToolDynamoDB.java
* (edit) 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java


> S3Guard init command uses global settings, not those of target bucket
> -
>
> Key: HADOOP-16393
> URL: https://issues.apache.org/jira/browse/HADOOP-16393
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Priority: Major
>
> If you call {{s3guard init s3a://name/}} then the custom bucket options of 
> fs.s3a.bucket.name are not picked up, instead the global value is used.
> Fix: take the name of the bucket and use that to eval properties and patch 
> the config used for the init command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882406#comment-16882406
 ] 

Hadoop QA commented on HADOOP-16418:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 59s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
32s{color} | {color:red} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen in trunk has 1 
extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
37s{color} | {color:red} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra in trunk has 5 extant 
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 15s{color} | {color:orange} hadoop-tools/hadoop-dynamometer: The patch 
generated 2 new + 0 unchanged - 8 fixed = 2 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  1s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} hadoop-dynamometer-workload in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
43s{color} | {color:green} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra generated 0 new + 0 
unchanged - 5 fixed = 0 total (was 5) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
39s{color} | {color:green} 
hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-blockgen generated 0 new + 0 
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
57s{color} | {color:green} hadoop-dynamometer-workload in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 29s{color} 
| {color:red} hadoop-dynamometer-infra in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
46s{color} | {color:green} hadoop-dynamometer-blockgen in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
38s{color} | {color:green} 

[GitHub] [hadoop] steveloughran commented on issue #1012: HADOOP-16393. S3Guard init command uses global settings, not those of target bucket

[GitBox]

steveloughran commented on issue #1012: HADOOP-16393. S3Guard init command uses 
global settings, not those of target bucket
URL: https://github.com/apache/hadoop/pull/1012#issuecomment-510205472
 
 
   thanks, merged


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran closed pull request #1012: HADOOP-16393. S3Guard init command uses global settings, not those of target bucket

[GitBox]

steveloughran closed pull request #1012: HADOOP-16393. S3Guard init command 
uses global settings, not those of target bucket
URL: https://github.com/apache/hadoop/pull/1012
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882398#comment-16882398
 ] 

Sean Mackrory commented on HADOOP-15729:


Oh it was an obsolete import. Pull request created: 
https://github.com/apache/hadoop/pull/1075

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> 
>
> Key: HADOOP-15729
> URL: https://issues.apache.org/jira/browse/HADOOP-15729
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15729.001.patch, HADOOP-15729.002.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS 
> SDK requires an unbounded threadpool. It places monitoring tasks on the work 
> queue before the tasks they wait on, so it's possible (has even happened with 
> larger-than-default threadpools) for the executor to become permanently 
> saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a 
> bounded, blocking threadpool service for everything else S3A needs (although 
> currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
> only limits this threadpool, however we also specified fs.s3a.max.threads as 
> the number of core threads in the unbounded threadpool, which in hindsight is 
> pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a 
> sort of minimum. Once that many tasks have been submitted, the threadpool 
> will be locked at that number until it bursts beyond that, but it will only 
> spin down that far. If fs.s3a.max.threads is set reasonably high and someone 
> uses a bunch of S3 buckets, they could easily have thousands of idle threads 
> constantly.
> We should either not use fs.s3a.max.threads for the corepool size and 
> introduce a new configuration, or we should simply allow core threads to 
> timeout. I'm reading the OpenJDK source now to see what subtle differences 
> there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] mackrorysd opened a new pull request #1075: HADOOP-15729. [s3a] Allow core threads to time out.

[GitBox]

mackrorysd opened a new pull request #1075: HADOOP-15729. [s3a] Allow core 
threads to time out.
URL: https://github.com/apache/hadoop/pull/1075
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] ajayydv opened a new pull request #1074: HDDS-1544. Support default Acls for volume, bucket, keys and prefix. Contributed by Ajay Kumar.

[GitBox]

ajayydv opened a new pull request #1074: HDDS-1544. Support default Acls for 
volume, bucket, keys and prefix. Contributed by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/1074
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HADOOP-16420.
-
Resolution: Cannot Reproduce

> S3A returns 400 "bad request" on a single path within an S3 bucket
> --
>
> Key: HADOOP-16420
> URL: https://issues.apache.org/jira/browse/HADOOP-16420
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: out.txt
>
>
> Filing this as "who knows?"; surfaced during testing. Notable that the 
> previous testing was playing with SSE-C, if that makes a difference: it could 
> be that there's a marker entry encrypted with SSE-C that is now being 
> rejected by a different run.
> Somehow, with a set of credentials I can work with all paths in a directory, 
> except read the dir marker /fork-0001/"; try that and a 400 bad request comes 
> back. AWS console views the path as an empty dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882348#comment-16882348
 ] 

Steve Loughran commented on HADOOP-16420:
-

trying to change the encryption setting didn't help.

Deleting the entry from the AWS did. So it wasn't a permissions problem on my 
a/c.

Closing as cannot reproduce right now, but if it surfaces it'd be worth looking 
at in some other s3 clients to see what they say. 



> S3A returns 400 "bad request" on a single path within an S3 bucket
> --
>
> Key: HADOOP-16420
> URL: https://issues.apache.org/jira/browse/HADOOP-16420
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: out.txt
>
>
> Filing this as "who knows?"; surfaced during testing. Notable that the 
> previous testing was playing with SSE-C, if that makes a difference: it could 
> be that there's a marker entry encrypted with SSE-C that is now being 
> rejected by a different run.
> Somehow, with a set of credentials I can work with all paths in a directory, 
> except read the dir marker /fork-0001/"; try that and a 400 bad request comes 
> back. AWS console views the path as an empty dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882345#comment-16882345
 ] 

Steve Loughran commented on HADOOP-16420:
-

And a debug log of a stat command. Strange

Also surfaces on an attempt to rm -r it, either on an explicit path or some 
wildcard 
{code}
bin/hadoop fs -rm -R s3a://hwdev-steve-ireland-new/fork-\*001
{code}

Trivial: the hadoop fs command doesn't itself print an error, that just appears 
in the debug logs. Looks like the FsShell doesn't print errors there, even 
though stat will. Return code is 1 though

> S3A returns 400 "bad request" on a single path within an S3 bucket
> --
>
> Key: HADOOP-16420
> URL: https://issues.apache.org/jira/browse/HADOOP-16420
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: out.txt
>
>
> Filing this as "who knows?"; surfaced during testing. Notable that the 
> previous testing was playing with SSE-C, if that makes a difference: it could 
> be that there's a marker entry encrypted with SSE-C that is now being 
> rejected by a different run.
> Somehow, with a set of credentials I can work with all paths in a directory, 
> except read the dir marker /fork-0001/"; try that and a 400 bad request comes 
> back. AWS console views the path as an empty dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-16420:

Attachment: out.txt

> S3A returns 400 "bad request" on a single path within an S3 bucket
> --
>
> Key: HADOOP-16420
> URL: https://issues.apache.org/jira/browse/HADOOP-16420
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: out.txt
>
>
> Filing this as "who knows?"; surfaced during testing. Notable that the 
> previous testing was playing with SSE-C, if that makes a difference: it could 
> be that there's a marker entry encrypted with SSE-C that is now being 
> rejected by a different run.
> Somehow, with a set of credentials I can work with all paths in a directory, 
> except read the dir marker /fork-0001/"; try that and a 400 bad request comes 
> back. AWS console views the path as an empty dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-16420:
---

 Summary: S3A returns 400 "bad request" on a single path within an 
S3 bucket
 Key: HADOOP-16420
 URL: https://issues.apache.org/jira/browse/HADOOP-16420
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3
Affects Versions: 3.3.0
Reporter: Steve Loughran


Filing this as "who knows?"; surfaced during testing. Notable that the previous 
testing was playing with SSE-C, if that makes a difference: it could be that 
there's a marker entry encrypted with SSE-C that is now being rejected by a 
different run.

Somehow, with a set of credentials I can work with all paths in a directory, 
except read the dir marker /fork-0001/"; try that and a 400 bad request comes 
back. AWS console views the path as an empty dir.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16420) S3A returns 400 "bad request" on a single path within an S3 bucket

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882342#comment-16882342
 ] 

Steve Loughran commented on HADOOP-16420:
-

Stack
{code}
[ERROR] testDiffCommand(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal) 
 Time elapsed: 0.833 s  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on fork-0001/: 
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon 
S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 
4F1AAE9F69A106FB; S3 Extended Request ID: 
w0pdC9uJax/ex0DYWrwu7h75DG7YfwpLt/fY+sIOoJCXKg8vqmz4dy+vjTIfaXPEEam+ZOQ3z8g=), 
S3 Extended Request ID: 
w0pdC9uJax/ex0DYWrwu7h75DG7YfwpLt/fY+sIOoJCXKg8vqmz4dy+vjTIfaXPEEam+ZOQ3z8g=:400
 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 
400 Bad Request; Request ID: 4F1AAE9F69A106FB; S3 Extended Request ID: 
w0pdC9uJax/ex0DYWrwu7h75DG7YfwpLt/fY+sIOoJCXKg8vqmz4dy+vjTIfaXPEEam+ZOQ3z8g=)
at 
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:237)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2774)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2707)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2601)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2570)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2532)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2300)
at 
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
at 
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
at 
org.apache.hadoop.fs.s3a.AbstractS3ATestBase.setup(AbstractS3ATestBase.java:55)
at 
org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.setup(AbstractS3GuardToolTestBase.java:168)
at 
org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolLocal.setup(ITestS3GuardToolLocal.java:63)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request 
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 
4F1AAE9F69A106FB; S3 Extended Request ID: 
w0pdC9uJax/ex0DYWrwu7h75DG7YfwpLt/fY+sIOoJCXKg8vqmz4dy+vjTIfaXPEEam+ZOQ3z8g=), 
S3 Extended Request ID: 
w0pdC9uJax/ex0DYWrwu7h75DG7YfwpLt/fY+sIOoJCXKg8vqmz4dy+vjTIfaXPEEam+ZOQ3z8g=
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4920)
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4866)
at 
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1320)
at 

[jira] [Updated] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HADOOP-15729:
---
Attachment: (was: HADOOP-15729.002.patch)

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> 
>
> Key: HADOOP-15729
> URL: https://issues.apache.org/jira/browse/HADOOP-15729
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15729.001.patch, HADOOP-15729.002.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS 
> SDK requires an unbounded threadpool. It places monitoring tasks on the work 
> queue before the tasks they wait on, so it's possible (has even happened with 
> larger-than-default threadpools) for the executor to become permanently 
> saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a 
> bounded, blocking threadpool service for everything else S3A needs (although 
> currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
> only limits this threadpool, however we also specified fs.s3a.max.threads as 
> the number of core threads in the unbounded threadpool, which in hindsight is 
> pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a 
> sort of minimum. Once that many tasks have been submitted, the threadpool 
> will be locked at that number until it bursts beyond that, but it will only 
> spin down that far. If fs.s3a.max.threads is set reasonably high and someone 
> uses a bunch of S3 buckets, they could easily have thousands of idle threads 
> constantly.
> We should either not use fs.s3a.max.threads for the corepool size and 
> introduce a new configuration, or we should simply allow core threads to 
> timeout. I'm reading the OpenJDK source now to see what subtle differences 
> there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HADOOP-15729:
---
Attachment: HADOOP-15729.002.patch

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> 
>
> Key: HADOOP-15729
> URL: https://issues.apache.org/jira/browse/HADOOP-15729
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15729.001.patch, HADOOP-15729.002.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS 
> SDK requires an unbounded threadpool. It places monitoring tasks on the work 
> queue before the tasks they wait on, so it's possible (has even happened with 
> larger-than-default threadpools) for the executor to become permanently 
> saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a 
> bounded, blocking threadpool service for everything else S3A needs (although 
> currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
> only limits this threadpool, however we also specified fs.s3a.max.threads as 
> the number of core threads in the unbounded threadpool, which in hindsight is 
> pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a 
> sort of minimum. Once that many tasks have been submitted, the threadpool 
> will be locked at that number until it bursts beyond that, but it will only 
> spin down that far. If fs.s3a.max.threads is set reasonably high and someone 
> uses a bunch of S3 buckets, they could easily have thousands of idle threads 
> constantly.
> We should either not use fs.s3a.max.threads for the corepool size and 
> introduce a new configuration, or we should simply allow core threads to 
> timeout. I'm reading the OpenJDK source now to see what subtle differences 
> there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies 
between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510176835
 
 
   thanks for the tests. Reviewing the failures
   
   {code}
   
testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir)
  
   {code}
   
   This is precisely the tombstone problem surfacing again, somehow. if the 
root dir is considered entry the test rm is becomes a no-op and succeeds. 
   
   {code}
   ITestS3AEncryptionSSEC.testRenameFile:178 ? NullPointer
   {code}
   Now, that is a regression, looks like the local test runs are doing it 
because there's no state. Reverted back to trunk. 
   The DDB AncestorState now does (At debug level only) a check on the state of 
the result, logs if there are issues. View it as an audit of the outcome. But 
I'd missed that the local store used the same rename tracker without any 
object. Switched back to the trunk close-quietly operation, and if/when the 
auditing actually does real validation (or even better: goes down tagging 
everything in the dest dir are auth) then it can be revisited.
   {code}
   ITestPartitionCommitMRJob
   {code}
   
   I've filed https://issues.apache.org/jira/browse/HADOOP-16407 on this...it 
looks like the to move to sequential runs and FS recycling is causing this. 
I've not been worrying about it precisely because it's clearly a test setup 
problem that I believe can be done in isolation. But I can take another look...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882330#comment-16882330
 ] 

Erik Krogen commented on HADOOP-16418:
--

I added the suppression for the JavadocStyle check before the comment as you 
suggested. Nice find.

I fixed the {{BlockInfo}} FindBugs and also cleaned up that class a bit; there 
was quite a bit of dead code hanging around.

I pulled out some helper methods for the long method warning. For the 
{{AMOptions}} constructor, I don't see an issue with the parameter number -- 
the constructor is just used internally to be able to create the object. I 
added a suppression and marked the constructor private.

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch, HADOOP-16418.003.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HADOOP-16418:
-
Attachment: HADOOP-16418.003.patch

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch, HADOOP-16418.003.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HADOOP-16418:
-
Attachment: (was: HADOOP-16418.003.patch)

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-16418) Fix checkstyle and findbugs warnings in hadoop-dynamometer

[Erik Krogen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erik Krogen updated HADOOP-16418:
-
Attachment: HADOOP-16418.003.patch

> Fix checkstyle and findbugs warnings in hadoop-dynamometer
> --
>
> Key: HADOOP-16418
> URL: https://issues.apache.org/jira/browse/HADOOP-16418
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: tools
>Reporter: Masatake Iwasaki
>Assignee: Erik Krogen
>Priority: Minor
> Attachments: HADOOP-16418.000.patch, HADOOP-16418.001.patch, 
> HADOOP-16418.002.patch, HADOOP-16418.003.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] anuengineer merged pull request #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar.

[GitBox]

anuengineer merged pull request #973: HDDS-1611. Evaluate ACL on volume bucket 
key and prefix to authorize access. Contributed by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/973
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1037: HADOOP-15847 limit the r/w capacity

[GitBox]

steveloughran commented on issue #1037: HADOOP-15847 limit the r/w capacity 
URL: https://github.com/apache/hadoop/pull/1037#issuecomment-510147345
 
 
   I've committed it locally -before I push it to the ASF repo, I want to list 
you as the author field on the commit. What name/email addr do you want for the 
credit so it wires up to your github account?
   
   (i.e. if I do a `git commit --author "name "` what goes in there?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] arp7 merged pull request #1068: HDDS-1778. Fix existing blockade tests.

[GitBox]

arp7 merged pull request #1068: HDDS-1778. Fix existing blockade tests.
URL: https://github.com/apache/hadoop/pull/1068
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302163297
 
 

 ##
 File path: hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
 ##
 @@ -57,6 +57,8 @@ log4j.logger.org.apache.hadoop.ipc.Server=WARN
 #log4j.logger.org.apache.hadoop.fs.s3a.Listing=INFO
 # Log S3Guard classes
 #log4j.logger.org.apache.hadoop.fs.s3a.s3guard=DEBUG
+# if set to debug, this will log the PUT/DELETE operations on a store
+log4j.logger.org.apache.hadoop.fs.s3a.s3guard.Operations=DEBUG
 
 Review comment:
   for test run's it's actually quite interesting. But yes, I'll turn it off


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302162922
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardRootOperations.java
 ##
 @@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import java.io.File;
+import java.net.URI;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.assertj.core.api.Assertions;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runners.MethodSorters;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.impl.StoreContext;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.Constants.ENABLE_MULTI_DELETE;
+import static 
org.apache.hadoop.fs.s3a.Constants.S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.assume;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3AUtils.applyLocatedFiles;
+
+/**
+ * This test run against the root of the FS, and operations which span the DDB
+ * table and the filesystem.
+ * For this reason, these tests are executed in the sequential phase of the
+ * integration tests.
+ * 
+ * The tests only run if DynamoDB is the metastore.
+ */
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+public class ITestS3GuardRootOperations extends AbstractS3ATestBase {
+
+  private StoreContext storeContext;
+
+  private String fsUriStr;
+
+  private DynamoDBMetadataStore metastore;
 
 Review comment:
   noe `ITestS3GuardDDBRootOperations`; patched pom to track


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302160584
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
 ##
 @@ -921,43 +913,6 @@ private DDBPathMetadata verifyInAncestor(AncestorState 
state,
 return md;
   }
 
-  @Test
 
 Review comment:
   (a) slow (b) regularly timing out on teardown. We don't need a separate test 
to show we can provision a table, as we do that in the class setup/teardown 
anyway. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302158164
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md
 ##
 @@ -1100,6 +1100,87 @@ property should be configured, and the name of that 
table should be different
  incurring AWS charges.
 
 
+### How to Dump the Table and Metastore State
+
+There's an unstable entry point to list the contents of a table
+and S3 filesystem ot a set of TSV files
+
+```
+hadoop org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable s3a://bucket-x/ 
dir/out
+```
+
+This generates a set of files prefixed `dir/out-` with different views of the 
worl. 
+ which can then be viewed on the command line or editor:
+
+```
+"type" "deleted"   "path"  "is_auth_dir"   "is_empty_dir"  "len"   
"updated"   "updated_s" "last_modified" "last_modified_s"   "etag"  
"version"
 
 Review comment:
   IDE not detecting this. Odd


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302156330
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PurgeS3GuardTable.java
 ##
 @@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.serviceMain;
+import static 
org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.PARENT;
+
+/**
+ * Purge the S3Guard table of a FileSystem from all entries related to
+ * that table.
+ * Will fail if there is no table, or the store is in auth mode.
+ * 
+ *   hadoop org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardTable \
+ *   -force s3a://example-bucket/
+ * 
+ *
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class PurgeS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(PurgeS3GuardTable.class);
+
+  public static final String NAME = "PurgeS3GuardTable";
+
+  public static final String FORCE = "-force";
+
+  private static final String USAGE_MESSAGE = NAME
+  + " [-force] ";
+
+  private boolean force;
+
+  private long filesFound;
+  private long filesDeleted;
+
+  public PurgeS3GuardTable(final String name) {
+super(name);
+  }
+
+  public PurgeS3GuardTable() {
+this(NAME);
+  }
+
+  public PurgeS3GuardTable(
+  final S3AFileSystem filesystem,
+  final DynamoDBMetadataStore store,
 
 Review comment:
   changed the classname. Not creating one for the local store as it makes no 
sense to write a command line entry point to purge the table unique to that 
process.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302155360
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DumpS3GuardTable.java
 ##
 @@ -0,0 +1,691 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.Listing;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ALocatedFileStatus;
+import org.apache.hadoop.fs.s3a.S3ListRequest;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.S3AUtils.ACCEPT_ALL;
+
+/**
+ * This is a low-level diagnostics entry point which does a CVE/TSV dump of
+ * the DDB state.
+ * As it also lists the filesystem, it actually changes the state of the store
+ * during the operation.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class DumpS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(DumpS3GuardTable.class);
+
+  /**
+   * Application name.
+   */
+  public static final String NAME = "DumpS3GuardTable";
+
+  /**
+   * Usage.
+   */
+  private static final String USAGE_MESSAGE = NAME
+  + "  ";
+
+  /**
+   * Suffix for the flat list: {@value}.
+   */
+  public static final String FLAT_CSV = "-flat.csv";
+
+  /**
+   * Suffix for the raw S3 dump: {@value}.
+   */
+  public static final String RAW_CSV = "-s3.csv";
+
+  /**
+   * Suffix for the DDB scan: {@value}.
+   */
+  public static final String SCAN_CSV = "-scan.csv";
+
+  /**
+   * Suffix for the second DDB scan: : {@value}.
+   */
+  public static final String SCAN2_CSV = "-scan-2.csv";
+
+  /**
+   * Suffix for the treewalk scan of the S3A Filesystem: {@value}.
+   */
+  public static final String TREE_CSV = "-tree.csv";
+
+  /**
+   * Suffix for a recursive treewalk through the metastore: {@value}.
+   */
+  public static final String STORE_CSV = "-store.csv";
+
+  /**
+   * Path in the local filesystem to save the data.
+   */
+  protected String destPath;
+
+  /**
+   * Instantiate.
+   * @param name application name.
+   */
+  public DumpS3GuardTable(final String name) {
+super(name);
+  }
+
+  /**
+   * Instantiate with default name.
+   */
+  public DumpS3GuardTable() {
+this(NAME);
+  }
+
+  /**
+   * Bind to a specific FS + store.
+   * @param fs filesystem
+   * @param store metastore to use
+   * @param destFile the base filename for output
+   * @param uri URI of store -only needed if FS is null.
+   */
+  public DumpS3GuardTable(
+  final S3AFileSystem fs,
+  final DynamoDBMetadataStore store,
+  final File destFile,
+  final URI uri) {
+super(NAME, fs, store, uri);
+this.destPath = destFile.getAbsolutePath();
+  }
+
+  /**
+   * Bind to the argument list, including validating the CLI
+   * @throws 

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302153058
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DumpS3GuardTable.java
 ##
 @@ -0,0 +1,691 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.Listing;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ALocatedFileStatus;
+import org.apache.hadoop.fs.s3a.S3ListRequest;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.S3AUtils.ACCEPT_ALL;
+
+/**
+ * This is a low-level diagnostics entry point which does a CVE/TSV dump of
+ * the DDB state.
+ * As it also lists the filesystem, it actually changes the state of the store
+ * during the operation.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class DumpS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(DumpS3GuardTable.class);
+
+  /**
+   * Application name.
+   */
+  public static final String NAME = "DumpS3GuardTable";
+
+  /**
+   * Usage.
+   */
+  private static final String USAGE_MESSAGE = NAME
+  + "  ";
+
+  /**
+   * Suffix for the flat list: {@value}.
+   */
+  public static final String FLAT_CSV = "-flat.csv";
+
+  /**
+   * Suffix for the raw S3 dump: {@value}.
+   */
+  public static final String RAW_CSV = "-s3.csv";
+
+  /**
+   * Suffix for the DDB scan: {@value}.
+   */
+  public static final String SCAN_CSV = "-scan.csv";
+
+  /**
+   * Suffix for the second DDB scan: : {@value}.
+   */
+  public static final String SCAN2_CSV = "-scan-2.csv";
+
+  /**
+   * Suffix for the treewalk scan of the S3A Filesystem: {@value}.
+   */
+  public static final String TREE_CSV = "-tree.csv";
+
+  /**
+   * Suffix for a recursive treewalk through the metastore: {@value}.
+   */
+  public static final String STORE_CSV = "-store.csv";
+
+  /**
+   * Path in the local filesystem to save the data.
+   */
+  protected String destPath;
+
+  /**
+   * Instantiate.
+   * @param name application name.
+   */
+  public DumpS3GuardTable(final String name) {
+super(name);
+  }
+
+  /**
+   * Instantiate with default name.
+   */
+  public DumpS3GuardTable() {
+this(NAME);
+  }
+
+  /**
+   * Bind to a specific FS + store.
+   * @param fs filesystem
+   * @param store metastore to use
+   * @param destFile the base filename for output
+   * @param uri URI of store -only needed if FS is null.
+   */
+  public DumpS3GuardTable(
+  final S3AFileSystem fs,
+  final DynamoDBMetadataStore store,
+  final File destFile,
+  final URI uri) {
+super(NAME, fs, store, uri);
+this.destPath = destFile.getAbsolutePath();
+  }
+
+  /**
+   * Bind to the argument list, including validating the CLI
+   * @throws 

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302152332
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardDiagnostic.java
 ##
 @@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.service.launcher.AbstractLaunchableService;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+
+import static org.apache.hadoop.service.launcher.LauncherExitCodes.EXIT_USAGE;
+
+/**
+ * Entry point for diagnostics operations.
+ */
+public class AbstractS3GuardDiagnostic extends AbstractLaunchableService {
+
+  private S3AFileSystem filesystem;
+
+  private DynamoDBMetadataStore store;
+
+  private URI uri;
+
+  private List arguments;
+
+  /**
+   * Constructor.
+   * @param name entry point name.
+   */
+  public AbstractS3GuardDiagnostic(final String name) {
+super(name);
+  }
+
+  /**
+   * Constructor. If the store is set then that is the store for the operation,
+   * otherwise the filesystem's binding is used instead.
+   * @param name entry point name.
+   * @param filesystem filesystem
+   * @param store optional metastore.
+   * @param uri URI. Must be set if filesystem == null.
+   */
+  public AbstractS3GuardDiagnostic(
+  final String name,
+  @Nullable final S3AFileSystem filesystem,
+  @Nullable final DynamoDBMetadataStore store,
+  @Nullable final URI uri) {
+super(name);
+this.store = store;
+this.filesystem = filesystem;
+if (store == null) {
+  require(filesystem != null, "No filesystem or URI");
+  bindStore(filesystem);
+}
+if (uri == null) {
+  require(filesystem != null, "No filesystem or URI");
+  setUri(filesystem.getUri());
+} else {
+  setUri(uri);
+}
+  }
+
+  /**
+   * Require a condition to hold, otherwise an exception is thrown.
+   * @param condition condition to be true
+   * @param error text on failure.
+   * @throws ServiceLaunchException if the condition is not met
+   */
+  protected static void require(boolean condition, String error) {
+if (!condition) {
+  throw failure(error);
+}
+  }
+
+  /**
+   * Generate a failure exception for throwing.
+   * @param message message
+   * @param ex optional nested exception.
+   * @return an exception to throw
+   */
+  protected static ServiceLaunchException failure(String message, Throwable 
ex) {
+return new ServiceLaunchException(LauncherExitCodes.EXIT_FAIL, message, 
ex);
+  }
+
+  /**
+   * Generate a failure exception for throwing.
+   * @param message message
+   * @return an exception to throw
+   */
+  protected static ServiceLaunchException failure(String message) {
+return new ServiceLaunchException(LauncherExitCodes.EXIT_FAIL, message);
+  }
+
+  @Override
+  public Configuration bindArgs(final Configuration config,
+  final List args)
+  throws Exception {
+this.arguments = args;
+return super.bindArgs(config, args);
+  }
+
+  /**
+   * Get the argument list.
+   * @return the argument list.
+   */
+  protected List getArguments() {
+return arguments;
+  }
+
+  /**
+   * Bind to the store from a CLI argument.
+   * @param fsURI filesystem URI
+   * @throws IOException failure
+   */
+  protected void bindFromCLI(String fsURI)
+  throws IOException {
+Configuration conf = getConfig();
+setUri(fsURI);
+FileSystem fs = FileSystem.get(getUri(), conf);
+require(fs instanceof S3AFileSystem,
+"Not an S3A Filesystem:  " + fsURI);
+filesystem = (S3AFileSystem) fs;
+bindStore(filesystem);
+setUri(fs.getUri());
+  }
+
+  /**
+   * Binds the {@link #store} field to the metastore of
+   * the 

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302151422
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java
 ##
 @@ -538,7 +538,7 @@ public CompleteMultipartUploadResult commitUpload(
   public BulkOperationState initiateCommitOperation(
   Path path) throws IOException {
 return S3Guard.initiateBulkWrite(owner.getMetadataStore(),
-BulkOperationState.OperationType.Put, path);
 
 Review comment:
   I hadn't defined the commit type, I'd just had a "Put". Now I've added that 
and an Fsck. It's useful when you log the s3guard.Operations log (Try it!) 
because we log the bulk operation Id and type as well as DDB 
PUT/DELETE/TOMBSTONE operations. Worth doing if you want to see what's being 
done to a table


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] mackrorysd commented on issue #992: HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists

[GitBox]

mackrorysd commented on issue #992: HADOOP-16357 TeraSort Job failing on S3 
DirectoryStagingCommitter: destination path exists
URL: https://github.com/apache/hadoop/pull/992#issuecomment-510126748
 
 
   Ok, sounds reasonable enough. +1 from me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302150639
 
 

 ##
 File path: hadoop-tools/hadoop-aws/pom.xml
 ##
 @@ -215,15 +219,21 @@
   
   
   
-**/ITestS3AContractRootDir.java
 **/ITestS3AFileContextStatistics.java
+
 **/ITestS3AHuge*.java
+
 **/ITestS3AEncryptionSSEC*.java
 
 **/ITestDynamoDBMetadataStoreScale.java
 
 
 **/ITestTerasort*.java
+
+**/ITest*CommitMRJob.java
 
 Review comment:
   Maybe.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] mackrorysd commented on issue #1012: HADOOP-16393. S3Guard init command uses global settings, not those of target bucket

[GitBox]

mackrorysd commented on issue #1012: HADOOP-16393. S3Guard init command uses 
global settings, not those of target bucket
URL: https://github.com/apache/hadoop/pull/1012#issuecomment-510126336
 
 
   +1 - LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302149768
 
 

 ##
 File path: hadoop-tools/hadoop-aws/pom.xml
 ##
 @@ -186,7 +186,11 @@
 **/ITestS3AHuge*.java
 
 **/ITestDynamoDBMetadataStoreScale.java
+
 
 Review comment:
   1. The terasort tests are *also* MR jobs; I consider the comment to be 
correctly placed. Sorry.
   2. The issue isn't that the minicluster uses lots of ram, it is that we span 
multiple processes as part of the job (this is good) and together they use up 
all your ram. If on a parallel test run >1 MR test suite is run at the same 
time, your machine pages out the VM, processes pause, tests timeout, my video 
on conf calls breaks up etc. Doing it sequentially ensures that the machine 
stays usable during a test run of, say, threads=8


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882209#comment-16882209
 ] 

Anu Engineer edited comment on HADOOP-13363 at 7/10/19 4:04 PM:


{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf 
upgrade event?
{quote}
Me. The support of 2.5 series in non-existent now. Either we need to fork 
protobuf 2.5 and make it part of Hadoop tool chain or move. We cannot forever 
depend on 2.5 being around.


was (Author: anu):
{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf 
upgrade event?
{quote}
Me.

> Upgrade protobuf from 2.5.0 to something newer
> --
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Major
>  Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, 
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms.  (See, for 
> example, https://gist.github.com/BennettSmith/7111094 ).  In order for us to 
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is 
> starting to slowly disappear as a standard install-able package for even 
> Linux/x86, we need to either upgrade or self bundle or something else.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302148149
 
 

 ##
 File path: 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java
 ##
 @@ -185,28 +191,57 @@ public void testListEmptyRootDirectory() throws 
IOException {
 for (FileStatus status : statuses) {
   ContractTestUtils.assertDeleted(fs, status.getPath(), true);
 }
-assertEquals("listStatus on empty root-directory returned a non-empty 
list",
-0, fs.listStatus(root).length);
-assertFalse("listFiles(/, false).hasNext",
-fs.listFiles(root, false).hasNext());
-assertFalse("listFiles(/, true).hasNext",
-fs.listFiles(root, true).hasNext());
-assertFalse("listLocatedStatus(/).hasNext",
-fs.listLocatedStatus(root).hasNext());
+FileStatus[] list1 = fs.listStatus(root);
+assertEquals("listStatus on empty root-directory returned found: "
++ join("\n", list1),
+0, list1.length);
+assertNoElements("listFiles(/, false)",
+fs.listFiles(root, false));
+assertNoElements("listFiles(/, true)",
+fs.listFiles(root, true));
+assertNoElements("listLocatedStatus(/)",
+fs.listLocatedStatus(root));
 assertIsDirectory(root);
   }
 
+  /**
+   * Assert that an iterator has no elements; the raised exception
+   * will include the element list.
+   * @param operation operation for assertion text.
+   * @param iter iterator
+   * @throws IOException failure retrieving the values.
+   */
+  protected void assertNoElements(String operation,
+  RemoteIterator iter) throws IOException {
+List resultList = toList(iter);
+if (!resultList.isEmpty()) {
+  fail("Expected no results from " + operation + ", but got "
 
 Review comment:
   thx. I'm picking up assertJ assertions in hadoop-aws code but not yet 
adopted any in hadoop-common


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302147780
 
 

 ##
 File path: 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java
 ##
 @@ -185,28 +191,57 @@ public void testListEmptyRootDirectory() throws 
IOException {
 for (FileStatus status : statuses) {
   ContractTestUtils.assertDeleted(fs, status.getPath(), true);
 }
-assertEquals("listStatus on empty root-directory returned a non-empty 
list",
-0, fs.listStatus(root).length);
-assertFalse("listFiles(/, false).hasNext",
-fs.listFiles(root, false).hasNext());
-assertFalse("listFiles(/, true).hasNext",
-fs.listFiles(root, true).hasNext());
-assertFalse("listLocatedStatus(/).hasNext",
-fs.listLocatedStatus(root).hasNext());
+FileStatus[] list1 = fs.listStatus(root);
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13363) Upgrade protobuf from 2.5.0 to something newer

[Anu Engineer (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882209#comment-16882209
 ] 

Anu Engineer commented on HADOOP-13363:
---

{quote}Who watching this JIRA thinks "it is time" to make Hadoop 3.3 a protobuf 
upgrade event?
{quote}
Me.

> Upgrade protobuf from 2.5.0 to something newer
> --
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
>Reporter: Allen Wittenauer
>Priority: Major
>  Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch, 
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms.  (See, for 
> example, https://gist.github.com/BennettSmith/7111094 ).  In order for us to 
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is 
> starting to slowly disappear as a standard install-able package for even 
> Linux/x86, we need to either upgrade or self bundle or something else.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1037: HADOOP-15847 limit the r/w capacity

[GitBox]

steveloughran commented on issue #1037: HADOOP-15847 limit the r/w capacity 
URL: https://github.com/apache/hadoop/pull/1037#issuecomment-510108406
 
 
   Patch LGTM
   
   +1
   
   when I commit I'll also cut the line "Tip: for agility, use DynamoDB 
autoscaling" as its obsolete now


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #992: HADOOP-16357 TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists

[GitBox]

steveloughran commented on issue #992: HADOOP-16357 TeraSort Job failing on S3 
DirectoryStagingCommitter: destination path exists
URL: https://github.com/apache/hadoop/pull/992#issuecomment-510107381
 
 
   we're changing the default mode to == that of FileOutputCommitter, namely 
"it's ok if the dest exists"
   
   Spark will refuse to execute if the dest exists, so to you use any of the 
conflict options you actually need to permit overwrite -so you won't change 
behaviour without noticing
   
   What this does do is make the committer more consistent with the existing one


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-16404) ABFS default blocksize change(256MB from 512MB)

[Arun Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Singh reassigned HADOOP-16404:
---

Assignee: Arun Singh

> ABFS default blocksize change(256MB from 512MB)
> ---
>
> Key: HADOOP-16404
> URL: https://issues.apache.org/jira/browse/HADOOP-16404
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Affects Versions: 3.1.2
>Reporter: Arun Singh
>Assignee: Arun Singh
>Priority: Major
>  Labels: patch
> Fix For: 3.1.2
>
> Attachments: HADOOP-16404.patch
>
>
> We intend to change the default blocksize of the abfs driver to 256Mb from 
> 512MB.
> After changing the blocksize we have performed a series of test(Spark Tera, 
> Spark DFSIO,TPCDS on HIVE) and have seen consistent improvements in order of 
> 4-5 %



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HADOOP-15729:
---
Status: Patch Available  (was: Open)

Resubmitting as checkstyle output has expired.

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> 
>
> Key: HADOOP-15729
> URL: https://issues.apache.org/jira/browse/HADOOP-15729
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15729.001.patch, HADOOP-15729.002.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS 
> SDK requires an unbounded threadpool. It places monitoring tasks on the work 
> queue before the tasks they wait on, so it's possible (has even happened with 
> larger-than-default threadpools) for the executor to become permanently 
> saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a 
> bounded, blocking threadpool service for everything else S3A needs (although 
> currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
> only limits this threadpool, however we also specified fs.s3a.max.threads as 
> the number of core threads in the unbounded threadpool, which in hindsight is 
> pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a 
> sort of minimum. Once that many tasks have been submitted, the threadpool 
> will be locked at that number until it bursts beyond that, but it will only 
> spin down that far. If fs.s3a.max.threads is set reasonably high and someone 
> uses a bunch of S3 buckets, they could easily have thousands of idle threads 
> constantly.
> We should either not use fs.s3a.max.threads for the corepool size and 
> introduce a new configuration, or we should simply allow core threads to 
> timeout. I'm reading the OpenJDK source now to see what subtle differences 
> there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-15729) [s3a] stop treat fs.s3a.max.threads as the long-term minimum

[Sean Mackrory (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-15729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory updated HADOOP-15729:
---
Status: Open  (was: Patch Available)

> [s3a] stop treat fs.s3a.max.threads as the long-term minimum
> 
>
> Key: HADOOP-15729
> URL: https://issues.apache.org/jira/browse/HADOOP-15729
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Major
> Attachments: HADOOP-15729.001.patch, HADOOP-15729.002.patch
>
>
> A while ago the s3a connector started experiencing deadlocks because the AWS 
> SDK requires an unbounded threadpool. It places monitoring tasks on the work 
> queue before the tasks they wait on, so it's possible (has even happened with 
> larger-than-default threadpools) for the executor to become permanently 
> saturated and deadlock.
> So we started giving an unbounded threadpool executor to the SDK, and using a 
> bounded, blocking threadpool service for everything else S3A needs (although 
> currently that's only in the S3ABlockOutputStream). fs.s3a.max.threads then 
> only limits this threadpool, however we also specified fs.s3a.max.threads as 
> the number of core threads in the unbounded threadpool, which in hindsight is 
> pretty terrible.
> Currently those core threads do not timeout, so this is actually setting a 
> sort of minimum. Once that many tasks have been submitted, the threadpool 
> will be locked at that number until it bursts beyond that, but it will only 
> spin down that far. If fs.s3a.max.threads is set reasonably high and someone 
> uses a bunch of S3 buckets, they could easily have thousands of idle threads 
> constantly.
> We should either not use fs.s3a.max.threads for the corepool size and 
> introduce a new configuration, or we should simply allow core threads to 
> timeout. I'm reading the OpenJDK source now to see what subtle differences 
> there are between core threads and other threads if core threads can timeout.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on issue #1003: HADOOP-16384: Avoid inconsistencies between 
DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510098818
 
 
   test results against ireland
   with dynamo: 
https://gist.github.com/bgaborg/260d484aa564b806cade1f486006336a#file-dynamo-hadoop-16384-avoid-inconsistencies-between-ddb-and-s3
   
   with local: 
https://gist.github.com/bgaborg/260d484aa564b806cade1f486006336a#file-localmetadatastore-hadoop-16384-avoid-inconsistencies-between-ddb-and-s3
   
   What I'm generally afraid of is that we left localms, and I see a lot of 
test errors with this pr.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302093454
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PurgeS3GuardTable.java
 ##
 @@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardTable.serviceMain;
+import static 
org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.PARENT;
+
+/**
+ * Purge the S3Guard table of a FileSystem from all entries related to
+ * that table.
+ * Will fail if there is no table, or the store is in auth mode.
+ * 
+ *   hadoop org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardTable \
+ *   -force s3a://example-bucket/
+ * 
+ *
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class PurgeS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(PurgeS3GuardTable.class);
+
+  public static final String NAME = "PurgeS3GuardTable";
+
+  public static final String FORCE = "-force";
+
+  private static final String USAGE_MESSAGE = NAME
+  + " [-force] ";
+
+  private boolean force;
+
+  private long filesFound;
+  private long filesDeleted;
+
+  public PurgeS3GuardTable(final String name) {
+super(name);
+  }
+
+  public PurgeS3GuardTable() {
+this(NAME);
+  }
+
+  public PurgeS3GuardTable(
+  final S3AFileSystem filesystem,
+  final DynamoDBMetadataStore store,
 
 Review comment:
   Please add dynamo to the classname and create an issue that we have to 
implement/abstract these for localmetadatastore.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302068921
 
 

 ##
 File path: hadoop-tools/hadoop-aws/pom.xml
 ##
 @@ -186,7 +186,11 @@
 **/ITestS3AHuge*.java
 
 **/ITestDynamoDBMetadataStoreScale.java
+
 
 Review comment:
   This comment should be one line beneath the current 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302099673
 
 

 ##
 File path: hadoop-tools/hadoop-aws/src/test/resources/log4j.properties
 ##
 @@ -57,6 +57,8 @@ log4j.logger.org.apache.hadoop.ipc.Server=WARN
 #log4j.logger.org.apache.hadoop.fs.s3a.Listing=INFO
 # Log S3Guard classes
 #log4j.logger.org.apache.hadoop.fs.s3a.s3guard=DEBUG
+# if set to debug, this will log the PUT/DELETE operations on a store
+log4j.logger.org.apache.hadoop.fs.s3a.s3guard.Operations=DEBUG
 
 Review comment:
   we may don't want to log on debug for everybody.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302110128
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java
 ##
 @@ -172,6 +174,23 @@ public void setup() throws Exception {
 
   @Override
   public void teardown() throws Exception {
+if (ddbms != null) {
+  S3GuardTableAccess tableAccess = new S3GuardTableAccess(ddbms);
+  ExpressionSpecBuilder builder = new ExpressionSpecBuilder();
+  builder.withKeyCondition(
 
 Review comment:
   use withCondition here instead of key, so you don't have to filter for the 
VersionMarker


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302064355
 
 

 ##
 File path: 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java
 ##
 @@ -185,28 +191,57 @@ public void testListEmptyRootDirectory() throws 
IOException {
 for (FileStatus status : statuses) {
   ContractTestUtils.assertDeleted(fs, status.getPath(), true);
 }
-assertEquals("listStatus on empty root-directory returned a non-empty 
list",
-0, fs.listStatus(root).length);
-assertFalse("listFiles(/, false).hasNext",
-fs.listFiles(root, false).hasNext());
-assertFalse("listFiles(/, true).hasNext",
-fs.listFiles(root, true).hasNext());
-assertFalse("listLocatedStatus(/).hasNext",
-fs.listLocatedStatus(root).hasNext());
+FileStatus[] list1 = fs.listStatus(root);
 
 Review comment:
   nit: please use some name which is easier to understand eg. rootListStatus


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302071233
 
 

 ##
 File path: hadoop-tools/hadoop-aws/pom.xml
 ##
 @@ -215,15 +219,21 @@
   
   
   
-**/ITestS3AContractRootDir.java
 **/ITestS3AFileContextStatistics.java
+
 **/ITestS3AHuge*.java
+
 **/ITestS3AEncryptionSSEC*.java
 
 **/ITestDynamoDBMetadataStoreScale.java
 
 
 **/ITestTerasort*.java
+
+**/ITest*CommitMRJob.java
 
 Review comment:
   Same: we had a lot of testMRJob failures. Are those fail because of OoME?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302074925
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java
 ##
 @@ -538,7 +538,7 @@ public CompleteMultipartUploadResult commitUpload(
   public BulkOperationState initiateCommitOperation(
   Path path) throws IOException {
 return S3Guard.initiateBulkWrite(owner.getMetadataStore(),
-BulkOperationState.OperationType.Put, path);
 
 Review comment:
   This was a bug that the operationtype was put instead of Commit, right?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302096415
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
 ##
 @@ -281,22 +289,6 @@ private void deleteAllMetadata() throws IOException {
   public static void deleteMetadataUnderPath(final DynamoDBMetadataStore ms,
   final Path path, final boolean suppressErrors) throws IOException {
 ThrottleTracker throttleTracker = new ThrottleTracker(ms);
-try (DurationInfo ignored = new DurationInfo(LOG, true, "prune")) {
 
 Review comment:
   Why did you remove this? We do this operation on a separate testing table, 
just for this test, so it won't affect any other.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302066096
 
 

 ##
 File path: 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractRootDirectoryTest.java
 ##
 @@ -185,28 +191,57 @@ public void testListEmptyRootDirectory() throws 
IOException {
 for (FileStatus status : statuses) {
   ContractTestUtils.assertDeleted(fs, status.getPath(), true);
 }
-assertEquals("listStatus on empty root-directory returned a non-empty 
list",
-0, fs.listStatus(root).length);
-assertFalse("listFiles(/, false).hasNext",
-fs.listFiles(root, false).hasNext());
-assertFalse("listFiles(/, true).hasNext",
-fs.listFiles(root, true).hasNext());
-assertFalse("listLocatedStatus(/).hasNext",
-fs.listLocatedStatus(root).hasNext());
+FileStatus[] list1 = fs.listStatus(root);
+assertEquals("listStatus on empty root-directory returned found: "
++ join("\n", list1),
+0, list1.length);
+assertNoElements("listFiles(/, false)",
+fs.listFiles(root, false));
+assertNoElements("listFiles(/, true)",
+fs.listFiles(root, true));
+assertNoElements("listLocatedStatus(/)",
+fs.listLocatedStatus(root));
 assertIsDirectory(root);
   }
 
+  /**
+   * Assert that an iterator has no elements; the raised exception
+   * will include the element list.
+   * @param operation operation for assertion text.
+   * @param iter iterator
+   * @throws IOException failure retrieving the values.
+   */
+  protected void assertNoElements(String operation,
+  RemoteIterator iter) throws IOException {
+List resultList = toList(iter);
+if (!resultList.isEmpty()) {
+  fail("Expected no results from " + operation + ", but got "
 
 Review comment:
     good that we will have more description


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302069432
 
 

 ##
 File path: hadoop-tools/hadoop-aws/pom.xml
 ##
 @@ -186,7 +186,11 @@
 **/ITestS3AHuge*.java
 
 **/ITestDynamoDBMetadataStoreScale.java
+
 
 Review comment:
   Also, if it uses that much ram we should create an upstream issue for this 
to solve this.
   For my note: we had a LOT of testMRJob failures, so excluding this will not 
mean that they will go away, it will still pop up during the sequential run.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302096634
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java
 ##
 @@ -921,43 +913,6 @@ private DDBPathMetadata verifyInAncestor(AncestorState 
state,
 return md;
   }
 
-  @Test
 
 Review comment:
   Why did you remove this test?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302088282
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DumpS3GuardTable.java
 ##
 @@ -0,0 +1,691 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.Listing;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ALocatedFileStatus;
+import org.apache.hadoop.fs.s3a.S3ListRequest;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.S3AUtils.ACCEPT_ALL;
+
+/**
+ * This is a low-level diagnostics entry point which does a CVE/TSV dump of
+ * the DDB state.
+ * As it also lists the filesystem, it actually changes the state of the store
+ * during the operation.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class DumpS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(DumpS3GuardTable.class);
+
+  /**
+   * Application name.
+   */
+  public static final String NAME = "DumpS3GuardTable";
+
+  /**
+   * Usage.
+   */
+  private static final String USAGE_MESSAGE = NAME
+  + "  ";
+
+  /**
+   * Suffix for the flat list: {@value}.
+   */
+  public static final String FLAT_CSV = "-flat.csv";
+
+  /**
+   * Suffix for the raw S3 dump: {@value}.
+   */
+  public static final String RAW_CSV = "-s3.csv";
+
+  /**
+   * Suffix for the DDB scan: {@value}.
+   */
+  public static final String SCAN_CSV = "-scan.csv";
+
+  /**
+   * Suffix for the second DDB scan: : {@value}.
+   */
+  public static final String SCAN2_CSV = "-scan-2.csv";
+
+  /**
+   * Suffix for the treewalk scan of the S3A Filesystem: {@value}.
+   */
+  public static final String TREE_CSV = "-tree.csv";
+
+  /**
+   * Suffix for a recursive treewalk through the metastore: {@value}.
+   */
+  public static final String STORE_CSV = "-store.csv";
+
+  /**
+   * Path in the local filesystem to save the data.
+   */
+  protected String destPath;
+
+  /**
+   * Instantiate.
+   * @param name application name.
+   */
+  public DumpS3GuardTable(final String name) {
+super(name);
+  }
+
+  /**
+   * Instantiate with default name.
+   */
+  public DumpS3GuardTable() {
+this(NAME);
+  }
+
+  /**
+   * Bind to a specific FS + store.
+   * @param fs filesystem
+   * @param store metastore to use
+   * @param destFile the base filename for output
+   * @param uri URI of store -only needed if FS is null.
+   */
+  public DumpS3GuardTable(
+  final S3AFileSystem fs,
+  final DynamoDBMetadataStore store,
+  final File destFile,
+  final URI uri) {
+super(NAME, fs, store, uri);
+this.destPath = destFile.getAbsolutePath();
+  }
+
+  /**
+   * Bind to the argument list, including validating the CLI
+   * @throws Exception 

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302098848
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardRootOperations.java
 ##
 @@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import java.io.File;
+import java.net.URI;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import org.assertj.core.api.Assertions;
+import org.junit.FixMethodOrder;
+import org.junit.Test;
+import org.junit.runners.MethodSorters;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.contract.ContractTestUtils;
+import org.apache.hadoop.fs.s3a.AbstractS3ATestBase;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ATestUtils;
+import org.apache.hadoop.fs.s3a.impl.StoreContext;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.Constants.ENABLE_MULTI_DELETE;
+import static 
org.apache.hadoop.fs.s3a.Constants.S3GUARD_DDB_BACKGROUND_SLEEP_MSEC_KEY;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.assume;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName;
+import static org.apache.hadoop.fs.s3a.S3ATestUtils.removeBucketOverrides;
+import static org.apache.hadoop.fs.s3a.S3AUtils.applyLocatedFiles;
+
+/**
+ * This test run against the root of the FS, and operations which span the DDB
+ * table and the filesystem.
+ * For this reason, these tests are executed in the sequential phase of the
+ * integration tests.
+ * 
+ * The tests only run if DynamoDB is the metastore.
+ */
+@FixMethodOrder(MethodSorters.NAME_ASCENDING)
+public class ITestS3GuardRootOperations extends AbstractS3ATestBase {
+
+  private StoreContext storeContext;
+
+  private String fsUriStr;
+
+  private DynamoDBMetadataStore metastore;
 
 Review comment:
   Add dynamo to the class name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302090898
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DumpS3GuardTable.java
 ##
 @@ -0,0 +1,691 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.Closeable;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+
+import com.amazonaws.services.dynamodbv2.xspec.ExpressionSpecBuilder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.fs.s3a.Listing;
+import org.apache.hadoop.fs.s3a.S3AFileStatus;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.fs.s3a.S3ALocatedFileStatus;
+import org.apache.hadoop.fs.s3a.S3ListRequest;
+import org.apache.hadoop.service.Service;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+import org.apache.hadoop.service.launcher.ServiceLauncher;
+import org.apache.hadoop.util.DurationInfo;
+import org.apache.hadoop.util.ExitUtil;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static org.apache.hadoop.fs.s3a.S3AUtils.ACCEPT_ALL;
+
+/**
+ * This is a low-level diagnostics entry point which does a CVE/TSV dump of
+ * the DDB state.
+ * As it also lists the filesystem, it actually changes the state of the store
+ * during the operation.
+ */
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public class DumpS3GuardTable extends AbstractS3GuardDiagnostic {
+
+  private static final Logger LOG =
+  LoggerFactory.getLogger(DumpS3GuardTable.class);
+
+  /**
+   * Application name.
+   */
+  public static final String NAME = "DumpS3GuardTable";
+
+  /**
+   * Usage.
+   */
+  private static final String USAGE_MESSAGE = NAME
+  + "  ";
+
+  /**
+   * Suffix for the flat list: {@value}.
+   */
+  public static final String FLAT_CSV = "-flat.csv";
+
+  /**
+   * Suffix for the raw S3 dump: {@value}.
+   */
+  public static final String RAW_CSV = "-s3.csv";
+
+  /**
+   * Suffix for the DDB scan: {@value}.
+   */
+  public static final String SCAN_CSV = "-scan.csv";
+
+  /**
+   * Suffix for the second DDB scan: : {@value}.
+   */
+  public static final String SCAN2_CSV = "-scan-2.csv";
+
+  /**
+   * Suffix for the treewalk scan of the S3A Filesystem: {@value}.
+   */
+  public static final String TREE_CSV = "-tree.csv";
+
+  /**
+   * Suffix for a recursive treewalk through the metastore: {@value}.
+   */
+  public static final String STORE_CSV = "-store.csv";
+
+  /**
+   * Path in the local filesystem to save the data.
+   */
+  protected String destPath;
+
+  /**
+   * Instantiate.
+   * @param name application name.
+   */
+  public DumpS3GuardTable(final String name) {
+super(name);
+  }
+
+  /**
+   * Instantiate with default name.
+   */
+  public DumpS3GuardTable() {
+this(NAME);
+  }
+
+  /**
+   * Bind to a specific FS + store.
+   * @param fs filesystem
+   * @param store metastore to use
+   * @param destFile the base filename for output
+   * @param uri URI of store -only needed if FS is null.
+   */
+  public DumpS3GuardTable(
+  final S3AFileSystem fs,
+  final DynamoDBMetadataStore store,
+  final File destFile,
+  final URI uri) {
+super(NAME, fs, store, uri);
+this.destPath = destFile.getAbsolutePath();
+  }
+
+  /**
+   * Bind to the argument list, including validating the CLI
+   * @throws Exception 

[GitHub] [hadoop] bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

bgaborg commented on a change in pull request #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302085607
 
 

 ##
 File path: 
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardDiagnostic.java
 ##
 @@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.s3a.s3guard;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.List;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.s3a.S3AFileSystem;
+import org.apache.hadoop.service.launcher.AbstractLaunchableService;
+import org.apache.hadoop.service.launcher.LauncherExitCodes;
+import org.apache.hadoop.service.launcher.ServiceLaunchException;
+
+import static org.apache.hadoop.service.launcher.LauncherExitCodes.EXIT_USAGE;
+
+/**
+ * Entry point for diagnostics operations.
+ */
+public class AbstractS3GuardDiagnostic extends AbstractLaunchableService {
+
+  private S3AFileSystem filesystem;
+
+  private DynamoDBMetadataStore store;
+
+  private URI uri;
+
+  private List arguments;
+
+  /**
+   * Constructor.
+   * @param name entry point name.
+   */
+  public AbstractS3GuardDiagnostic(final String name) {
+super(name);
+  }
+
+  /**
+   * Constructor. If the store is set then that is the store for the operation,
+   * otherwise the filesystem's binding is used instead.
+   * @param name entry point name.
+   * @param filesystem filesystem
+   * @param store optional metastore.
+   * @param uri URI. Must be set if filesystem == null.
+   */
+  public AbstractS3GuardDiagnostic(
+  final String name,
+  @Nullable final S3AFileSystem filesystem,
+  @Nullable final DynamoDBMetadataStore store,
+  @Nullable final URI uri) {
+super(name);
+this.store = store;
+this.filesystem = filesystem;
+if (store == null) {
+  require(filesystem != null, "No filesystem or URI");
+  bindStore(filesystem);
+}
+if (uri == null) {
+  require(filesystem != null, "No filesystem or URI");
+  setUri(filesystem.getUri());
+} else {
+  setUri(uri);
+}
+  }
+
+  /**
+   * Require a condition to hold, otherwise an exception is thrown.
+   * @param condition condition to be true
+   * @param error text on failure.
+   * @throws ServiceLaunchException if the condition is not met
+   */
+  protected static void require(boolean condition, String error) {
+if (!condition) {
+  throw failure(error);
+}
+  }
+
+  /**
+   * Generate a failure exception for throwing.
+   * @param message message
+   * @param ex optional nested exception.
+   * @return an exception to throw
+   */
+  protected static ServiceLaunchException failure(String message, Throwable 
ex) {
+return new ServiceLaunchException(LauncherExitCodes.EXIT_FAIL, message, 
ex);
+  }
+
+  /**
+   * Generate a failure exception for throwing.
+   * @param message message
+   * @return an exception to throw
+   */
+  protected static ServiceLaunchException failure(String message) {
+return new ServiceLaunchException(LauncherExitCodes.EXIT_FAIL, message);
+  }
+
+  @Override
+  public Configuration bindArgs(final Configuration config,
+  final List args)
+  throws Exception {
+this.arguments = args;
+return super.bindArgs(config, args);
+  }
+
+  /**
+   * Get the argument list.
+   * @return the argument list.
+   */
+  protected List getArguments() {
+return arguments;
+  }
+
+  /**
+   * Bind to the store from a CLI argument.
+   * @param fsURI filesystem URI
+   * @throws IOException failure
+   */
+  protected void bindFromCLI(String fsURI)
+  throws IOException {
+Configuration conf = getConfig();
+setUri(fsURI);
+FileSystem fs = FileSystem.get(getUri(), conf);
+require(fs instanceof S3AFileSystem,
+"Not an S3A Filesystem:  " + fsURI);
+filesystem = (S3AFileSystem) fs;
+bindStore(filesystem);
+setUri(fs.getUri());
+  }
+
+  /**
+   * Binds the {@link #store} field to the metastore of
+   * the filesystem 

[GitHub] [hadoop] ajayydv commented on issue #973: HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar.

[GitBox]

ajayydv commented on issue #973: HDDS-1611. Evaluate ACL on volume bucket key 
and prefix to authorize access. Contributed by Ajay Kumar.
URL: https://github.com/apache/hadoop/pull/973#issuecomment-510092818
 
 
   failed junit tests look unrelated, TestWatchForCommit & 
TestFailureHandlingByClient fail for trunk as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] lokeshj1703 commented on a change in pull request #1072: HDDS-1766. ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed by Mukul Kumar Singh.

[GitBox]

lokeshj1703 commented on a change in pull request #1072: HDDS-1766. 
ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed 
by  Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1072#discussion_r302098470
 
 

 ##
 File path: 
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java
 ##
 @@ -554,12 +556,12 @@ private ByteString getCachedStateMachineData(Long 
logIndex, long term,
   }
 } catch (Exception e) {
   metrics.incNumReadStateMachineFails();
-  LOG.error("unable to read stateMachineData:" + e);
+  LOG.error("{} unable to read stateMachineData:", gid, e);
   return completeExceptionally(e);
 }
   }
 
-  private void updateLastApplied() {
+  private synchronized void updateLastApplied() {
 
 Review comment:
   @mukul1987  Thanks for working on this! The patch looks good to me.
   I think we can avoid using synchronized by using a concurrent sorted map. 
Each applyTransaction removes its corresponding entry from the map and we 
update lastApplied based on the first entry in the map?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities

[Adam Antal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882108#comment-16882108
 ] 

Adam Antal commented on HADOOP-15691:
-

Hi [~ste...@apache.org], is there any update on this?

Would be happy to see this committed for YARN-9607 and YARN-9525.

> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
>  Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch, 
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to 
> declare the capabilities under the path of a filesystem through both the 
> FileSystem and FileContext APIs
> This is needed for 
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside 
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different 
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't 
> hold, especially for object stores —and to efficiently use them, callers need 
> to be able to ask for specific features.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek merged pull request #1065: HDDS-1525. Mapreduce failure when using Hadoop 2.7.5

[GitBox]

elek merged pull request #1065: HDDS-1525. Mapreduce failure when using Hadoop 
2.7.5
URL: https://github.com/apache/hadoop/pull/1065
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek commented on issue #1065: HDDS-1525. Mapreduce failure when using Hadoop 2.7.5

[GitBox]

elek commented on issue #1065: HDDS-1525. Mapreduce failure when using Hadoop 
2.7.5
URL: https://github.com/apache/hadoop/pull/1065#issuecomment-510063256
 
 
   Thanks the review @xiaoyuyao (and the initial fix for the problem).
   
   I rebased the version, will commit it soon.
   
   Tested again locally and all the mr tests are passed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] hadoop-yetus commented on issue #1068: HDDS-1778. Fix existing blockade tests.

[GitBox]

hadoop-yetus commented on issue #1068: HDDS-1778. Fix existing blockade tests.
URL: https://github.com/apache/hadoop/pull/1068#issuecomment-510061876
 
 
   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime | Comment |
   |::|--:|:|:|
   | 0 | reexec | 58 | Docker mode activated. |
   ||| _ Prechecks _ |
   | +1 | dupname | 1 | No case conflicting files found. |
   | 0 | shelldocs | 0 | Shelldocs was not available. |
   | +1 | @author | 0 | The patch does not contain any @author tags. |
   | +1 | test4tests | 0 | The patch appears to include 19 new or modified test 
files. |
   ||| _ trunk Compile Tests _ |
   | 0 | mvndep | 47 | Maven dependency ordering for branch |
   | +1 | mvninstall | 526 | trunk passed |
   | +1 | compile | 254 | trunk passed |
   | +1 | mvnsite | 0 | trunk passed |
   | -1 | pylint | 6 | Error running pylint. Please check pylint stderr files. |
   | +1 | shadedclient | 768 | branch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 153 | trunk passed |
   ||| _ Patch Compile Tests _ |
   | 0 | mvndep | 22 | Maven dependency ordering for patch |
   | +1 | mvninstall | 448 | the patch passed |
   | +1 | compile | 258 | the patch passed |
   | +1 | javac | 258 | the patch passed |
   | +1 | mvnsite | 0 | the patch passed |
   | -1 | pylint | 12 | Error running pylint. Please check pylint stderr files. 
|
   | +1 | pylint | 12 | There were no new pylint issues. |
   | +1 | shellcheck | 1 | There were no new shellcheck issues. |
   | +1 | whitespace | 0 | The patch has no whitespace issues. |
   | +1 | shadedclient | 654 | patch has no errors when building and testing 
our client artifacts. |
   | +1 | javadoc | 164 | the patch passed |
   ||| _ Other Tests _ |
   | +1 | unit | 325 | hadoop-hdds in the patch passed. |
   | -1 | unit | 1983 | hadoop-ozone in the patch failed. |
   | +1 | asflicense | 59 | The patch does not generate ASF License warnings. |
   | | | 5952 | |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
   |   | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | Client=17.05.0-ce Server=17.05.0-ce base: 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1068 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient shellcheck shelldocs pylint |
   | uname | Linux a5c1c7fe53b7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | personality/hadoop.sh |
   | git revision | trunk / e0fa104 |
   | Default Java | 1.8.0_212 |
   | pylint | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/artifact/out/branch-pylint-stderr.txt
 |
   | pylint | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/artifact/out/patch-pylint-stderr.txt
 |
   | unit | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/artifact/out/patch-unit-hadoop-ozone.txt
 |
   |  Test Results | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/testReport/ |
   | Max. process+thread count | 4113 (vs. ulimit of 5500) |
   | modules | C: hadoop-ozone/fault-injection-test/network-tests 
hadoop-ozone/dist U: hadoop-ozone |
   | Console output | 
https://builds.apache.org/job/hadoop-multibranch/job/PR-1068/1/console |
   | versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 pylint=1.9.2 |
   | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bshashikant opened a new pull request #1073: HDDS-1780. TestFailureHandlingByClient tests are flaky.

[GitBox]

bshashikant opened a new pull request #1073: HDDS-1780. 
TestFailureHandlingByClient tests are flaky.
URL: https://github.com/apache/hadoop/pull/1073
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bshashikant commented on issue #1072: HDDS-1766. ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed by Mukul Kumar Singh.

[GitBox]

bshashikant commented on issue #1072: HDDS-1766. ContainerStateMachine is 
unable to increment lastAppliedTermIndex. Contributed by  Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1072#issuecomment-510045856
 
 
   Looks good. Pending test results.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] mukul1987 opened a new pull request #1072: HDDS-1766. ContainerStateMachine is unable to increment lastAppliedTermIndex. Contributed by Mukul Kumar Singh.

[GitBox]

mukul1987 opened a new pull request #1072: HDDS-1766. ContainerStateMachine is 
unable to increment lastAppliedTermIndex. Contributed by  Mukul Kumar Singh.
URL: https://github.com/apache/hadoop/pull/1072
 
 
   The last applied term index wasn't updated as the method is called by 
multiple executors and the method was not synchronized.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran edited a comment on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran edited a comment on issue #1003: HADOOP-16384: Avoid 
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510040326
 
 
   Got a failure when testing this patch on top of an internal branch.
   ```
   [ERROR] 
testListingReturnsVersionMetadata(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
  Time elapsed: 8.835 s  <<< FAILURE!
   java.lang.AssertionError: expected:<1> but was:<17>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListingReturnsVersionMetadata(ITestS3GuardListConsistency.java:605)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
   ```
   
   This is close enough to look related to something. 
   
   ~~The assert is that the #of entries from a getFileStatus was 1, not 17, 
which could again be implicitly a hint that a dir was considered empty so 
getFileStatus returned the empty dir marker, not children~~
   
   _no, it's that we expected one entry and got 17 back_
   
   need more detail on the failure, which assertj can help with


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies between DDB and S3

[GitBox]

steveloughran commented on issue #1003: HADOOP-16384: Avoid inconsistencies 
between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#issuecomment-510040326
 
 
   Got a failure when testing this patch on top of an internal branch.
   ```
   [ERROR] 
testListingReturnsVersionMetadata(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency)
  Time elapsed: 8.835 s  <<< FAILURE!
   java.lang.AssertionError: expected:<1> but was:<17>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListingReturnsVersionMetadata(ITestS3GuardListConsistency.java:605)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
   ```
   
   This is close enough to look related to something. The assert is that the 
#of entries from a getFileStatus was 1, not 17, which could again be 
implicitaly a hint that a dir was considered empty so getFileStatus returned 
the empty dir marker, not children.
   
   need more detail on the failure, which assertj can help with


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek commented on issue #1070: HDDS-1725. pv-test example to test csi is not working

[GitBox]

elek commented on issue #1070: HDDS-1725. pv-test example to test csi is not 
working
URL: https://github.com/apache/hadoop/pull/1070#issuecomment-510039418
 
 
   The mentioned problems are fixed.
   
   ozone-csi is created before HDDS-1646 (persistence support). Instead of 
having ozone-csi I generated the csi examples to the subdirectories of ozone 
AND ozone-dev. With this approach the csi can be tested with or without real 
persistence.
   
   I also fixed the pv test. Earlier nginx container has a permission problem. 
The simple python test doesn't have such problem, but the permission handling 
of the csi mounting should be tested later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-16415) Speed up S3A test runs

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881996#comment-16881996
 ] 

Steve Loughran commented on HADOOP-16415:
-

All the tests of S3A committers are slow because they spin up a yarn cluster 
each (slow) then the actual MR jobs

Proposal: rework so that the miniyarn cluster comes up once, with each job 
running as its own test suite. This is complex for the Terasort tests  (only 
run on scale) as they are implemented as ordered set of test cases; we'd need 
to copy the base suite and rework to run terasort for dir and magic committers 
in sequence

The ITest*CommitProtocol suites are slow too; 300-400+ seconds each
 * do we need the staging one?
 * what can we do for better parallelism here? Even if its just faster creation 
of temp files, deletion etc


> Speed up S3A test runs
> --
>
> Key: HADOOP-16415
> URL: https://issues.apache.org/jira/browse/HADOOP-16415
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Major
>
> S3A Test runs are way too slow.
> Speed them by
> * reducing test setup/teardown costs
> * eliminating obsolete test cases
> * merge small tests into larger ones.
> One thing i see is that the main S3A test cases create and destroy new FS 
> instances; There's both a setup and teardown cost there, but it does 
> guarantee better isolation.
> Maybe if we know all test cases in a specific suite need the same options, we 
> can manage that better; demand create the FS but only delete it in an 
> @Afterclass method. That'd give us the OO-inheritance based setup of tests, 
> but mean only one instance is done per suite



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bshashikant opened a new pull request #1071: HDDS-1779. TestWatchForCommit tests are flaky.

[GitBox]

bshashikant opened a new pull request #1071: HDDS-1779. TestWatchForCommit 
tests are flaky.
URL: https://github.com/apache/hadoop/pull/1071
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek opened a new pull request #1070: HDDS-1725. pv-test example to test csi is not working

[GitBox]

elek opened a new pull request #1070: HDDS-1725. pv-test example to test csi is 
not working
URL: https://github.com/apache/hadoop/pull/1070
 
 
   [~rmaruthiyodan] reported two problems regarding to the pv-test example in 
csi examples folder.
   
   pv-test folder contains an example nginx deployment which can use an ozone 
PVC/PV to publish content of a folder via http.
   
   Two problems are identified:
* The label based matching filter of service doesn't point to the nginx 
deployment
* The configmap mounting is missing from nginx deployment
   
   See: https://issues.apache.org/jira/browse/HDDS-1725


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek closed pull request #1059: HDDS-1764. Fix hidden errors in acceptance tests

[GitBox]

elek closed pull request #1059: HDDS-1764. Fix hidden errors in acceptance tests
URL: https://github.com/apache/hadoop/pull/1059
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bshashikant closed pull request #1069: HDDS-1779. TestWatchForCommit tests are flaky.

[GitBox]

bshashikant closed pull request #1069: HDDS-1779. TestWatchForCommit tests are 
flaky.
URL: https://github.com/apache/hadoop/pull/1069
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] bshashikant opened a new pull request #1069: HDDS-1779. TestWatchForCommit tests are flaky.

[GitBox]

bshashikant opened a new pull request #1069: HDDS-1779. TestWatchForCommit 
tests are flaky.
URL: https://github.com/apache/hadoop/pull/1069
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] elek commented on issue #1059: HDDS-1764. Fix hidden errors in acceptance tests

[GitBox]

elek commented on issue #1059: HDDS-1764. Fix hidden errors in acceptance tests
URL: https://github.com/apache/hadoop/pull/1059#issuecomment-510025594
 
 
   Thanks the review @xiaoyuyao I rebased it and will merge it soon.
   
   I cross-checked it with your PR (#1066) and found that I didn't notice 
ozonesecure-mr (which was also fixed in your patch). I also added this line of 
change.
   
   (ps: after committing the other patch with ozone-mr acceptance tests, it can 
be copied to support ozonesecure-mr to keep it stable)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] ChenSammi commented on issue #931: HDDS-1586. Allow Ozone RPC client to read with topology awareness.

[GitBox]

ChenSammi commented on issue #931: HDDS-1586. Allow Ozone RPC client to read 
with topology awareness.
URL: https://github.com/apache/hadoop/pull/931#issuecomment-510022820
 
 
   Thanks @xiaoyuyao  for reviewing the code. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] ChenSammi commented on a change in pull request #931: HDDS-1586. Allow Ozone RPC client to read with topology awareness.

[GitBox]

ChenSammi commented on a change in pull request #931: HDDS-1586. Allow Ozone 
RPC client to read with topology awareness.
URL: https://github.com/apache/hadoop/pull/931#discussion_r302015013
 
 

 ##
 File path: 
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
 ##
 @@ -2347,7 +2347,7 @@ public OmKeyInfo lookupKey(OmKeyArgs args) throws 
IOException {
 boolean auditSuccess = true;
 try {
   metrics.incNumKeyLookups();
-  return keyManager.lookupKey(args);
+  return keyManager.lookupKey(args, getClientAddress());
 
 Review comment:
   No, use Ip address or hostname depends on the value of   
"dfs.datanode.use.datanode.hostname" property. If it's true, then use hostname, 
if it's false, the use Ipaddress. The default value of this property is false. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org