[jira] [Updated] (HADOOP-16866) Upgrade spotbugs' version
[ https://issues.apache.org/jira/browse/HADOOP-16866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated HADOOP-16866: Description: [https://github.com/spotbugs/spotbugs/releases] spotbugs 4.0.0 is now released. We can upgrade spotbugs' version to: 1. 3.1.12 (conservative option) 2. 4.0.0 (which might includes incompatible changes, according to the migration guide: [https://spotbugs.readthedocs.io/en/stable/migration.html]) Step by step approach is also acceptable. was: [https://github.com/spotbugs/spotbugs/releases] spotbugs 4.0.0 is now released. We can upgrade spotbugs' version to: 1. 3.1.12 (conservative option) 2. 4.0.0 (which might includes incompatible changes) Step by step approach is also acceptable. > Upgrade spotbugs' version > - > > Key: HADOOP-16866 > URL: https://issues.apache.org/jira/browse/HADOOP-16866 > Project: Hadoop Common > Issue Type: Task >Reporter: Tsuyoshi Ozawa >Priority: Minor > > [https://github.com/spotbugs/spotbugs/releases] > spotbugs 4.0.0 is now released. > > We can upgrade spotbugs' version to: > 1. 3.1.12 (conservative option) > 2. 4.0.0 (which might includes incompatible changes, according to the > migration guide: [https://spotbugs.readthedocs.io/en/stable/migration.html]) > > Step by step approach is also acceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16866) Upgrade spotbugs' version
Tsuyoshi Ozawa created HADOOP-16866: --- Summary: Upgrade spotbugs' version Key: HADOOP-16866 URL: https://issues.apache.org/jira/browse/HADOOP-16866 Project: Hadoop Common Issue Type: Task Reporter: Tsuyoshi Ozawa [https://github.com/spotbugs/spotbugs/releases] spotbugs 4.0.0 is now released. We can upgrade spotbugs' version to: 1. 3.1.12 (conservative option) 2. 4.0.0 (which might includes incompatible changes) Step by step approach is also acceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[
https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038675#comment-17038675
]
Hudson commented on HADOOP-16759:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17961 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17961/])
HADOOP-16759. FileSystem Javadocs to list what breaks on API changes (github:
rev a562942b05a29d29e4a4fe3df4f35e2a8e7d208d)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
> Filesystem openFile() builder to take a FileStatus param
>
>
> Key: HADOOP-16759
> URL: https://issues.apache.org/jira/browse/HADOOP-16759
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.0
>
>
> Let us pass in a file status to openFile() so that S3A & ABFS will skip
> their own HEAD requests just to see if a file is there, a normal file and get
> its length + etag,
> {code}
> CompletableFuture streamF = fs.openFile(stat.getPath())
> .withFileStatus(stat).build()
> {code}
> code opening files off a listing of everything in a directory can eliminate a
> lot of requests here.
> Also: change the specification of openFile's completable future to say
> "returned stream may only raise FNFE, access restrictions on the first read"
> That is: it's not just potentially an async open, it's possibly lazy
> evaluated entirely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038666#comment-17038666
]
Hudson commented on HADOOP-15961:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17960 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17960/])
HADOOP-15961. S3A committers: make sure there's regular progress() (stevel: rev
c77fc6971b5194c9dae184703caa87da271a85eb)
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ProgressCounter.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/staging/StagingCommitter.java
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch,
> HADOOP-15961-003.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1850: HADOOP-16759. FileSystem Javadocs to list what breaks.
steveloughran commented on issue #1850: HADOOP-16759. FileSystem Javadocs to list what breaks. URL: https://github.com/apache/hadoop/pull/1850#issuecomment-587175637 merged this myself as something which is a doc-only followon from the openFile().withStatus command; turns out Hive was overriding the protected method I changed. with this patch in, nobody can say they weren't warned. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran merged pull request #1850: HADOOP-16759. FileSystem Javadocs to list what breaks.
steveloughran merged pull request #1850: HADOOP-16759. FileSystem Javadocs to list what breaks. URL: https://github.com/apache/hadoop/pull/1850 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15961) S3A committers: make sure there's regular progress() calls
[
https://issues.apache.org/jira/browse/HADOOP-15961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15961:
Fix Version/s: 3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
fixed in trunk; thanks for the patch, apologies for the very late
review...catching up on all of these before we ship!
> S3A committers: make sure there's regular progress() calls
> --
>
> Key: HADOOP-15961
> URL: https://issues.apache.org/jira/browse/HADOOP-15961
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: lqjacklee
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-15961-001.patch, HADOOP-15961-002.patch,
> HADOOP-15961-003.patch
>
>
> MAPREDUCE-7164 highlights how inside job/task commit more context.progress()
> callbacks are needed, just for HDFS.
> the S3A committers should be reviewed similarly.
> At a glance:
> StagingCommitter.commitTaskInternal() is at risk if a task write upload
> enough data to the localfs that the upload takes longer than the timeout.
> it should call progress it every single file commits, or better: modify
> {{uploadFileToPendingCommit}} to take a Progressable for progress callbacks
> after every part upload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran closed pull request #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls.
steveloughran closed pull request #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls. URL: https://github.com/apache/hadoop/pull/1848 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls.
steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls. URL: https://github.com/apache/hadoop/pull/1848#issuecomment-587172193 I'm happy with the patch, yetus is happy. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a change in pull request #1851: HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries - WIP!!
steveloughran commented on a change in pull request #1851: HADOOP-16858.
S3Guard fsck: Add option to remove orphaned entries - WIP!!
URL: https://github.com/apache/hadoop/pull/1851#discussion_r380378809
##
File path:
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardFsckViolationHandler.java
##
@@ -152,6 +184,12 @@ public NoMetadataEntry(S3GuardFsck.ComparePair
comparePair) {
public String getError() {
return "No PathMetadata for this path in the MS.";
}
+
+@Override
+public String fixViolation(S3AFileSystem fs, DynamoDBMetadataStore ddbms) {
Review comment:
you could probably do this in the base class for now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16861) Trunk is broken on OS X
[
https://issues.apache.org/jira/browse/HADOOP-16861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein resolved HADOOP-16861.
Resolution: Invalid
The problem was in the local repository settings on my local Mac.
> Trunk is broken on OS X
> ---
>
> Key: HADOOP-16861
> URL: https://issues.apache.org/jira/browse/HADOOP-16861
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Priority: Major
>
> Trunk does not compiler on Mac OS X with Java "1.8.0_242".
> {code:bash}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project hadoop-maven-plugins: Compilation failure
> [ERROR]
> /home/ahussein/hadoop/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/protoc/ProtocRunner.java:[16,39]
> package com.fasterxml.jackson.core.type does not exist
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038586#comment-17038586
]
Hudson commented on HADOOP-13666:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17959 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17959/])
HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in
(ayushsaxena: rev 84f763884021980c456e2ebc21c1a1c1b18fec6c)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetworkTopology.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/net/TestNetworkTopology.java
> Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
>
>
> Key: HADOOP-13666
> URL: https://issues.apache.org/jira/browse/HADOOP-13666
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.3
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-13666.000.patch, HADOOP-13666.001.patch
>
>
> Right now, the counting of nodes in {{NetworkTopology}} assumes the
> exclusions are leaves. We should count the proper number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HADOOP-13666:
--
Fix Version/s: 3.3.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
>
>
> Key: HADOOP-13666
> URL: https://issues.apache.org/jira/browse/HADOOP-13666
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.3
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-13666.000.patch, HADOOP-13666.001.patch
>
>
> Right now, the counting of nodes in {{NetworkTopology}} assumes the
> exclusions are leaves. We should count the proper number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038583#comment-17038583
]
Ayush Saxena commented on HADOOP-13666:
---
Committed to trunk.
Thanx [~elgoiri] for the contribution!!!
> Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
>
>
> Key: HADOOP-13666
> URL: https://issues.apache.org/jira/browse/HADOOP-13666
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.3
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HADOOP-13666.000.patch, HADOOP-13666.001.patch
>
>
> Right now, the counting of nodes in {{NetworkTopology}} assumes the
> exclusions are leaves. We should count the proper number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038563#comment-17038563
]
Íñigo Goiri commented on HADOOP-13666:
--
Thanks for the review and tracking this [~ayushtkn], do you mind pushing the
patch?
> Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
>
>
> Key: HADOOP-13666
> URL: https://issues.apache.org/jira/browse/HADOOP-13666
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.3
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HADOOP-13666.000.patch, HADOOP-13666.001.patch
>
>
> Right now, the counting of nodes in {{NetworkTopology}} assumes the
> exclusions are leaves. We should count the proper number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16865) Backport HADOOP-16152 to branch-2.10
Wei-Chiu Chuang created HADOOP-16865: Summary: Backport HADOOP-16152 to branch-2.10 Key: HADOOP-16865 URL: https://issues.apache.org/jira/browse/HADOOP-16865 Project: Hadoop Common Issue Type: Task Reporter: Wei-Chiu Chuang We should make the effort to migrate from Jetty 9.3 to 9.4 in branch-2.10. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1851: HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries - WIP!!
hadoop-yetus commented on issue #1851: HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries - WIP!! URL: https://github.com/apache/hadoop/pull/1851#issuecomment-587107658 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 26m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 59s | trunk passed | | +1 :green_heart: | compile | 0m 35s | trunk passed | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 40s | trunk passed | | +1 :green_heart: | shadedclient | 15m 12s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 29s | trunk passed | | +0 :ok: | spotbugs | 1m 0s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 58s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 29s | the patch passed | | +1 :green_heart: | javac | 0m 29s | the patch passed | | -0 :warning: | checkstyle | 0m 20s | hadoop-tools/hadoop-aws: The patch generated 2 new + 21 unchanged - 0 fixed = 23 total (was 21) | | +1 :green_heart: | mvnsite | 0m 32s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 40s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | | +1 :green_heart: | findbugs | 1m 2s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 84m 22s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1851/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1851 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 590010f072c2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 439d935 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1851/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1851/1/testReport/ | | Max. process+thread count | 447 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1851/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg opened a new pull request #1851: HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries - WIP!!
bgaborg opened a new pull request #1851: HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries - WIP!! URL: https://github.com/apache/hadoop/pull/1851 No tests yet. I'll add the test. Use this PR just to play with. Change-Id: I2cdb6601fea1d859b54370046b827ef06eb1107d This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16858) S3Guard fsck: Add option to remove orphaned entries
[ https://issues.apache.org/jira/browse/HADOOP-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16858: Summary: S3Guard fsck: Add option to remove orphaned entries (was: S3Guard fsck: Add option to prune orphaned entries) > S3Guard fsck: Add option to remove orphaned entries > --- > > Key: HADOOP-16858 > URL: https://issues.apache.org/jira/browse/HADOOP-16858 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Gabor Bota >Assignee: Gabor Bota >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038363#comment-17038363
]
Ayush Saxena commented on HADOOP-13666:
---
Test failures doesn't seems related.
Seems safe to me, and I can't think any reason, to hold this.
v001 LGTM +1
> Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
>
>
> Key: HADOOP-13666
> URL: https://issues.apache.org/jira/browse/HADOOP-13666
> Project: Hadoop Common
> Issue Type: Improvement
> Components: net
>Affects Versions: 2.7.3
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HADOOP-13666.000.patch, HADOOP-13666.001.patch
>
>
> Right now, the counting of nodes in {{NetworkTopology}} assumes the
> exclusions are leaves. We should count the proper number.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1850: HADOOP-16759. FileSystem Javadocs to list what breaks.
hadoop-yetus commented on issue #1850: HADOOP-16759. FileSystem Javadocs to list what breaks. URL: https://github.com/apache/hadoop/pull/1850#issuecomment-586987831 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 22s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 22m 1s | trunk passed | | +1 :green_heart: | compile | 18m 6s | trunk passed | | +1 :green_heart: | checkstyle | 0m 47s | trunk passed | | +1 :green_heart: | mvnsite | 1m 26s | trunk passed | | +1 :green_heart: | shadedclient | 17m 48s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 57s | trunk passed | | +0 :ok: | spotbugs | 2m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 2m 8s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 51s | the patch passed | | +1 :green_heart: | compile | 17m 26s | the patch passed | | +1 :green_heart: | javac | 17m 26s | the patch passed | | -0 :warning: | checkstyle | 0m 46s | hadoop-common-project/hadoop-common: The patch generated 1 new + 76 unchanged - 0 fixed = 77 total (was 76) | | +1 :green_heart: | mvnsite | 1m 26s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 17s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | | +1 :green_heart: | findbugs | 2m 16s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 9m 32s | hadoop-common in the patch passed. | | +1 :green_heart: | asflicense | 0m 48s | The patch does not generate ASF License warnings. | | | | 114m 55s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1850/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1850 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 89bfe8345eff 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 439d935 | | Default Java | 1.8.0_242 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1850/1/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1850/1/testReport/ | | Max. process+thread count | 3191 (vs. ulimit of 5500) | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1850/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls.
hadoop-yetus commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls. URL: https://github.com/apache/hadoop/pull/1848#issuecomment-586969029 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 25m 43s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 1s | The patch appears to include 3 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 24s | trunk passed | | +1 :green_heart: | compile | 0m 36s | trunk passed | | +1 :green_heart: | checkstyle | 0m 27s | trunk passed | | +1 :green_heart: | mvnsite | 0m 38s | trunk passed | | +1 :green_heart: | shadedclient | 15m 9s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 29s | trunk passed | | +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 56s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 32s | the patch passed | | +1 :green_heart: | compile | 0m 28s | the patch passed | | +1 :green_heart: | javac | 0m 28s | the patch passed | | +1 :green_heart: | checkstyle | 0m 18s | hadoop-tools/hadoop-aws: The patch generated 0 new + 7 unchanged - 1 fixed = 7 total (was 8) | | +1 :green_heart: | mvnsite | 0m 31s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 32s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 25s | the patch passed | | +1 :green_heart: | findbugs | 1m 1s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 23s | hadoop-aws in the patch passed. | | +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. | | | | 83m 33s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.6 Server=19.03.6 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1848/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1848 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux aad093542709 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 439d935 | | Default Java | 1.8.0_242 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1848/2/testReport/ | | Max. process+thread count | 420 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1848/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran opened a new pull request #1850: HADOOP-16759. FileSystem Javadocs to list what breaks.
steveloughran opened a new pull request #1850: HADOOP-16759. FileSystem Javadocs to list what breaks. URL: https://github.com/apache/hadoop/pull/1850 Followup to the main openFile().withStatus() patch. It turns out that this broke the hive builds, which was not well appreciated. This patch lists places to review in the hadoop codebase, and external projects where changes may cause problems. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls.
steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls. URL: https://github.com/apache/hadoop/pull/1848#issuecomment-586929389 checkstyle; new ProgressCounter needs reformatting This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls.
steveloughran commented on issue #1848: HADOOP-15961. S3A committers: make sure there's regular progress() calls. URL: https://github.com/apache/hadoop/pull/1848#issuecomment-586928960 failure ``` [ERROR] Tests run: 26, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 100.901 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.ITestAssumeRole [ERROR] testRestrictedCommitActions(org.apache.hadoop.fs.s3a.auth.ITestAssumeRole) Time elapsed: 2.808 s <<< FAILURE! java.lang.AssertionError: progress counter not expected. expected:<1> but was:<0> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:834) at org.junit.Assert.assertEquals(Assert.java:645) at org.apache.hadoop.fs.s3a.auth.ProgressCounter.assertCount(ProgressCounter.java:40) at org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.testRestrictedCommitActions(ITestAssumeRole.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13666) Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology
[
https://issues.apache.org/jira/browse/HADOOP-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038212#comment-17038212
]
Hadoop QA commented on HADOOP-13666:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 41s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}230m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.6 Server=19.03.6 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | HADOOP-13666 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12993615/HADOOP-13666.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f8635750b33b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 810783d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
|
