[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15682: -- Resolution: Fixed Status: Resolved (was: Patch Available) commit 3ea9074571b56f63ac04bb5b83f90dbdaec3aa24 Author: Thomas Marquardt Date: Fri Aug 24 01:28:17 2018 + HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. Contributed by Thomas Marquardt. > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591034#comment-16591034 ] Da Zhou commented on HADOOP-15682: -- Cool, it looks good to me. +1 > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9567) Provide auto-renewal for keytab based logins
[
https://issues.apache.org/jira/browse/HADOOP-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591024#comment-16591024
]
genericqa commented on HADOOP-9567:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 16 new + 253 unchanged - 0 fixed = 269 total (was 253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
46s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-9567 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936915/HADOOP-9567-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 80919f48d779 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ca29fb7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15089/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15089/testReport/ |
| Max. process+thread count | 1669 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[
https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590962#comment-16590962
]
genericqa commented on HADOOP-15682:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-15407 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
5s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} HADOOP-15407 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} HADOOP-15407 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} hadoop-tools/hadoop-azure: The patch generated 0 new
+ 0 unchanged - 2 fixed = 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
10s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15682 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936908/HADOOP-15682-HADOOP-15407-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 63e671463dce 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HADOOP-15407 / c05963b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15088/testReport/ |
| Max. process+thread count | 344 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15088/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ABFS: Add support
[jira] [Commented] (HADOOP-9567) Provide auto-renewal for keytab based logins
[ https://issues.apache.org/jira/browse/HADOOP-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590932#comment-16590932 ] Hrishikesh Gadre commented on HADOOP-9567: -- [~ghelmling] Are you planning to work on this jira? If not, I would like to assign it to myself. I have attached a patch against trunk (File: HADOOP-9567-001.patch) using your patch as a reference. This patch implements * A property to enable/disable auto-renewal for keytab based logins. By default the auto renewal is disabled for backwards compatibility. * Changed the implementation to use ExecutorService (instead of Thread) so as to facilitate the shutdown when the user logs out. * The current implementation for ticket cache based renewal spawns the thread as part of the getLoginUser() API in UGI class. This doesn't work for keytab based logins since loginUser reference is already initialized as part of the earlier invocation of loginByKeytab API. Hence for keytab based logins, the logic to spawn auto-renewal thread is implemented in loginByKeytab API. * New unit tests are added for this functionality as well as updated existing tests whenever necessary. [~daryn] could you please review this patch as you have made changes in this area recently ? > Provide auto-renewal for keytab based logins > > > Key: HADOOP-9567 > URL: https://issues.apache.org/jira/browse/HADOOP-9567 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Gary Helmling >Priority: Minor > Attachments: HADOOP-9567-001.patch, HADOOP-9567.branch-2.7.001.patch > > > We do a renewal for cached tickets (obtained via kinit before using a Hadoop > application) but we explicitly seem to avoid doing a renewal for keytab based > logins (done from within the client code) when we could do that as well via a > similar thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9567) Provide auto-renewal for keytab based logins
[ https://issues.apache.org/jira/browse/HADOOP-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hrishikesh Gadre updated HADOOP-9567: - Status: Patch Available (was: Open) > Provide auto-renewal for keytab based logins > > > Key: HADOOP-9567 > URL: https://issues.apache.org/jira/browse/HADOOP-9567 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Gary Helmling >Priority: Minor > Attachments: HADOOP-9567-001.patch, HADOOP-9567.branch-2.7.001.patch > > > We do a renewal for cached tickets (obtained via kinit before using a Hadoop > application) but we explicitly seem to avoid doing a renewal for keytab based > logins (done from within the client code) when we could do that as well via a > similar thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9567) Provide auto-renewal for keytab based logins
[ https://issues.apache.org/jira/browse/HADOOP-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hrishikesh Gadre updated HADOOP-9567: - Attachment: HADOOP-9567-001.patch > Provide auto-renewal for keytab based logins > > > Key: HADOOP-9567 > URL: https://issues.apache.org/jira/browse/HADOOP-9567 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Gary Helmling >Priority: Minor > Attachments: HADOOP-9567-001.patch, HADOOP-9567.branch-2.7.001.patch > > > We do a renewal for cached tickets (obtained via kinit before using a Hadoop > application) but we explicitly seem to avoid doing a renewal for keytab based > logins (done from within the client code) when we could do that as well via a > similar thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15582) Document ABFS
[
https://issues.apache.org/jira/browse/HADOOP-15582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Marquardt reassigned HADOOP-15582:
-
Assignee: Thomas Marquardt
> Document ABFS
> -
>
> Key: HADOOP-15582
> URL: https://issues.apache.org/jira/browse/HADOOP-15582
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/azure
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Thomas Marquardt
>Priority: Major
>
> Add documentation for abfs under
> {{hadoop-tools/hadoop-azure/src/site/markdown}}
> Possible topics include
> * intro to scheme
> * why abfs (link to MSDN, etc)
> * config options
> * switching from wasb/interop
> * troubleshooting
> testing.md should add a section on testing this stuff too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt reassigned HADOOP-15682: - Assignee: Thomas Marquardt > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15682: -- Attachment: HADOOP-15682-HADOOP-15407-001.patch Status: Patch Available (was: Open) Submitting patch 001 which adds StreamCapabilities support and tests for ABFS. It also fixes some javadoc and checkstyle errors. All tests pass using my US storage account: *Tests run: 265, Failures: 0, Errors: 0, Skipped: 11* *Tests run: 1, Failures: 0, Errors: 0, Skipped: 0* *Tests run: 863, Failures: 0, Errors: 0, Skipped: 262* *Tests run: 186, Failures: 0, Errors: 0, Skipped: 10* > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Priority: Major > Attachments: HADOOP-15682-HADOOP-15407-001.patch > > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15682: -- Description: Add support for the new StreamCapabilities interface. This work is similar to what was done for WASB [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. Also fix javadoc and checkstyle errors. was:Add support for the new StreamCapabilities interface. This work is similar to what was done for WASB [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > ABFS: Add support for StreamCapabilities > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Priority: Major > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15682) ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle
[ https://issues.apache.org/jira/browse/HADOOP-15682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15682: -- Summary: ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle (was: ABFS: Add support for StreamCapabilities) > ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle > > > Key: HADOOP-15682 > URL: https://issues.apache.org/jira/browse/HADOOP-15682 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3. >Reporter: Thomas Marquardt >Priority: Major > > Add support for the new StreamCapabilities interface. This work is similar > to what was done for WASB > [HADOOP-15677|https://jira.apache.org/jira/browse/HADOOP-15677]. > Also fix javadoc and checkstyle errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15692) ABFS: extensible support for custom oauth
Thomas Marquardt created HADOOP-15692: - Summary: ABFS: extensible support for custom oauth Key: HADOOP-15692 URL: https://issues.apache.org/jira/browse/HADOOP-15692 Project: Hadoop Common Issue Type: Sub-task Components: fs/azure Reporter: Thomas Marquardt ABFS supports oauth in various forms and needs to export interfaces for customization of FileSystem.getDelegationToken and getAccessToken. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Closed] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
[ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt closed HADOOP-15688. - commit c05963b14a7bf991f90293d6f90eab28fda1df16 Author: Thomas Marquardt Date: Thu Aug 23 20:43:52 2018 + HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. Contributed by Sean Mackrory. > ABFS: InputStream wrapped in FSDataInputStream twice > > > Key: HADOOP-15688 > URL: https://issues.apache.org/jira/browse/HADOOP-15688 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15688-HADOOP-15407-002.patch, > HADOOP-15688.001.patch > > > I can't read Parquet files from ABFS. It has 2 different implementations to > read seekable streams, and it'll use the one that uses ByteBuffer reads if it > can. It currently decides to use the ByteBuffer read implementation because > the FSDataInputStream it gets back wraps another FSDataInputStream, which > implements ByteBufferReadable. > That's not the most robust way to check that ByteBufferReads are supported by > the ultimately underlying InputStream, but it's unnecessary and probably a > mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
[ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15688: -- Resolution: Fixed Status: Resolved (was: Patch Available) commit c05963b14a7bf991f90293d6f90eab28fda1df16 Author: Thomas Marquardt Date: Thu Aug 23 20:43:52 2018 + HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. Contributed by Sean Mackrory. > ABFS: InputStream wrapped in FSDataInputStream twice > > > Key: HADOOP-15688 > URL: https://issues.apache.org/jira/browse/HADOOP-15688 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15688-HADOOP-15407-002.patch, > HADOOP-15688.001.patch > > > I can't read Parquet files from ABFS. It has 2 different implementations to > read seekable streams, and it'll use the one that uses ByteBuffer reads if it > can. It currently decides to use the ByteBuffer read implementation because > the FSDataInputStream it gets back wraps another FSDataInputStream, which > implements ByteBufferReadable. > That's not the most robust way to check that ByteBufferReads are supported by > the ultimately underlying InputStream, but it's unnecessary and probably a > mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15688) ABFS: InputStream wrapped in FSDataInputStream twice
[ https://issues.apache.org/jira/browse/HADOOP-15688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590787#comment-16590787 ] Thomas Marquardt commented on HADOOP-15688: --- I will fix the checkstyle issue and push the change. > ABFS: InputStream wrapped in FSDataInputStream twice > > > Key: HADOOP-15688 > URL: https://issues.apache.org/jira/browse/HADOOP-15688 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Major > Attachments: HADOOP-15688-HADOOP-15407-002.patch, > HADOOP-15688.001.patch > > > I can't read Parquet files from ABFS. It has 2 different implementations to > read seekable streams, and it'll use the one that uses ByteBuffer reads if it > can. It currently decides to use the ByteBuffer read implementation because > the FSDataInputStream it gets back wraps another FSDataInputStream, which > implements ByteBufferReadable. > That's not the most robust way to check that ByteBufferReads are supported by > the ultimately underlying InputStream, but it's unnecessary and probably a > mistake to double-wrap the InputStream, so let's not. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-12125) Retrying UnknownHostException on a proxy does not actually retry hostname resolution
[ https://issues.apache.org/jira/browse/HADOOP-12125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-12125: --- Assignee: John Zhuge (was: Rushabh S Shah) > Retrying UnknownHostException on a proxy does not actually retry hostname > resolution > > > Key: HADOOP-12125 > URL: https://issues.apache.org/jira/browse/HADOOP-12125 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Jason Lowe >Assignee: John Zhuge >Priority: Major > > When RetryInvocationHandler attempts to retry an UnknownHostException the > hostname fails to be resolved again. The InetSocketAddress in the > ConnectionId has cached the fact that the hostname is unresolvable, and when > the proxy tries to setup a new Connection object with that ConnectionId it > checks if the (cached) resolution result is unresolved and immediately throws. > The end result is we sleep and retry for no benefit. The hostname resolution > is never attempted again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590674#comment-16590674
]
genericqa commented on HADOOP-15679:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
8s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
33s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 115 unchanged - 1 fixed = 116 total (was 116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
37s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:a716388 |
| JIRA Issue | HADOOP-15679 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936856/HADOOP-15679-branch-2-001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 319d0b0bb442 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / c4e3df2 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15087/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15087/testReport/ |
| Max. process+thread count | 1458 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15087/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ShutdownHookManager shutdown time needs to be configurable & extended
> -
>
>
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590670#comment-16590670
]
Xiao Chen commented on HADOOP-14445:
Good to know, thanks for the update [~daryn].
I agree we can take advantage of the kind-less token lookup to build the silver
bullet. Was trying to make the new behavior correctly based on token kind in
the last patch, but looking forward to the magical patch before I really hold
any opinion.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14946) S3Guard testPruneCommandCLI can fail
[
https://issues.apache.org/jira/browse/HADOOP-14946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14946:
---
Fix Version/s: 3.2.0
> S3Guard testPruneCommandCLI can fail
>
>
> Key: HADOOP-14946
> URL: https://issues.apache.org/jira/browse/HADOOP-14946
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Gabor Bota
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HADOOP-14946.001.patch, HADOOP-14946.002.patch,
> HADOOP-14946.003.patch
>
>
> The test of the S3Guard CLI prune can sometimes fail on parallel test runs.
> Assumption: it is the parallelism which is causing the problem
> {code}
> org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB
> testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
> Time elapsed: 10.765 sec <<< FAILURE!
> java.lang.AssertionError: Pruned children count [] expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590619#comment-16590619
]
Daryn Sharp commented on HADOOP-14445:
--
Close to having a magic bullet patch done, writing tests.
# It's backwards compatible
# Requires no config changes
# Possibly "logical" name support w/o being config-based. Maybe
The trick to compatibility is non-standard behavior of the kms/authenticated
url token lookup. Instead of selecting a token based on its contained service,
the lookup is via the credentials alias – it assumes it's the service. The
token's actual service is meaningless. We can exploit that behavior by setting
the token service to a uri to allow renew/cancel to instantiate the issuing
provider, but leave the credentials alias as the host:port of one of the
providers so current client can find it.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
Steve Loughran created HADOOP-15691:
---
Summary: Add PathCapabilities to FS and FC to complement
StreamCapabilities
Key: HADOOP-15691
URL: https://issues.apache.org/jira/browse/HADOOP-15691
Project: Hadoop Common
Issue Type: New Feature
Reporter: Steve Loughran
Assignee: Steve Loughran
Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
declare the capabilities under the path of an FS
This is needed for
* HADOOP-15407: declare that a dest FS supports permissions
* object stores to declare that they offer PUT-in-place alongside (slow-rename)
* Anything else where the implementation semantics of an FS is so different
caller apps would benefit from probing for the underlying semantics
I know, we want all filesystem to work *exactly* the same. But it doesn't hold,
especially for object stores —and to efficiently use them, callers need to be
able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15664) ABFS: Reduce test run time via parallelization and grouping
[ https://issues.apache.org/jira/browse/HADOOP-15664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Da Zhou reassigned HADOOP-15664: Assignee: Da Zhou > ABFS: Reduce test run time via parallelization and grouping > --- > > Key: HADOOP-15664 > URL: https://issues.apache.org/jira/browse/HADOOP-15664 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Da Zhou >Priority: Major > > 1) Let's reduce the total test runtime by improving parallelization of the > tests. > 2) Let's make it possible to select WASB tests, ABFS tests, or both so > developers can run only the tests appropriate for the change they've made. > 3) Update the testing-azure.md accordingly -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 3.0.3) Status: Open (was: Patch Available) > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Target Version/s: 3.0.3, 2.10.0, 2.9.2, 2.8.5 (was: 2.10.0, 2.9.2, 3.0.3, 2.8.5) Status: Patch Available (was: Open) branch-2 patch; main difference is HadoopExecutors class doesn't exist. As I backport further, the move to commons-logging will force more change > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 3.0.0, 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15679) ShutdownHookManager shutdown time needs to be configurable & extended
[ https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15679: Attachment: HADOOP-15679-branch-2-001.patch > ShutdownHookManager shutdown time needs to be configurable & extended > - > > Key: HADOOP-15679 > URL: https://issues.apache.org/jira/browse/HADOOP-15679 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 2.8.0, 3.0.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch, > HADOOP-15679-002.patch, HADOOP-15679-003.patch, > HADOOP-15679-branch-2-001.patch > > > HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging > shutdowns. But the timeout is too short for applications where a large flush > of data is needed on shutdown. > A key example of this is Spark apps which save their history to object > stores, where the file close() call triggers an upload of the final local > cached block of data (could be 32+MB), and then execute the final mutipart > commit. > Proposed > # make the default sleep time 30s, not 10s > # make it configurable with a time duration property (with minimum time of > 1s.?) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15656) Support byteman in hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15656: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Support byteman in hadoop-runner baseimage > -- > > Key: HADOOP-15656 > URL: https://issues.apache.org/jira/browse/HADOOP-15656 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15656-docker-hadoop-runner.001.patch, > HADOOP-15656-docker-hadoop-runner.002.patch > > > [Byteman|http://byteman.jboss.org/] is an easy to use tool to instrument a > java process with agent string. > For example [this > script|https://gist.githubusercontent.com/elek/0589a91b4d55afb228279f6c4f04a525/raw/8bb4e03de7397c8a9d9bb74a5ec80028b42575c4/hadoop.btm] > defines a rule to print out all the hadoop rpc traffic to the standard > output (which is extremely useful for testing development). > This patch adds the byteman.jar to the baseimage and defines a simple logic > to add agent instrumentation string to the HADOOP_OPTS (optional it also > could download the byteman script from an url) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15656) Support byteman in hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590367#comment-16590367 ] Elek, Marton commented on HADOOP-15656: --- Thanks [~anu] the review. I uploaded the rebased version (no change, just different line numbers) and will commit to the docker-hadoop-runner branch, soon... > Support byteman in hadoop-runner baseimage > -- > > Key: HADOOP-15656 > URL: https://issues.apache.org/jira/browse/HADOOP-15656 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15656-docker-hadoop-runner.001.patch, > HADOOP-15656-docker-hadoop-runner.002.patch > > > [Byteman|http://byteman.jboss.org/] is an easy to use tool to instrument a > java process with agent string. > For example [this > script|https://gist.githubusercontent.com/elek/0589a91b4d55afb228279f6c4f04a525/raw/8bb4e03de7397c8a9d9bb74a5ec80028b42575c4/hadoop.btm] > defines a rule to print out all the hadoop rpc traffic to the standard > output (which is extremely useful for testing development). > This patch adds the byteman.jar to the baseimage and defines a simple logic > to add agent instrumentation string to the HADOOP_OPTS (optional it also > could download the byteman script from an url) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15656) Support byteman in hadoop-runner baseimage
[ https://issues.apache.org/jira/browse/HADOOP-15656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15656: -- Attachment: HADOOP-15656-docker-hadoop-runner.002.patch > Support byteman in hadoop-runner baseimage > -- > > Key: HADOOP-15656 > URL: https://issues.apache.org/jira/browse/HADOOP-15656 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15656-docker-hadoop-runner.001.patch, > HADOOP-15656-docker-hadoop-runner.002.patch > > > [Byteman|http://byteman.jboss.org/] is an easy to use tool to instrument a > java process with agent string. > For example [this > script|https://gist.githubusercontent.com/elek/0589a91b4d55afb228279f6c4f04a525/raw/8bb4e03de7397c8a9d9bb74a5ec80028b42575c4/hadoop.btm] > defines a rule to print out all the hadoop rpc traffic to the standard > output (which is extremely useful for testing development). > This patch adds the byteman.jar to the baseimage and defines a simple logic > to add agent instrumentation string to the HADOOP_OPTS (optional it also > could download the byteman script from an url) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590204#comment-16590204 ] Hudson commented on HADOOP-14314: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14819 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14819/]) HADOOP-14314. The OpenSolaris taxonomy link is dead in (tasanuma: rev bed8cb6979e0460141ed77e3b15d4f18db098a8e) * (edit) hadoop-common-project/hadoop-common/src/site/markdown/InterfaceClassification.md > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 2.8.5, 3.0.4 > > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HADOOP-14314: -- Resolution: Fixed Fix Version/s: 3.0.4 2.8.5 2.9.2 3.1.1 3.2.0 2.10.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-3.1, branch-3.0, branch-2, branch-2.9 and branch-2.8. Thanks for the contribution, [~demongaorui]! Also thanks [~templedf] and [~yuanbo]! > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 2.8.5, 3.0.4 > > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15680) ITestNativeAzureFileSystemConcurrencyLive times out
[
https://issues.apache.org/jira/browse/HADOOP-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590163#comment-16590163
]
genericqa commented on HADOOP-15680:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15680 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936819/HADOOP-15680.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 24c86134ca3d 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1ac0144 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15086/testReport/ |
| Max. process+thread count | 301 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15086/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ITestNativeAzureFileSystemConcurrencyLive times out
> ---
>
> Key: HADOOP-15680
> URL:
[jira] [Commented] (HADOOP-15680) ITestNativeAzureFileSystemConcurrencyLive times out
[ https://issues.apache.org/jira/browse/HADOOP-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590110#comment-16590110 ] Andras Bokor commented on HADOOP-15680: --- Thanks, attached a patch with 30 sec timeout. > ITestNativeAzureFileSystemConcurrencyLive times out > --- > > Key: HADOOP-15680 > URL: https://issues.apache.org/jira/browse/HADOOP-15680 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-15680.001.patch, HADOOP-15680.002.patch > > > When I am running tests locally ITestNativeAzureFileSystemConcurrencyLive > sometimes times out. > I would like to increase the timeout to avoid unnecessary noise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15680) ITestNativeAzureFileSystemConcurrencyLive times out
[ https://issues.apache.org/jira/browse/HADOOP-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-15680: -- Attachment: HADOOP-15680.002.patch > ITestNativeAzureFileSystemConcurrencyLive times out > --- > > Key: HADOOP-15680 > URL: https://issues.apache.org/jira/browse/HADOOP-15680 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andras Bokor >Assignee: Andras Bokor >Priority: Major > Attachments: HADOOP-15680.001.patch, HADOOP-15680.002.patch > > > When I am running tests locally ITestNativeAzureFileSystemConcurrencyLive > sometimes times out. > I would like to increase the timeout to avoid unnecessary noise. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590016#comment-16590016 ] Rui Gao commented on HADOOP-14314: -- OK! Thanks for review and committing! [~tasanuma0829] > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589987#comment-16589987 ] Takanobu Asanuma commented on HADOOP-14314: --- +1. I think you don't need to create another patch for branch-2 since HADOOP-14314.03.patch can be applied to branch-2 safely. Will commit it later. > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15196) Zlib decompression fails when file having trailing garbage
[
https://issues.apache.org/jira/browse/HADOOP-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589940#comment-16589940
]
genericqa commented on HADOOP-15196:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-15196 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936789/HADOOP-15196.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 7d981600a210 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1ac0144 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15085/testReport/ |
| Max. process+thread count | 1570 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15085/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Zlib decompression
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[
https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589867#comment-16589867
]
genericqa commented on HADOOP-14314:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
36m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HADOOP-14314 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12936786/HADOOP-14314.03.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 9e6ffc075392 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1ac0144 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 334 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15084/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The OpenSolaris taxonomy link is dead in InterfaceClassification.md
> ---
>
> Key: HADOOP-14314
> URL: https://issues.apache.org/jira/browse/HADOOP-14314
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: Rui Gao
>Priority: Major
> Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch,
> HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch,
> image-2018-08-23-15-55-51-587.png, screenshot-1.png
>
>
> Unfortunately, Oracle took down opensolaris.org, so the link is dead. The
> only replacement I could find with a quick search was this PDF:
> http://cuddletech.com/opensolaris/osdevref.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15196) Zlib decompression fails when file having trailing garbage
[
https://issues.apache.org/jira/browse/HADOOP-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589827#comment-16589827
]
Brahma Reddy Battula commented on HADOOP-15196:
---
Uploaded the patch to ignore the '*trailing garbage*'. Kindly review.
> Zlib decompression fails when file having trailing garbage
> --
>
> Key: HADOOP-15196
> URL: https://issues.apache.org/jira/browse/HADOOP-15196
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15196.patch
>
>
> *When file has trailing garbage gzip will ignore.*
> {noformat}
> gzip -d 2018011309-js.rishenglipin.com.gz
> gzip: 2018011309-js.rishenglipin.com.gz: decompression OK, trailing garbage
> ignored
> {noformat}
> *when we use same file and decompress,we got following.*
> {noformat}
> 2018-01-13 14:23:43,151 | WARN | task-result-getter-3 | Lost task 0.0 in
> stage 345.0 (TID 5686, node-core-gyVYT, executor 3): java.io.IOException:
> unknown compression method
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
> Method)
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:225)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15196) Zlib decompression fails when file having trailing garbage
[
https://issues.apache.org/jira/browse/HADOOP-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-15196:
--
Status: Patch Available (was: Open)
> Zlib decompression fails when file having trailing garbage
> --
>
> Key: HADOOP-15196
> URL: https://issues.apache.org/jira/browse/HADOOP-15196
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15196.patch
>
>
> *When file has trailing garbage gzip will ignore.*
> {noformat}
> gzip -d 2018011309-js.rishenglipin.com.gz
> gzip: 2018011309-js.rishenglipin.com.gz: decompression OK, trailing garbage
> ignored
> {noformat}
> *when we use same file and decompress,we got following.*
> {noformat}
> 2018-01-13 14:23:43,151 | WARN | task-result-getter-3 | Lost task 0.0 in
> stage 345.0 (TID 5686, node-core-gyVYT, executor 3): java.io.IOException:
> unknown compression method
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
> Method)
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:225)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15196) Zlib decompression fails when file having trailing garbage
[
https://issues.apache.org/jira/browse/HADOOP-15196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-15196:
--
Attachment: HADOOP-15196.patch
> Zlib decompression fails when file having trailing garbage
> --
>
> Key: HADOOP-15196
> URL: https://issues.apache.org/jira/browse/HADOOP-15196
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>Priority: Major
> Attachments: HADOOP-15196.patch
>
>
> *When file has trailing garbage gzip will ignore.*
> {noformat}
> gzip -d 2018011309-js.rishenglipin.com.gz
> gzip: 2018011309-js.rishenglipin.com.gz: decompression OK, trailing garbage
> ignored
> {noformat}
> *when we use same file and decompress,we got following.*
> {noformat}
> 2018-01-13 14:23:43,151 | WARN | task-result-getter-3 | Lost task 0.0 in
> stage 345.0 (TID 5686, node-core-gyVYT, executor 3): java.io.IOException:
> unknown compression method
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native
> Method)
> at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:225)
> at
> org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91)
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589821#comment-16589821 ] Rui Gao commented on HADOOP-14314: -- Will upload branch-2 patch after trunk patch build finished. > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Gao updated HADOOP-14314: - Attachment: HADOOP-14314.03.patch > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.03.patch, HADOOP-14314.branch-2.01.patch, > image-2018-08-23-15-55-51-587.png, screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589797#comment-16589797 ] Rui Gao edited comment on HADOOP-14314 at 8/23/18 6:59 AM: --- [~tasanuma0829] Make sense. Uploaded new patch. And the change of doc looks like: # Before !screenshot-1.png! # After !image-2018-08-23-15-55-51-587.png! was (Author: demongaorui): [~tasanuma0829] Make sense. Uploaded new patch. And the change of doc looks like: ## Before !screenshot-1.png! ## After !image-2018-08-23-15-55-51-587.png! > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.branch-2.01.patch, image-2018-08-23-15-55-51-587.png, > screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589797#comment-16589797 ] Rui Gao commented on HADOOP-14314: -- [~tasanuma0829] Make sense. Uploaded new patch. And the change of doc looks like: ## Before !screenshot-1.png! ## After !image-2018-08-23-15-55-51-587.png! > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.branch-2.01.patch, image-2018-08-23-15-55-51-587.png, > screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Gao updated HADOOP-14314: - Attachment: screenshot-1.png > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.branch-2.01.patch, image-2018-08-23-15-55-51-587.png, > screenshot-1.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14314) The OpenSolaris taxonomy link is dead in InterfaceClassification.md
[ https://issues.apache.org/jira/browse/HADOOP-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Gao updated HADOOP-14314: - Attachment: image-2018-08-23-15-55-51-587.png > The OpenSolaris taxonomy link is dead in InterfaceClassification.md > --- > > Key: HADOOP-14314 > URL: https://issues.apache.org/jira/browse/HADOOP-14314 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha2 >Reporter: Daniel Templeton >Assignee: Rui Gao >Priority: Major > Attachments: HADOOP-14314.01.patch, HADOOP-14314.02.patch, > HADOOP-14314.branch-2.01.patch, image-2018-08-23-15-55-51-587.png > > > Unfortunately, Oracle took down opensolaris.org, so the link is dead. The > only replacement I could find with a quick search was this PDF: > http://cuddletech.com/opensolaris/osdevref.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15689) Add "*.patch" into .gitignore file of branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589786#comment-16589786 ] Rui Gao commented on HADOOP-15689: -- [~tasanuma0829] Thanks for reviewing and committing! > Add "*.patch" into .gitignore file of branch-2 > -- > > Key: HADOOP-15689 > URL: https://issues.apache.org/jira/browse/HADOOP-15689 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Rui Gao >Assignee: Rui Gao >Priority: Major > Fix For: 2.10.0, 2.9.2, 2.8.5 > > Attachments: HADOOP-15689.branch-2.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
