[jira] [Commented] (HADOOP-15235) Authentication Tokens should use HMAC instead of MAC
[
https://issues.apache.org/jira/browse/HADOOP-15235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375379#comment-16375379
]
Xiao Chen commented on HADOOP-15235:
Hi [~rkanter],
Anecdotally I'm reviewing an hdfs jira, and it seems this one broke
{{TestHttpFSServerWebServer#testStartStop}}:
{noformat}
2018-02-23 23:13:29,791 WARN ServletHandler - /webhdfs/v1/
java.lang.IllegalArgumentException: Empty key
at javax.crypto.spec.SecretKeySpec.(SecretKeySpec.java:96)
at
org.apache.hadoop.security.authentication.util.Signer.computeSignature(Signer.java:93)
at
org.apache.hadoop.security.authentication.util.Signer.sign(Signer.java:59)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:587)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1617)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
java.lang.AssertionError:
Expected :500
Actual :200
{noformat}
Mind take a look?
> Authentication Tokens should use HMAC instead of MAC
>
>
> Key: HADOOP-15235
> URL: https://issues.apache.org/jira/browse/HADOOP-15235
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.10.0, 3.2.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Fix For: 2.10.0, 3.2.0
>
> Attachments: HADOOP-15235.001.patch, HADOOP-15235.002.patch
>
>
> We currently use {{MessageDigest}} to compute a "SHA" MAC for signing
> Authentication Tokens. Firstly, what "SHA" maps to is dependent on the JVM
> and Cryptography Provider. While they _should_ do something reasonable, it's
> probably a safer idea to pick a specific algorithm. It looks like the Oracle
> JVM picks SHA-1; though something like SHA-256 would be better.
> In any case, it would also be better to use an HMAC algorithm instead.
> Changing from SHA-1 to SHA-256 or MAC to HMAC won't generate equivalent
> signatures, so this would normally be an incompatible change because the
> server wouldn't accept previous tokens it issued with the older algorithm.
> However, Authentication Tokens are used as a cheaper shortcut for Kerberos,
> so it's expected for users to also have Kerberos credentials; in this case,
> the Authentication Token will be rejected, but it
[jira] [Commented] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[
https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375324#comment-16375324
]
genericqa commented on HADOOP-15254:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 5s{color} | {color:orange} root: The patch generated 1 new + 432 unchanged
- 1 fixed = 433 total (was 433) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 49s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 40s{color}
| {color:red} hadoop-kms in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 63m
32s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
17s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
21s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
17s{color} | {color:green} hadoop-streaming in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}295m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Commented] (HADOOP-15251) Upgrade surefire version in branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375286#comment-16375286
]
genericqa commented on HADOOP-15251:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
42s{color} | {color:green} branch-2 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 11m
46s{color} | {color:red} root in branch-2 failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
46s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 11m
34s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 32s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
40s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}166m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.sftp.TestSFTPFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HADOOP-15251 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911852/HADOOP-15251-branch-2.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux e937d3581957 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / a0b5539 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| mvnsite |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/artifact/out/branch-mvnsite-root.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/artifact/out/patch-mvnsite-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1604 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws hadoop-tools/hadoop-azure
. U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14207/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HADOOP-13374) Add the L verification script
[ https://issues.apache.org/jira/browse/HADOOP-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375214#comment-16375214 ] Hudson commented on HADOOP-13374: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13712 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13712/]) HADOOP-13374. Add the L verification script. Contributed by Allen (cdouglas: rev 329a4fdd07ab007615f34c8e0e651360f988064d) * (add) dev-support/bin/verify-license-files > Add the L verification script > --- > > Key: HADOOP-13374 > URL: https://issues.apache.org/jira/browse/HADOOP-13374 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Allen Wittenauer >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-13374.01.patch, HADOOP-13374.02.patch, > HADOOP-13374.03.patch, HADOOP-13374.04.patch > > > This is the script that's used for L change verification during > HADOOP-12893. We should commit this as [~ozawa] > [suggested|https://issues.apache.org/jira/browse/HADOOP-13298?focusedCommentId=15374498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15374498]. > I was > [initially|https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15283040=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15283040] > verifying from an on-fly shell command, and [~andrew.wang] contributed the > script later in [a comment| > https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15303281=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303281], > so most credit should go to him. :) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13374) Add the L verification script
[ https://issues.apache.org/jira/browse/HADOOP-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-13374: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) +1 I committed this. > Add the L verification script > --- > > Key: HADOOP-13374 > URL: https://issues.apache.org/jira/browse/HADOOP-13374 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Allen Wittenauer >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-13374.01.patch, HADOOP-13374.02.patch, > HADOOP-13374.03.patch, HADOOP-13374.04.patch > > > This is the script that's used for L change verification during > HADOOP-12893. We should commit this as [~ozawa] > [suggested|https://issues.apache.org/jira/browse/HADOOP-13298?focusedCommentId=15374498=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15374498]. > I was > [initially|https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15283040=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15283040] > verifying from an on-fly shell command, and [~andrew.wang] contributed the > script later in [a comment| > https://issues.apache.org/jira/browse/HADOOP-12893?focusedCommentId=15303281=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15303281], > so most credit should go to him. :) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15251) Upgrade surefire version in branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-15251: --- Attachment: HADOOP-15251-branch-2.002.patch > Upgrade surefire version in branch-2 > > > Key: HADOOP-15251 > URL: https://issues.apache.org/jira/browse/HADOOP-15251 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Chris Douglas >Priority: Major > Attachments: HADOOP-15251-branch-2.001.patch, > HADOOP-15251-branch-2.002.patch > > > Tests in branch-2 are not running reliably in Jenkins, and due to > SUREFIRE-524, these are not being cleaned up properly (see HADOOP-15153). > Upgrading to a more recent version of the surefire plugin will help make the > problem easier to address in branch-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[ https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375175#comment-16375175 ] fang zhenyi commented on HADOOP-15254: -- Thanks [~arpitagarwal] for giving me suggestion.I have removed jdiff xml files or changes.md in HADOOP-15254.002.patch. > Correct the wrong word spelling 'intialize' > --- > > Key: HADOOP-15254 > URL: https://issues.apache.org/jira/browse/HADOOP-15254 > Project: Hadoop Common > Issue Type: Bug >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15254.001.patch, HADOOP-15254.002.patch > > > The correct wording of 'intialize' should be 'initialize'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[ https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15254: - Status: Patch Available (was: In Progress) > Correct the wrong word spelling 'intialize' > --- > > Key: HADOOP-15254 > URL: https://issues.apache.org/jira/browse/HADOOP-15254 > Project: Hadoop Common > Issue Type: Bug >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15254.001.patch, HADOOP-15254.002.patch > > > The correct wording of 'intialize' should be 'initialize'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[ https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15254: - Status: In Progress (was: Patch Available) > Correct the wrong word spelling 'intialize' > --- > > Key: HADOOP-15254 > URL: https://issues.apache.org/jira/browse/HADOOP-15254 > Project: Hadoop Common > Issue Type: Bug >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15254.001.patch, HADOOP-15254.002.patch > > > The correct wording of 'intialize' should be 'initialize'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[ https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fang zhenyi updated HADOOP-15254: - Attachment: HADOOP-15254.002.patch > Correct the wrong word spelling 'intialize' > --- > > Key: HADOOP-15254 > URL: https://issues.apache.org/jira/browse/HADOOP-15254 > Project: Hadoop Common > Issue Type: Bug >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15254.001.patch, HADOOP-15254.002.patch > > > The correct wording of 'intialize' should be 'initialize'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375141#comment-16375141
]
Hudson commented on HADOOP-15255:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13711 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13711/])
HADOOP-15255. Upper/Lower case conversion support for group names in (arp: rev
033f9c68ea82b6293e760df457bb6eb23c6889c8)
* (add)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/RuleBasedLdapGroupsMapping.java
* (edit)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
* (add)
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestRuleBasedLdapGroupsMapping.java
* (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-15255.000.patch, HADOOP-15255.001.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-15255:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
The patch and tests look great! +1.
I've committed this.
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HADOOP-15255.000.patch, HADOOP-15255.001.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375088#comment-16375088
]
genericqa commented on HADOOP-15255:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 5 unchanged - 0 fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
50s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15255 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911805/HADOOP-15255.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 8ced8d78df91 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 59cf758 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14205/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14205/testReport/ |
| Max. process+thread count | 1758 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374992#comment-16374992
]
Nanda kumar commented on HADOOP-15255:
--
Fixed unit test failure and checkstyle issue in patch v001.
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch, HADOOP-15255.001.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HADOOP-15255:
-
Attachment: HADOOP-15255.001.patch
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch, HADOOP-15255.001.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15178) Generalize NetUtils#wrapException to handle other subclasses with String Constructor.
[ https://issues.apache.org/jira/browse/HADOOP-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374896#comment-16374896 ] Ajay Kumar commented on HADOOP-15178: - Test failures are unrelated. Both of them passed locally. > Generalize NetUtils#wrapException to handle other subclasses with String > Constructor. > - > > Key: HADOOP-15178 > URL: https://issues.apache.org/jira/browse/HADOOP-15178 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HADOOP-15178.001.patch, HADOOP-15178.002.patch, > HADOOP-15178.003.patch > > > NetUtils#wrapException returns an IOException if exception passed to it is > not of type > SocketException,EOFException,NoRouteToHostException,SocketTimeoutException,UnknownHostException,ConnectException,BindException. > By default, it should always return instance (subclass of IOException) of > same type unless a String constructor is not available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374880#comment-16374880 ] Hudson commented on HADOOP-9747: SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13707 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13707/]) HADOOP-9747. Reduce unnecessary UGI synchronization. Contributed by (kihwal: rev 59cf7588779145ad5850ad63426743dfe03d8347) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGIWithMiniKdc.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUGILoginFromKeytab.java > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 3.1.0, 3.2.0, 3.0.2 > > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374870#comment-16374870 ] Kihwal Lee commented on HADOOP-9747: I've committed this to trunk, branch-3.1 and branch-3.0. Please provide a branch-2 version. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 3.1.0, 3.2.0, 3.0.2 > > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-9747: --- Fix Version/s: 3.0.2 3.2.0 3.1.0 > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Fix For: 3.1.0, 3.2.0, 3.0.2 > > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374840#comment-16374840 ] Hudson commented on HADOOP-15007: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13706 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13706/]) HADOOP-15007. Stabilize and document Configuration element. (aengineer: rev 3688e491d528edb9efe54c4ef110d2ded62db3e8) * (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/HDFSPropertyTag.java * (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/PropertyTag.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/YarnPropertyTag.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java * (delete) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/CorePropertyTag.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374834#comment-16374834
]
genericqa commented on HADOOP-13761:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 13 unchanged - 1 fixed = 14 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
30s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-13761 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911763/HADOOP-13761-010.patch
|
| Optional Tests | asflicense findbugs xml compile javac javadoc
mvninstall mvnsite unit shadedclient checkstyle |
| uname | Linux fa9b40046281 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14204/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14204/testReport/ |
| Max. process+thread count | 328 (vs. ulimit of 1) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374822#comment-16374822 ] Anu Engineer commented on HADOOP-15007: --- [~leftnoteasy] if you are ok with this, Can I commit this to 3.1 ? Thanks. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374821#comment-16374821 ] Ajay Kumar commented on HADOOP-15007: - [~anu], [~elek],[~ste...@apache.org], [~leftnoteasy] thanks for valuable inputs. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-15007: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) [~ajayydv] I have committed this to the trunk. Thank you for the contribution. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Fix For: 3.2.0 > > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15254) Correct the wrong word spelling 'intialize'
[ https://issues.apache.org/jira/browse/HADOOP-15254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374813#comment-16374813 ] Arpit Agarwal commented on HADOOP-15254: Hi [~zhenyi], we should not edit the jdiff xml files or changes.md. Changes should only be made to source files. > Correct the wrong word spelling 'intialize' > --- > > Key: HADOOP-15254 > URL: https://issues.apache.org/jira/browse/HADOOP-15254 > Project: Hadoop Common > Issue Type: Bug >Reporter: fang zhenyi >Assignee: fang zhenyi >Priority: Minor > Attachments: HADOOP-15254.001.patch > > > The correct wording of 'intialize' should be 'initialize'. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15007) Stabilize and document Configuration element
[ https://issues.apache.org/jira/browse/HADOOP-15007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374809#comment-16374809 ] Anu Engineer commented on HADOOP-15007: --- [~ste...@apache.org] and [~elek] I am committing this change. With this change, all of our three perspectives are addressed. # Lose types and Enums - This patch makes tags simple strings. # Detect tags which are not correct - The systems tags and custom tags list in the XML config can be used to cross check if a tag is valid or not. If it is not, we add a single line of log that tells us that tag is being ignored. # Remove unnecessary logging - With this patch, Ajay has removed the repeated warnings that were filling up the logs. [~ajayydv], +1, thanks for getting this done, I will commit this shortly. > Stabilize and document Configuration element > -- > > Key: HADOOP-15007 > URL: https://issues.apache.org/jira/browse/HADOOP-15007 > Project: Hadoop Common > Issue Type: Improvement > Components: conf >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Ajay Kumar >Priority: Blocker > Attachments: HADOOP-15007.000.patch, HADOOP-15007.001.patch, > HADOOP-15007.002.patch, HADOOP-15007.003.patch > > > HDFS-12350 (moved to HADOOP-15005). Adds the ability to tag properties with a > value. > We need to make sure that this feature is backwards compatible & usable in > production. That's docs, testing, marshalling etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374799#comment-16374799 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks.. I added some @ annotations for checkstyle but it didn't seem to pick them up. Didn't we used to be able to use bin/dev-support/test-patch to preview checkstyle deltas? I had a completely green run of that script before submitting v9 patch so I don't think it is working. I'll probably start a thread on common-dev. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374780#comment-16374780
]
genericqa commented on HADOOP-15255:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 53s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 48s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.conf.TestCommonConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15255 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911752/HADOOP-15255.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d00085d6b93c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14203/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
[jira] [Resolved] (HADOOP-7675) Ant option to run disabled kerberos authentication tests.
[ https://issues.apache.org/jira/browse/HADOOP-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HADOOP-7675. Resolution: Won't Fix This ticket is ancient history. Resolving. > Ant option to run disabled kerberos authentication tests. > - > > Key: HADOOP-7675 > URL: https://issues.apache.org/jira/browse/HADOOP-7675 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jitendra Nath Pandey >Priority: Major > > The kerberos tests, TestKerberosAuthenticator and > TestKerberosAuthenticationHandler, are disabled using @Ignore. A better > approach would be to have an ant option to run them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374746#comment-16374746 ] Steve Loughran commented on HADOOP-13761: - attached: patch 010; fixes findbugs by marking one field as private, turning of serialization warnigns for InconsistentS3Object and turning off all IDE sync warnings for S3AInputStream. The latter is risky, but I couldn't do any other way given the names of lambdas aren't known/declared. Apparently findbugs now supports some @annotations which can be used to disable the checks, they may be things you can apply on a specific lambda-exp. If/when we adopt them in hadoop, this exclusion can be revisited. Until then, IDE only. > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Status: Patch Available (was: Open) > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Status: Open (was: Patch Available) > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374744#comment-16374744 ] Bharat Viswanadham commented on HADOOP-9747: Thanks for the info [~daryn]. Yup, it will not create multiple threads. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Attachment: HADOOP-13761-010.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13761: Attachment: HADOOP-13761-010.patch > S3Guard: implement retries for DDB failures and throttling; translate > exceptions > > > Key: HADOOP-13761 > URL: https://issues.apache.org/jira/browse/HADOOP-13761 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Blocker > Attachments: HADOOP-13761-004-to-005.patch, > HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch, > HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch, > HADOOP-13761-009.patch, HADOOP-13761-010.patch, HADOOP-13761-010.patch, > HADOOP-13761.001.patch, HADOOP-13761.002.patch, HADOOP-13761.003.patch, > HADOOP-13761.004.patch > > > Following the S3AFileSystem integration patch in HADOOP-13651, we need to add > retry logic. > In HADOOP-13651, I added TODO comments in most of the places retry loops are > needed, including: > - open(path). If MetadataStore reflects recent create/move of file path, but > we fail to read it from S3, retry. > - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the > file exists, retry. > - rename(src,dest). If source path is not visible in S3 yet, retry. > - listFiles(). Skip for now. Not currently implemented in S3Guard. I will > create a separate JIRA for this as it will likely require interface changes > (i.e. prefix or subtree scan). > We may miss some cases initially and we should do failure injection testing > to make sure we're covered. Failure injection tests can be a separate JIRA > to make this easier to review. > We also need basic configuration parameters around retry policy. There > should be a way to specify maximum retry duration, as some applications would > prefer to receive an error eventually, than waiting indefinitely. We should > also be keeping statistics when inconsistency is detected and we enter a > retry loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374631#comment-16374631 ] Kihwal Lee commented on HADOOP-9747: Thanks for the review [~bharatviswa]. With my official +1, I will commit it shortly. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions
[
https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374625#comment-16374625
]
Steve Loughran commented on HADOOP-13761:
-
LGTM. Like the opcontext model.
+1
just trying to see if I can get findbugs to shut up.
One minor worry of mine: if someone deletes a file being read, when the client
with the open stream does a seek(), the FNFE may get retried a few times before
giving up. It's a failure case anyway, so I'm not that worried, and trying to
cleanly model it would need for the input stream to record whether it had done
1+ successful open, and pass that info to the read conext with
{{getReadInvoker()}} adding a param {{isFirstRead}} to use in its decision
making.
> S3Guard: implement retries for DDB failures and throttling; translate
> exceptions
>
>
> Key: HADOOP-13761
> URL: https://issues.apache.org/jira/browse/HADOOP-13761
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Aaron Fabbri
>Assignee: Aaron Fabbri
>Priority: Blocker
> Attachments: HADOOP-13761-004-to-005.patch,
> HADOOP-13761-005-to-006-approx.diff.txt, HADOOP-13761-005.patch,
> HADOOP-13761-006.patch, HADOOP-13761-007.patch, HADOOP-13761-008.patch,
> HADOOP-13761-009.patch, HADOOP-13761.001.patch, HADOOP-13761.002.patch,
> HADOOP-13761.003.patch, HADOOP-13761.004.patch
>
>
> Following the S3AFileSystem integration patch in HADOOP-13651, we need to add
> retry logic.
> In HADOOP-13651, I added TODO comments in most of the places retry loops are
> needed, including:
> - open(path). If MetadataStore reflects recent create/move of file path, but
> we fail to read it from S3, retry.
> - delete(path). If deleteObject() on S3 fails, but MetadataStore shows the
> file exists, retry.
> - rename(src,dest). If source path is not visible in S3 yet, retry.
> - listFiles(). Skip for now. Not currently implemented in S3Guard. I will
> create a separate JIRA for this as it will likely require interface changes
> (i.e. prefix or subtree scan).
> We may miss some cases initially and we should do failure injection testing
> to make sure we're covered. Failure injection tests can be a separate JIRA
> to make this easier to review.
> We also need basic configuration parameters around retry policy. There
> should be a way to specify maximum retry duration, as some applications would
> prefer to receive an error eventually, than waiting indefinitely. We should
> also be keeping statistics when inconsistency is detected and we enter a
> retry loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey reassigned HADOOP-15255:
-
Assignee: Nanda kumar
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HADOOP-15255:
-
Status: Patch Available (was: Open)
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-15255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HADOOP-15255:
-
Attachment: HADOOP-15255.000.patch
> Upper/Lower case conversion support for group names in LdapGroupsMapping
>
>
> Key: HADOOP-15255
> URL: https://issues.apache.org/jira/browse/HADOOP-15255
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Nanda kumar
>Priority: Major
> Attachments: HADOOP-15255.000.patch
>
>
> On most LDAP servers the user and group names are case-insensitive. When we
> use {{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it
> is possible to configure {{SSSD}} to force the group names to be returned in
> lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
> This jira proposes to introduce a new {{hadoop.security.group.mapping}}
> implementation based on LdapGroupsMapping which supports force lower/upper
> case group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374566#comment-16374566
]
Elek, Marton commented on HADOOP-14898:
---
I also created a quick video about the main use cases to make it easier to
review:
https://www.youtube.com/watch?v=Om_k83K_9C8=youtu.be
(Sorry for the flickering, this is the intel video driver under linux)
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of
[jira] [Commented] (HADOOP-9747) Reduce unnecessary UGI synchronization
[ https://issues.apache.org/jira/browse/HADOOP-9747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374494#comment-16374494 ] Daryn Sharp commented on HADOOP-9747: - Regarding auto-renewal, I specifically made getLoginUser only auto-renew if it's the first to set the login user. Ie. If another concurrent login (extremely unlikely in practice, but possible) races with the other and loses the race, it doesn't spawn a renewal thread. That's why it is guarded by a CAS. > Reduce unnecessary UGI synchronization > -- > > Key: HADOOP-9747 > URL: https://issues.apache.org/jira/browse/HADOOP-9747 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HADOOP-9747-trunk-03.patch, HADOOP-9747-trunk-04.patch, > HADOOP-9747-trunk.01.patch, HADOOP-9747-trunk.02.patch, > HADOOP-9747.2.branch-2.patch, HADOOP-9747.2.trunk.patch, > HADOOP-9747.branch-2.patch, HADOOP-9747.trunk.patch > > > Jstacks of heavily loaded NNs show up to dozens of threads blocking in the > UGI. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15258) Create example docker-compose file for documentations
[
https://issues.apache.org/jira/browse/HADOOP-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374479#comment-16374479
]
genericqa commented on HADOOP-15258:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911728/HADOOP-15258.001.patch
|
| Optional Tests | asflicense mvnsite shellcheck shelldocs |
| uname | Linux 6d2c698c00b9 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14199/artifact/out/whitespace-eol.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14199/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14199/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create example docker-compose file for documentations
> -
>
> Key: HADOOP-15258
> URL: https://issues.apache.org/jira/browse/HADOOP-15258
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15258.001.patch
>
>
> An other user case for docker is to use it in the documentation. For example
> in the HA documentation we can provide an example docker-compose file and
> configuration with all the required settings to getting started easily with
> an HA cluster.
> 1. I would add an example to a documetation page
> 2. It will use the hadoop3 image (which contains latest hadoop3) as the user
> of the documentation may not build a hadoop
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12585) [Umbrella] Removing the usages of deprecated methods
[ https://issues.apache.org/jira/browse/HADOOP-12585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor resolved HADOOP-12585. --- Resolution: Done Target Version/s: (was: ) This umbrella is no longer in use. > [Umbrella] Removing the usages of deprecated methods > > > Key: HADOOP-12585 > URL: https://issues.apache.org/jira/browse/HADOOP-12585 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Major > > There are lots usages of deprecated methods in hadoop - we should avoid using > them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-7675) Ant option to run disabled kerberos authentication tests.
[ https://issues.apache.org/jira/browse/HADOOP-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374467#comment-16374467 ] Andras Bokor commented on HADOOP-7675: -- I don't see the @Ignore annotation on the referenced classes. Is this ticket still valid? > Ant option to run disabled kerberos authentication tests. > - > > Key: HADOOP-7675 > URL: https://issues.apache.org/jira/browse/HADOOP-7675 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 1.1.0 >Reporter: Jitendra Nath Pandey >Priority: Major > > The kerberos tests, TestKerberosAuthenticator and > TestKerberosAuthenticationHandler, are disabled using @Ignore. A better > approach would be to have an ant option to run them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14898) Create official Docker images for development and testing features
[
https://issues.apache.org/jira/browse/HADOOP-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-14898:
--
Status: Patch Available (was: In Progress)
Moving it to patch-available state as all the sub-issues are in patch-available.
> Create official Docker images for development and testing features
> ---
>
> Key: HADOOP-14898
> URL: https://issues.apache.org/jira/browse/HADOOP-14898
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-14898.001.tar.gz, HADOOP-14898.002.tar.gz,
> HADOOP-14898.003.tgz, docker_design.pdf
>
>
> This is the original mail from the mailing list:
> {code}
> TL;DR: I propose to create official hadoop images and upload them to the
> dockerhub.
> GOAL/SCOPE: I would like improve the existing documentation with easy-to-use
> docker based recipes to start hadoop clusters with various configuration.
> The images also could be used to test experimental features. For example
> ozone could be tested easily with these compose file and configuration:
> https://gist.github.com/elek/1676a97b98f4ba561c9f51fce2ab2ea6
> Or even the configuration could be included in the compose file:
> https://github.com/elek/hadoop/blob/docker-2.8.0/example/docker-compose.yaml
> I would like to create separated example compose files for federation, ha,
> metrics usage, etc. to make it easier to try out and understand the features.
> CONTEXT: There is an existing Jira
> https://issues.apache.org/jira/browse/HADOOP-13397
> But it’s about a tool to generate production quality docker images (multiple
> types, in a flexible way). If no objections, I will create a separated issue
> to create simplified docker images for rapid prototyping and investigating
> new features. And register the branch to the dockerhub to create the images
> automatically.
> MY BACKGROUND: I am working with docker based hadoop/spark clusters quite a
> while and run them succesfully in different environments (kubernetes,
> docker-swarm, nomad-based scheduling, etc.) My work is available from here:
> https://github.com/flokkr but they could handle more complex use cases (eg.
> instrumenting java processes with btrace, or read/reload configuration from
> consul).
> And IMHO in the official hadoop documentation it’s better to suggest to use
> official apache docker images and not external ones (which could be changed).
> {code}
> The next list will enumerate the key decision points regarding to docker
> image creating
> A. automated dockerhub build / jenkins build
> Docker images could be built on the dockerhub (a branch pattern should be
> defined for a github repository and the location of the Docker files) or
> could be built on a CI server and pushed.
> The second one is more flexible (it's more easy to create matrix build, for
> example)
> The first one had the advantage that we can get an additional flag on the
> dockerhub that the build is automated (and built from the source by the
> dockerhub).
> The decision is easy as ASF supports the first approach: (see
> https://issues.apache.org/jira/browse/INFRA-12781?focusedCommentId=15824096=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15824096)
> B. source: binary distribution or source build
> The second question is about creating the docker image. One option is to
> build the software on the fly during the creation of the docker image the
> other one is to use the binary releases.
> I suggest to use the second approach as:
> 1. In that case the hadoop:2.7.3 could contain exactly the same hadoop
> distrubution as the downloadable one
> 2. We don't need to add development tools to the image, the image could be
> more smaller (which is important as the goal for this image to getting
> started as fast as possible)
> 3. The docker definition will be more simple (and more easy to maintain)
> Usually this approach is used in other projects (I checked Apache Zeppelin
> and Apache Nutch)
> C. branch usage
> Other question is the location of the Docker file. It could be on the
> official source-code branches (branch-2, trunk, etc.) or we can create
> separated branches for the dockerhub (eg. docker/2.7 docker/2.8 docker/3.0)
> For the first approach it's easier to find the docker images, but it's less
> flexible. For example if we had a Dockerfile for on the source code it should
> be used for every release (for example the Docker file from the tag
> release-3.0.0 should be used for the 3.0 hadoop docker image). In that case
> the release process is much more harder: in case of a Dockerfile error (which
> could be test on dockerhub only after the taging), a new release should be
> added after
[jira] [Commented] (HADOOP-15250) Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
[
https://issues.apache.org/jira/browse/HADOOP-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374402#comment-16374402
]
Steve Loughran commented on HADOOP-15250:
-
I think that AD setup has just gone beyond my everyday experience. We are into
kerberos-land.
I think the general stance on long-haul links is that webhdfs is the way things
should be done, especially for distcp. I have no idea if that will help here,
or merely change the stack traces and config options. Have you tried?
> Split-DNS MultiHomed Server Network Cluster Network IPC Client Bind Addr Wrong
> --
>
> Key: HADOOP-15250
> URL: https://issues.apache.org/jira/browse/HADOOP-15250
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, net
>Affects Versions: 2.7.3, 2.9.0, 3.0.0
> Environment: Multihome cluster with split DNS and rDNS lookup of
> localhost returning non-routable IPAddr
>Reporter: Greg Senia
>Priority: Critical
> Attachments: HADOOP-15250.patch
>
>
> We run our Hadoop clusters with two networks attached to each node. These
> network are as follows a server network that is firewalled with firewalld
> allowing inbound traffic: only SSH and things like Knox and Hiveserver2 and
> the HTTP YARN RM/ATS and MR History Server. The second network is the cluster
> network on the second network interface this uses Jumbo frames and is open no
> restrictions and allows all cluster traffic to flow between nodes.
>
> To resolve DNS within the Hadoop Cluster we use DNS Views via BIND so if the
> traffic is originating from nodes with cluster networks we return the
> internal DNS record for the nodes. This all works fine with all the
> multi-homing features added to Hadoop 2.x
> Some logic around views:
> a. The internal view is used by cluster machines when performing lookups. So
> hosts on the cluster network should get answers from the internal view in DNS
> b. The external view is used by non-local-cluster machines when performing
> lookups. So hosts not on the cluster network should get answers from the
> external view in DNS
>
> So this brings me to our problem. We created some firewall rules to allow
> inbound traffic from each clusters server network to allow distcp to occur.
> But we noticed a problem almost immediately that when YARN attempted to talk
> to the Remote Cluster it was binding outgoing traffic to the cluster network
> interface which IS NOT routable. So after researching the code we noticed the
> following in NetUtils.java and Client.java
> Basically in Client.java it looks as if it takes whatever the hostname is and
> attempts to bind to whatever the hostname is resolved to. This is not valid
> in a multi-homed network with one routable interface and one non routable
> interface. After reading through the java.net.Socket documentation it is
> valid to perform socket.bind(null) which will allow the OS routing table and
> DNS to send the traffic to the correct interface. I will also attach the
> nework traces and a test patch for 2.7.x and 3.x code base. I have this test
> fix below in my Hadoop Test Cluster.
> Client.java:
>
> |/*|
> | | * Bind the socket to the host specified in the principal name of the|
> | | * client, to ensure Server matching address of the client connection|
> | | * to host name in principal passed.|
> | | */|
> | |InetSocketAddress bindAddr = null;|
> | |if (ticket != null && ticket.hasKerberosCredentials()) {|
> | |KerberosInfo krbInfo =|
> | |remoteId.getProtocol().getAnnotation(KerberosInfo.class);|
> | |if (krbInfo != null) {|
> | |String principal = ticket.getUserName();|
> | |String host = SecurityUtil.getHostFromPrincipal(principal);|
> | |// If host name is a valid local address then bind socket to it|
> | |{color:#FF}*InetAddress localAddr =
> NetUtils.getLocalInetAddress(host);*{color}|
> |{color:#FF} ** {color}|if (localAddr != null) {|
> | |this.socket.setReuseAddress(true);|
> | |if (LOG.isDebugEnabled()) {|
> | |LOG.debug("Binding " + principal + " to " + localAddr);|
> | |}|
> | |*{color:#FF}bindAddr = new InetSocketAddress(localAddr, 0);{color}*|
> | *{color:#FF}{color}* |*{color:#FF}}{color}*|
> | |}|
> | |}|
>
> So in my Hadoop 2.7.x Cluster I made the following changes and traffic flows
> correctly out the correct interfaces:
>
> diff --git
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
>
> b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> index e1be271..c5b4a42 100644
> ---
> a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java
> +++
>
[jira] [Commented] (HADOOP-15256) Create docker images for latest stable hadoop3 build
[
https://issues.apache.org/jira/browse/HADOOP-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374401#comment-16374401
]
genericqa commented on HADOOP-15256:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15256 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911729/HADOOP-15256-docker-hadoop-3.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14201/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create docker images for latest stable hadoop3 build
>
>
> Key: HADOOP-15256
> URL: https://issues.apache.org/jira/browse/HADOOP-15256
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15256-docker-hadoop-3.001.patch
>
>
> Similar to the hadoop2 image we can provide a developer hadoop image which
> contains the latest hadoop from the binary release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[
https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374400#comment-16374400
]
genericqa commented on HADOOP-15084:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-15084 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15084 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911730/HADOOP-15084-docker-hadoop-2.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14200/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create docker images for latest stable hadoop2 build
>
>
> Key: HADOOP-15084
> URL: https://issues.apache.org/jira/browse/HADOOP-15084
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15084-docker-hadoop-2.001.patch,
> HADOOP-15084-docker-hadoop-2.002.patch
>
>
> HADOOP-15083 provides an empty runner container without the hadoop. The issue
> is creating a docker image based on the HADOOP-15083 image with adding the
> latest stable hadoop2 binary release to it.
> It also conains an example docker-compose file to run it.
> This image should be uploaded to the dockerhub and could be used for example
> configurations in the documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15260) Hive queries on tez are overwriting records on azure wasb storage.
Vineet Singh created HADOOP-15260: - Summary: Hive queries on tez are overwriting records on azure wasb storage. Key: HADOOP-15260 URL: https://issues.apache.org/jira/browse/HADOOP-15260 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 2.7.3 Environment: This scenario occurs on hdp 2.5(hadoop 2.7.3) hdfs on WASB microsoft Azure platform. The same query yields proper result on regular hdfs on hdp 2.5(hadoop 2.7.3) on premise cluster. Reporter: Vineet Singh Attachments: On Premise Cluster.JPG, azure cloud.JPG, sample_query.txt When running multiple hive queries on Tez (see example ) the same mapper task number gets overwritten by the next union query. As seen in the azure snapshot the directories /1 ,2 ...,100 get overwritten again and again since the mapper numbers launch write again and again in the same directories. But in the on premise hadoop cluster version 2.7.3 . The directories are created as 1_copy_0,1_copy_2 and so on. Creating copies does not overwrite the data. The queries would be usually 600-1000 queries union together. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15256) Create docker images for latest stable hadoop3 build
[ https://issues.apache.org/jira/browse/HADOOP-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15256: -- Attachment: HADOOP-15256-docker-hadoop-3.001.patch > Create docker images for latest stable hadoop3 build > > > Key: HADOOP-15256 > URL: https://issues.apache.org/jira/browse/HADOOP-15256 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15256-docker-hadoop-3.001.patch > > > Similar to the hadoop2 image we can provide a developer hadoop image which > contains the latest hadoop from the binary release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[ https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15084: -- Attachment: HADOOP-15084-docker-hadoop-2.002.patch > Create docker images for latest stable hadoop2 build > > > Key: HADOOP-15084 > URL: https://issues.apache.org/jira/browse/HADOOP-15084 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15084-docker-hadoop-2.001.patch, > HADOOP-15084-docker-hadoop-2.002.patch > > > HADOOP-15083 provides an empty runner container without the hadoop. The issue > is creating a docker image based on the HADOOP-15083 image with adding the > latest stable hadoop2 binary release to it. > It also conains an example docker-compose file to run it. > This image should be uploaded to the dockerhub and could be used for example > configurations in the documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15256) Create docker images for latest stable hadoop3 build
[ https://issues.apache.org/jira/browse/HADOOP-15256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15256: -- Target Version/s: 3.2.0 Status: Patch Available (was: Open) > Create docker images for latest stable hadoop3 build > > > Key: HADOOP-15256 > URL: https://issues.apache.org/jira/browse/HADOOP-15256 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15256-docker-hadoop-3.001.patch > > > Similar to the hadoop2 image we can provide a developer hadoop image which > contains the latest hadoop from the binary release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15258) Create example docker-compose file for documentations
[ https://issues.apache.org/jira/browse/HADOOP-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15258: -- Target Version/s: 3.2.0 Status: Patch Available (was: Open) > Create example docker-compose file for documentations > - > > Key: HADOOP-15258 > URL: https://issues.apache.org/jira/browse/HADOOP-15258 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15258.001.patch > > > An other user case for docker is to use it in the documentation. For example > in the HA documentation we can provide an example docker-compose file and > configuration with all the required settings to getting started easily with > an HA cluster. > 1. I would add an example to a documetation page > 2. It will use the hadoop3 image (which contains latest hadoop3) as the user > of the documentation may not build a hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15258) Create example docker-compose file for documentations
[ https://issues.apache.org/jira/browse/HADOOP-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15258: -- Attachment: HADOOP-15258.001.patch > Create example docker-compose file for documentations > - > > Key: HADOOP-15258 > URL: https://issues.apache.org/jira/browse/HADOOP-15258 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15258.001.patch > > > An other user case for docker is to use it in the documentation. For example > in the HA documentation we can provide an example docker-compose file and > configuration with all the required settings to getting started easily with > an HA cluster. > 1. I would add an example to a documetation page > 2. It will use the hadoop3 image (which contains latest hadoop3) as the user > of the documentation may not build a hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15259) Provide docker file for the development builds
[
https://issues.apache.org/jira/browse/HADOOP-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374330#comment-16374330
]
genericqa commented on HADOOP-15259:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15259 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911716/HADOOP-15259.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml shellcheck shelldocs |
| uname | Linux df702d6f38e6 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14197/testReport/ |
| Max. process+thread count | 329 (vs. ulimit of 1) |
| modules | C: hadoop-dist U: hadoop-dist |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14197/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Provide docker file for the development builds
> --
>
> Key: HADOOP-15259
>
[jira] [Resolved] (HADOOP-2829) JT should consider the disk each task is on before scheduling jobs...
[ https://issues.apache.org/jira/browse/HADOOP-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor resolved HADOOP-2829. -- Resolution: Invalid It seems obsolete. > JT should consider the disk each task is on before scheduling jobs... > - > > Key: HADOOP-2829 > URL: https://issues.apache.org/jira/browse/HADOOP-2829 > Project: Hadoop Common > Issue Type: Improvement >Reporter: eric baldeschwieler >Priority: Major > > The DataNode can support a JBOD config, where blocks exist on explicit disks. > But this information is not exported or considered by the JT when assigning > tasks. This leads to non-optimal disk use. if 4 slots are used, 2 running > tasks will likely be on the same disk and we observe them running more slowly > then other tasks on the same machine. > We could follow a number of strategies to address this. > for example: The data nodes could support a what disk is this block on call. > Then the JT could discover the info and assign jobs accordingly. > Of course the TT itself uses disks for merge and temp space and the datanodes > on the same machine can be used by off node sources, so it is not clear > optimizing all of this is simple enough to be worth it. > This issue deserves study. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15257) Provide example docker compose file for developer builds
[
https://issues.apache.org/jira/browse/HADOOP-15257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374322#comment-16374322
]
genericqa commented on HADOOP-15257:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911715/HADOOP-15257.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml shellcheck shelldocs |
| uname | Linux ed2cc6e72a32 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14196/testReport/ |
| Max. process+thread count | 319 (vs. ulimit of 1) |
| modules | C: hadoop-dist U: hadoop-dist |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14196/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Provide example docker compose file for developer builds
>
>
> Key:
[jira] [Resolved] (HADOOP-6291) Confusing warn message from Configuration
[
https://issues.apache.org/jira/browse/HADOOP-6291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor resolved HADOOP-6291.
--
Resolution: Duplicate
> Confusing warn message from Configuration
> -
>
> Key: HADOOP-6291
> URL: https://issues.apache.org/jira/browse/HADOOP-6291
> Project: Hadoop Common
> Issue Type: Bug
> Components: conf
>Affects Versions: 0.21.0
>Reporter: Tsz Wo Nicholas Sze
>Priority: Major
>
> Starting a cluster without setting mapreduce.task.attempt.id and then
> {noformat}
> $ ./bin/hadoop fs -put README.txt r.txt
> 09/09/29 22:28:10 WARN conf.Configuration: mapred.task.id is deprecated.
> Instead, use mapreduce.task.attempt.id
> 09/09/29 22:28:10 INFO hdfs.DFSClient: Done flushing
> 09/09/29 22:28:10 INFO hdfs.DFSClient: Closing the streams...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15252) Checkstyle version is not compatible with IDEA's checkstyle plugin
[
https://issues.apache.org/jira/browse/HADOOP-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374318#comment-16374318
]
genericqa commented on HADOOP-15252:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m
23s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15252 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911719/HADOOP-15252.002.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14198/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Checkstyle version is not compatible with IDEA's checkstyle plugin
> --
>
> Key: HADOOP-15252
> URL: https://issues.apache.org/jira/browse/HADOOP-15252
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-15252.001.patch, HADOOP-15252.002.patch,
> idea_checkstyle_settings.png
>
>
> After upgrading to the latest IDEA the IDE throws error messages in every few
> minutes like
> {code:java}
> The Checkstyle rules file could not be parsed.
> SuppressionCommentFilter is not allowed as a child in Checker
> The file has been blacklisted for 60s.{code}
> This is caused by some backward incompatible changes in checkstyle source
> code:
> [http://checkstyle.sourceforge.net/releasenotes.html]
> * 8.1: Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter
> children of TreeWalker.
> * 8.2: remove FileContentsHolder module as FileContents object is available
> for filters on TreeWalker in TreeWalkerAudit Event.
> IDEA uses checkstyle 8.8
> We should upgrade our checkstyle version to be compatible with IDEA's
> checkstyle plugin.
> Also it's a good time to upgrade maven-checkstyle-plugin as well to brand
> new 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15252) Checkstyle version is not compatible with IDEA's checkstyle plugin
[
https://issues.apache.org/jira/browse/HADOOP-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Bokor updated HADOOP-15252:
--
Attachment: HADOOP-15252.002.patch
> Checkstyle version is not compatible with IDEA's checkstyle plugin
> --
>
> Key: HADOOP-15252
> URL: https://issues.apache.org/jira/browse/HADOOP-15252
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-15252.001.patch, HADOOP-15252.002.patch,
> idea_checkstyle_settings.png
>
>
> After upgrading to the latest IDEA the IDE throws error messages in every few
> minutes like
> {code:java}
> The Checkstyle rules file could not be parsed.
> SuppressionCommentFilter is not allowed as a child in Checker
> The file has been blacklisted for 60s.{code}
> This is caused by some backward incompatible changes in checkstyle source
> code:
> [http://checkstyle.sourceforge.net/releasenotes.html]
> * 8.1: Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter
> children of TreeWalker.
> * 8.2: remove FileContentsHolder module as FileContents object is available
> for filters on TreeWalker in TreeWalkerAudit Event.
> IDEA uses checkstyle 8.8
> We should upgrade our checkstyle version to be compatible with IDEA's
> checkstyle plugin.
> Also it's a good time to upgrade maven-checkstyle-plugin as well to brand
> new 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15252) Checkstyle version is not compatible with IDEA's checkstyle plugin
[
https://issues.apache.org/jira/browse/HADOOP-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374316#comment-16374316
]
Andras Bokor commented on HADOOP-15252:
---
Attaching the same patch to kick Hadoop QA.
> Checkstyle version is not compatible with IDEA's checkstyle plugin
> --
>
> Key: HADOOP-15252
> URL: https://issues.apache.org/jira/browse/HADOOP-15252
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>Priority: Major
> Attachments: HADOOP-15252.001.patch, HADOOP-15252.002.patch,
> idea_checkstyle_settings.png
>
>
> After upgrading to the latest IDEA the IDE throws error messages in every few
> minutes like
> {code:java}
> The Checkstyle rules file could not be parsed.
> SuppressionCommentFilter is not allowed as a child in Checker
> The file has been blacklisted for 60s.{code}
> This is caused by some backward incompatible changes in checkstyle source
> code:
> [http://checkstyle.sourceforge.net/releasenotes.html]
> * 8.1: Make SuppressionCommentFilter and SuppressWithNearbyCommentFilter
> children of TreeWalker.
> * 8.2: remove FileContentsHolder module as FileContents object is available
> for filters on TreeWalker in TreeWalkerAudit Event.
> IDEA uses checkstyle 8.8
> We should upgrade our checkstyle version to be compatible with IDEA's
> checkstyle plugin.
> Also it's a good time to upgrade maven-checkstyle-plugin as well to brand
> new 3.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15259) Provide docker file for the development builds
[
https://issues.apache.org/jira/browse/HADOOP-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-15259:
--
Target Version/s: 3.2.0
Status: Patch Available (was: Open)
Initial patch is uploaded.
To test:
1. Do a full build with -Pdist
2. cd hadoop-dist/target
3. docker build -t yourname/hadoop .
Now you can use the customied image:
{code}
docker run yourname/hadoop hdfs datanode
{code}
Or you can push it to an external location (eg. kubernetes registry).
{code}
docker push yourname/hadoop
{code}
> Provide docker file for the development builds
> --
>
> Key: HADOOP-15259
> URL: https://issues.apache.org/jira/browse/HADOOP-15259
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15259.001.patch
>
>
> An other use case for using docker image is creating custom docker image
> (base image + custom hadoop build). The custom image could be used to test
> easily the hadoop build on external dockerized cluster (eg. Kubernetes)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15258) Create example docker-compose file for documentations
[ https://issues.apache.org/jira/browse/HADOOP-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15258: -- Summary: Create example docker-compose file for documentations (was: Create example docker-compse file for documentations) > Create example docker-compose file for documentations > - > > Key: HADOOP-15258 > URL: https://issues.apache.org/jira/browse/HADOOP-15258 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > An other user case for docker is to use it in the documentation. For example > in the HA documentation we can provide an example docker-compose file and > configuration with all the required settings to getting started easily with > an HA cluster. > 1. I would add an example to a documetation page > 2. It will use the hadoop3 image (which contains latest hadoop3) as the user > of the documentation may not build a hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15259) Provide docker file for the development builds
[ https://issues.apache.org/jira/browse/HADOOP-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15259: -- Attachment: HADOOP-15259.001.patch > Provide docker file for the development builds > -- > > Key: HADOOP-15259 > URL: https://issues.apache.org/jira/browse/HADOOP-15259 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15259.001.patch > > > An other use case for using docker image is creating custom docker image > (base image + custom hadoop build). The custom image could be used to test > easily the hadoop build on external dockerized cluster (eg. Kubernetes) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[
https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374286#comment-16374286
]
genericqa commented on HADOOP-15084:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
28s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15084 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911706/HADOOP-15084-docker-hadoop-2.001.patch
|
| Optional Tests | asflicense shellcheck shelldocs mvnsite |
| uname | Linux ba99b8bd06f6 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d1cd573 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14195/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 302 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14195/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Create docker images for latest stable hadoop2 build
>
>
> Key: HADOOP-15084
> URL: https://issues.apache.org/jira/browse/HADOOP-15084
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15084-docker-hadoop-2.001.patch
>
>
> HADOOP-15083 provides an empty runner container without the hadoop. The issue
> is creating a docker image based on the HADOOP-15083 image with adding the
> latest stable hadoop2 binary release to it.
> It also conains an example docker-compose file to run it.
> This image should be uploaded to the dockerhub and could be used for example
> configurations in the documentation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15257) Provide example docker compose file for developer builds
[
https://issues.apache.org/jira/browse/HADOOP-15257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HADOOP-15257:
--
Target Version/s: 3.2.0
Status: Patch Available (was: Open)
First patch is uploaded.
To test:
1. Do a full maven build with -Pdist
2. cd hadoop-dist/target/compose/hdfs
3. docker-compose up -d
Now you have a running hdfs cluster.
You can scale it up with :
{code}
docker-compoase scale datanode=3
{code}
And you can check the namenode ui http://localhost:9870
> Provide example docker compose file for developer builds
>
>
> Key: HADOOP-15257
> URL: https://issues.apache.org/jira/browse/HADOOP-15257
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HADOOP-15257.001.patch
>
>
> This issue is about creating example docker-compose files which use the
> latest build from the hadoop-dist directory.
> These docker-compose files would help to run a specific hadoop cluster based
> on the latest custom build without the need to build customized docker image
> (with mounting hadoop fro hadoop-dist to the container
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15257) Provide example docker compose file for developer builds
[ https://issues.apache.org/jira/browse/HADOOP-15257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15257: -- Attachment: HADOOP-15257.001.patch > Provide example docker compose file for developer builds > > > Key: HADOOP-15257 > URL: https://issues.apache.org/jira/browse/HADOOP-15257 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15257.001.patch > > > This issue is about creating example docker-compose files which use the > latest build from the hadoop-dist directory. > These docker-compose files would help to run a specific hadoop cluster based > on the latest custom build without the need to build customized docker image > (with mounting hadoop fro hadoop-dist to the container -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[ https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15084: -- Status: Patch Available (was: Open) > Create docker images for latest stable hadoop2 build > > > Key: HADOOP-15084 > URL: https://issues.apache.org/jira/browse/HADOOP-15084 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15084-docker-hadoop-2.001.patch > > > HADOOP-15083 provides an empty runner container without the hadoop. The issue > is creating a docker image based on the HADOOP-15083 image with adding the > latest stable hadoop2 binary release to it. > It also conains an example docker-compose file to run it. > This image should be uploaded to the dockerhub and could be used for example > configurations in the documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[ https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15084: -- Attachment: HADOOP-15084-docker-hadoop-2.001.patch > Create docker images for latest stable hadoop2 build > > > Key: HADOOP-15084 > URL: https://issues.apache.org/jira/browse/HADOOP-15084 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HADOOP-15084-docker-hadoop-2.001.patch > > > HADOOP-15083 provides an empty runner container without the hadoop. The issue > is creating a docker image based on the HADOOP-15083 image with adding the > latest stable hadoop2 binary release to it. > It also conains an example docker-compose file to run it. > This image should be uploaded to the dockerhub and could be used for example > configurations in the documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15258) Create example docker-compse file for documentations
Elek, Marton created HADOOP-15258: - Summary: Create example docker-compse file for documentations Key: HADOOP-15258 URL: https://issues.apache.org/jira/browse/HADOOP-15258 Project: Hadoop Common Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton An other user case for docker is to use it in the documentation. For example in the HA documentation we can provide an example docker-compose file and configuration with all the required settings to getting started easily with an HA cluster. 1. I would add an example to a documetation page 2. It will use the hadoop3 image (which contains latest hadoop3) as the user of the documentation may not build a hadoop -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15084) Create docker images for latest stable hadoop2 build
[ https://issues.apache.org/jira/browse/HADOOP-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HADOOP-15084: -- Description: HADOOP-15083 provides an empty runner container without the hadoop. The issue is creating a docker image based on the HADOOP-15083 image with adding the latest stable hadoop2 binary release to it. It also conains an example docker-compose file to run it. This image should be uploaded to the dockerhub and could be used for example configurations in the documentation. > Create docker images for latest stable hadoop2 build > > > Key: HADOOP-15084 > URL: https://issues.apache.org/jira/browse/HADOOP-15084 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > HADOOP-15083 provides an empty runner container without the hadoop. The issue > is creating a docker image based on the HADOOP-15083 image with adding the > latest stable hadoop2 binary release to it. > It also conains an example docker-compose file to run it. > This image should be uploaded to the dockerhub and could be used for example > configurations in the documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15257) Provide example docker compose file for developer builds
Elek, Marton created HADOOP-15257: - Summary: Provide example docker compose file for developer builds Key: HADOOP-15257 URL: https://issues.apache.org/jira/browse/HADOOP-15257 Project: Hadoop Common Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton This issue is about creating example docker-compose files which use the latest build from the hadoop-dist directory. These docker-compose files would help to run a specific hadoop cluster based on the latest custom build without the need to build customized docker image (with mounting hadoop fro hadoop-dist to the container -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15256) Create docker images for latest stable hadoop3 build
Elek, Marton created HADOOP-15256: - Summary: Create docker images for latest stable hadoop3 build Key: HADOOP-15256 URL: https://issues.apache.org/jira/browse/HADOOP-15256 Project: Hadoop Common Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton Similar to the hadoop2 image we can provide a developer hadoop image which contains the latest hadoop from the binary release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15259) Provide docker file for the development builds
Elek, Marton created HADOOP-15259: - Summary: Provide docker file for the development builds Key: HADOOP-15259 URL: https://issues.apache.org/jira/browse/HADOOP-15259 Project: Hadoop Common Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton An other use case for using docker image is creating custom docker image (base image + custom hadoop build). The custom image could be used to test easily the hadoop build on external dockerized cluster (eg. Kubernetes) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15255) Upper/Lower case conversion support for group names in LdapGroupsMapping
Nanda kumar created HADOOP-15255:
Summary: Upper/Lower case conversion support for group names in
LdapGroupsMapping
Key: HADOOP-15255
URL: https://issues.apache.org/jira/browse/HADOOP-15255
Project: Hadoop Common
Issue Type: Improvement
Reporter: Nanda kumar
On most LDAP servers the user and group names are case-insensitive. When we use
{{JniBasedUnixGroupsMappingWithFallback}} and have {{SSSD}} in place, it is
possible to configure {{SSSD}} to force the group names to be returned in
lowercase. If we use {{LdapGroupsMapping}}, we don't have any such option.
This jira proposes to introduce a new {{hadoop.security.group.mapping}}
implementation based on LdapGroupsMapping which supports force lower/upper case
group names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
