[jira] [Updated] (HADOOP-15158) AliyunOSS: Supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15158: - Summary: AliyunOSS: Supports role based credential (was: AliyunOSS: AliyunCredentialsProvider supports role based credential ) > AliyunOSS: Supports role based credential > - > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: 2.9.1, 3.0.1 > > Attachments: HADOOP-15158.001.patch > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info(role) from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312621#comment-16312621 ] wujinhu commented on HADOOP-15158: -- Attach patch. However, should we add an implementation for this? > AliyunOSS: AliyunCredentialsProvider supports role based credential > > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: 2.9.1, 3.0.1 > > Attachments: HADOOP-15158.001.patch > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info(role) from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu reassigned HADOOP-15158: Assignee: wujinhu > AliyunOSS: AliyunCredentialsProvider supports role based credential > > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Fix For: 2.9.1, 3.0.1 > > Attachments: HADOOP-15158.001.patch > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info(role) from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15158: - Attachment: HADOOP-15158.001.patch > AliyunOSS: AliyunCredentialsProvider supports role based credential > > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu > Fix For: 2.9.1, 3.0.1 > > Attachments: HADOOP-15158.001.patch > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info(role) from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15158: - Description: Currently, AliyunCredentialsProvider supports credential by configuration(core-site.xml). Sometimes, admin wants to create different temporary credential(key/secret/token) for different roles so that one role cannot read data that belongs to another role. So, our code should support pass in the URI when creates an XXXCredentialsProvider so that we can get user info(role) from the URI was: Currently, AliyunCredentialsProvider supports credential by configuration(core-site.xml). Sometimes, admin wants to create different temporary credential(key/secret/token) for different roles so that one role cannot read data that belongs to another role. So, our code should support pass in the URI when creates an XXXCredentialsProvider so that we can get user info from the URI > AliyunOSS: AliyunCredentialsProvider supports role based credential > > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu > Fix For: 2.9.1, 3.0.1 > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info(role) from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
[ https://issues.apache.org/jira/browse/HADOOP-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15158: - Description: Currently, AliyunCredentialsProvider supports credential by configuration(core-site.xml). Sometimes, admin wants to create different temporary credential(key/secret/token) for different roles so that one role cannot read data that belongs to another role. So, our code should support pass in the URI when creates an XXXCredentialsProvider so that we can get user info from the URI was: Currently, AliyunCredentialsProvider supports credential by configuration(core-site.xml). Sometimes, admin wants to create different temporary credential(key/secret/token) for different roles so that one role cannot read data that belongs to another role. So, our code should support pass in the URI when creates an XXXCredentialsProvider. > AliyunOSS: AliyunCredentialsProvider supports role based credential > > > Key: HADOOP-15158 > URL: https://issues.apache.org/jira/browse/HADOOP-15158 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu > Fix For: 2.9.1, 3.0.1 > > > Currently, AliyunCredentialsProvider supports credential by > configuration(core-site.xml). Sometimes, admin wants to create different > temporary credential(key/secret/token) for different roles so that one role > cannot read data that belongs to another role. > So, our code should support pass in the URI when creates an > XXXCredentialsProvider so that we can get user info from the URI -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15158) AliyunOSS: AliyunCredentialsProvider supports role based credential
wujinhu created HADOOP-15158: Summary: AliyunOSS: AliyunCredentialsProvider supports role based credential Key: HADOOP-15158 URL: https://issues.apache.org/jira/browse/HADOOP-15158 Project: Hadoop Common Issue Type: Improvement Components: fs/oss Affects Versions: 3.0.0 Reporter: wujinhu Fix For: 2.9.1, 3.0.1 Currently, AliyunCredentialsProvider supports credential by configuration(core-site.xml). Sometimes, admin wants to create different temporary credential(key/secret/token) for different roles so that one role cannot read data that belongs to another role. So, our code should support pass in the URI when creates an XXXCredentialsProvider. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14969) Improve diagnostics in secure DataNode startup
[
https://issues.apache.org/jira/browse/HADOOP-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312259#comment-16312259
]
genericqa commented on HADOOP-14969:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 154 unchanged - 0 fixed = 155 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
59s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Useless condition:haveResources == true at this point At
DataNode.java:[line 1503] |
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer |
| | hadoop.hdfs.server.federation.store.driver.TestStateStoreZK |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14969 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904678/HADOOP-14969.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 69068c757a2b 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |
[jira] [Updated] (HADOOP-12611) TestZKSignerSecretProvider#testMultipleInit occasionally fail
[ https://issues.apache.org/jira/browse/HADOOP-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated HADOOP-12611: --- Fix Version/s: 2.7.6 Cherry-picked to branch-2.7 so we could commit HADOOP-14246. > TestZKSignerSecretProvider#testMultipleInit occasionally fail > - > > Key: HADOOP-12611 > URL: https://issues.apache.org/jira/browse/HADOOP-12611 > Project: Hadoop Common > Issue Type: Bug >Reporter: Wei-Chiu Chuang >Assignee: Eric Badger > Fix For: 2.8.0, 3.0.0-alpha2, 2.7.6 > > Attachments: HADOOP-12611.001.patch, HADOOP-12611.002.patch, > HADOOP-12611.003.patch, HADOOP-12611.004.patch, HADOOP-12611.005.patch, > HADOOP-12611.006.patch > > > https://builds.apache.org/job/Hadoop-Common-trunk/2053/testReport/junit/org.apache.hadoop.security.authentication.util/TestZKSignerSecretProvider/testMultipleInit/ > Error Message > expected null, but was:<[B@142bad79> > Stacktrace > java.lang.AssertionError: expected null, but was:<[B@142bad79> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotNull(Assert.java:664) > at org.junit.Assert.assertNull(Assert.java:646) > at org.junit.Assert.assertNull(Assert.java:656) > at > org.apache.hadoop.security.authentication.util.TestZKSignerSecretProvider.testMultipleInit(TestZKSignerSecretProvider.java:149) > I think the failure was introduced after HADOOP-12181 > This is likely where the root cause is: > 2015-11-29 00:24:33,325 ERROR ZKSignerSecretProvider - An unexpected > exception occurred while pulling data fromZooKeeper > java.lang.IllegalStateException: instance must be started before calling this > method > at > com.google.common.base.Preconditions.checkState(Preconditions.java:145) > at > org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:363) > at > org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider.pullFromZK(ZKSignerSecretProvider.java:341) > at > org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider.rollSecret(ZKSignerSecretProvider.java:264) > at > org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider$$EnhancerByMockitoWithCGLIB$$575f06d8.CGLIB$rollSecret$2() > at > org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider$$EnhancerByMockitoWithCGLIB$$575f06d8$$FastClassByMockitoWithCGLIB$$6f94a716.invoke() > at org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216) > at > org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10) > at > org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22) > at > org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27) > at > org.mockito.internal.invocation.Invocation.callRealMethod(Invocation.java:211) > at > org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36) > at org.mockito.internal.MockHandler.handle(MockHandler.java:99) > at > org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:47) > at > org.apache.hadoop.security.authentication.util.ZKSignerSecretProvider$$EnhancerByMockitoWithCGLIB$$575f06d8.rollSecret() > at > org.apache.hadoop.security.authentication.util.RolloverSignerSecretProvider$1.run(RolloverSignerSecretProvider.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12181) Fix intermittent test failure of TestZKSignerSecretProvider
[
https://issues.apache.org/jira/browse/HADOOP-12181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-12181:
---
Fix Version/s: 2.7.6
Cherry-picked to branch-2.7 so we could commit HADOOP-12611 and HADOOP-14246.
> Fix intermittent test failure of TestZKSignerSecretProvider
> ---
>
> Key: HADOOP-12181
> URL: https://issues.apache.org/jira/browse/HADOOP-12181
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.6
>
> Attachments: HADOOP-12181.001.patch
>
>
> TestZKSignerSecretProvider wait for the condition by just sleeping
> {{rolloverFrequency + 2000}} millisecs. Depending on timing only is fragile
> and it makes test time unnecessary long.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312223#comment-16312223
]
Robert Kanter commented on HADOOP-14246:
It applies cleanly if I also first backport HADOOP-12181 and HADOOP-12611,
which also apply cleanly.
So I went ahead and backported all three to branch-2.7.
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4, 2.7.6
>
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated HADOOP-14246:
---
Fix Version/s: 2.7.6
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4, 2.7.6
>
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312157#comment-16312157 ] Xiao Chen commented on HADOOP-15006: Had a sync with [~moist], summarizing some points discussed: - Using a new hadoop crypto would give us more freedom and cleaner interfaces. - We can be fancy to support hybrid hdfs + s3 setup in crypto commands, but perhaps only makes sense if we actually support hybrid hadoop cluster. - I'm okay with option 5 to store BEZI and OEMI, but we should get [~fabbri]'s comments from his past experience. - No support on messing with things directly in S3. - Atomicity is delegated to DynamoDB (read after write), and it seems there should not be scenarios where the BEZI and OEMI mismatching with the actual dir / file in DynamoDB. Need to think about this carefully though since any miss could become a data loss > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15107) Prove the correctness of the new committers, or fix where they are not correct
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312122#comment-16312122 ] Steve Loughran commented on HADOOP-15107: - Draft 001: A zero rename committer https://github.com/steveloughran/zero-rename-committer/releases/tag/tag_draft_001 * would like some review of my documentation of the Hadoop and spark commit algorithms * no benchmarks yet > Prove the correctness of the new committers, or fix where they are not correct > -- > > Key: HADOOP-15107 > URL: https://issues.apache.org/jira/browse/HADOOP-15107 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > I'm writing about the paper on the committers, one which, being a proper > paper, requires me to show the committers work. > # define the requirements of a "Correct" committed job (this applies to the > FileOutputCommitter too) > # show that the Staging committer meets these requirements (most of this is > implicit in that it uses the V1 FileOutputCommitter to marshall .pendingset > lists from committed tasks to the final destination, where they are read and > committed. > # Show the magic committer also works. > I'm now not sure that the magic committer works. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312115#comment-16312115
]
genericqa commented on HADOOP-15156:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
58s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 1
new + 31 unchanged - 0 fixed = 32 total (was 31) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
4s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | HADOOP-15156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904677/HADOOP-15156-branch-2-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7d1dab55d3de 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 42964c2 |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13923/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13923/testReport/ |
| Max. process+thread count | 195 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13923/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> backport HADOOP-15086 rename fix to branch-2
>
>
> Key: HADOOP-15156
> URL: https://issues.apache.org/jira/browse/HADOOP-15156
> Project: Hadoop
[jira] [Commented] (HADOOP-14969) Improve diagnostics in secure DataNode startup
[
https://issues.apache.org/jira/browse/HADOOP-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312079#comment-16312079
]
Ajay Kumar commented on HADOOP-14969:
-
Added initial patch for review. [~arpitagarwal],[~ste...@apache.org] could you
please have a look. Error message points to existing documentation for secure
datanode in error message. (i.e
https://hadoop.apache.org/docs/current/hadoop-project-dist/ )
> Improve diagnostics in secure DataNode startup
> --
>
> Key: HADOOP-14969
> URL: https://issues.apache.org/jira/browse/HADOOP-14969
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HADOOP-14969.001.patch
>
>
> When DN secure mode configuration is incorrect, it throws the following
> exception from Datanode#checkSecureConfig
> {code}
> private static void checkSecureConfig(DNConf dnConf, Configuration conf,
> SecureResources resources) throws RuntimeException {
> if (!UserGroupInformation.isSecurityEnabled()) {
> return;
> }
> ...
> throw new RuntimeException("Cannot start secure DataNode without " +
> "configuring either privileged resources or SASL RPC data transfer " +
> "protection and SSL for HTTP. Using privileged resources in " +
> "combination with SASL RPC data transfer protection is not supported.");
> {code}
> The DN should print more useful diagnostics as to what exactly what went
> wrong.
> Also when starting secure DN with resources then the startup scripts should
> launch the SecureDataNodeStarter class. If no SASL is configured and
> SecureDataNodeStarter is not used, then we could mention that too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14969) Improve diagnostics in secure DataNode startup
[
https://issues.apache.org/jira/browse/HADOOP-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HADOOP-14969:
Status: Patch Available (was: Open)
> Improve diagnostics in secure DataNode startup
> --
>
> Key: HADOOP-14969
> URL: https://issues.apache.org/jira/browse/HADOOP-14969
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HADOOP-14969.001.patch
>
>
> When DN secure mode configuration is incorrect, it throws the following
> exception from Datanode#checkSecureConfig
> {code}
> private static void checkSecureConfig(DNConf dnConf, Configuration conf,
> SecureResources resources) throws RuntimeException {
> if (!UserGroupInformation.isSecurityEnabled()) {
> return;
> }
> ...
> throw new RuntimeException("Cannot start secure DataNode without " +
> "configuring either privileged resources or SASL RPC data transfer " +
> "protection and SSL for HTTP. Using privileged resources in " +
> "combination with SASL RPC data transfer protection is not supported.");
> {code}
> The DN should print more useful diagnostics as to what exactly what went
> wrong.
> Also when starting secure DN with resources then the startup scripts should
> launch the SecureDataNodeStarter class. If no SASL is configured and
> SecureDataNodeStarter is not used, then we could mention that too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14969) Improve diagnostics in secure DataNode startup
[
https://issues.apache.org/jira/browse/HADOOP-14969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HADOOP-14969:
Attachment: HADOOP-14969.001.patch
> Improve diagnostics in secure DataNode startup
> --
>
> Key: HADOOP-14969
> URL: https://issues.apache.org/jira/browse/HADOOP-14969
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HADOOP-14969.001.patch
>
>
> When DN secure mode configuration is incorrect, it throws the following
> exception from Datanode#checkSecureConfig
> {code}
> private static void checkSecureConfig(DNConf dnConf, Configuration conf,
> SecureResources resources) throws RuntimeException {
> if (!UserGroupInformation.isSecurityEnabled()) {
> return;
> }
> ...
> throw new RuntimeException("Cannot start secure DataNode without " +
> "configuring either privileged resources or SASL RPC data transfer " +
> "protection and SSL for HTTP. Using privileged resources in " +
> "combination with SASL RPC data transfer protection is not supported.");
> {code}
> The DN should print more useful diagnostics as to what exactly what went
> wrong.
> Also when starting secure DN with resources then the startup scripts should
> launch the SecureDataNodeStarter class. If no SASL is configured and
> SecureDataNodeStarter is not used, then we could mention that too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15156: -- Attachment: HADOOP-15156-branch-2-001.patch Correcting the name of the patch: HADOOP-15156-branch-2-001.patch. > backport HADOOP-15086 rename fix to branch-2 > > > Key: HADOOP-15156 > URL: https://issues.apache.org/jira/browse/HADOOP-15156 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-15156-branch-2-001.patch > > > backport HADOOP-15086 (rename fix) to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15156: -- Attachment: (was: HADOOP-15156-001.patch) > backport HADOOP-15086 rename fix to branch-2 > > > Key: HADOOP-15156 > URL: https://issues.apache.org/jira/browse/HADOOP-15156 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > > backport HADOOP-15086 (rename fix) to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[
https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312060#comment-16312060
]
genericqa commented on HADOOP-15156:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15156 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904676/HADOOP-15156-001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13922/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> backport HADOOP-15086 rename fix to branch-2
>
>
> Key: HADOOP-15156
> URL: https://issues.apache.org/jira/browse/HADOOP-15156
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Thomas Marquardt
>Assignee: Thomas Marquardt
> Attachments: HADOOP-15156-001.patch
>
>
> backport HADOOP-15086 (rename fix) to branch-2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15156: -- Target Version/s: 2.7.5, 2.9.0 (was: 2.9.0, 2.7.5) Status: Patch Available (was: Open) > backport HADOOP-15086 rename fix to branch-2 > > > Key: HADOOP-15156 > URL: https://issues.apache.org/jira/browse/HADOOP-15156 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-15156-001.patch > > > backport HADOOP-15086 (rename fix) to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15156) backport HADOOP-15086 rename fix to branch-2
[ https://issues.apache.org/jira/browse/HADOOP-15156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Marquardt updated HADOOP-15156: -- Attachment: HADOOP-15156-001.patch Attaching branch-2 patch HADOOP-15156-001.patch. Fix is identical to the fix for HADOOP-15086, except the test case was rewritten without the use of lambdas. All tests are passing against my West US 2 Azure Storage account. Results for "mvn -T 1C clean verify": Tests run: 242, Failures: 0, Errors: 0, Skipped: 4 and Tests run: 559, Failures: 0, Errors: 0, Skipped: 12 > backport HADOOP-15086 rename fix to branch-2 > > > Key: HADOOP-15156 > URL: https://issues.apache.org/jira/browse/HADOOP-15156 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Reporter: Thomas Marquardt >Assignee: Thomas Marquardt > Attachments: HADOOP-15156-001.patch > > > backport HADOOP-15086 (rename fix) to branch-2 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15086) NativeAzureFileSystem file rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311755#comment-16311755 ] Yin Huai commented on HADOOP-15086: --- thank you [~tmarquardt]! > NativeAzureFileSystem file rename is not atomic > --- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Shixiong Zhu >Assignee: Thomas Marquardt > Fix For: 3.1.0, 3.0.1 > > Attachments: HADOOP-15086-001.patch, HADOOP-15086-002.patch, > RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[
https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311664#comment-16311664
]
genericqa commented on HADOOP-15157:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m
34s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 34s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 4s{color} | {color:orange} root: The patch generated 4 new + 83 unchanged -
0 fixed = 87 total (was 83) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
38s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15157 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904597/HADOOP-15157.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 57a765ee9324 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2a48b35 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| compile |
[jira] [Commented] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311658#comment-16311658
]
Robert Kanter commented on HADOOP-14246:
I can take a look later today.
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4
>
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311646#comment-16311646 ] Larry McCay commented on HADOOP-15157: -- Hi [~grepas] - this is a good idea. Couple comments/questions: 1. The general implementation pattern doesn't have the URIs set as the param value as far as I know - I would have expected to either use the same credential.provider.path property to have a credential store for zkAuth or to have a separate property for zkAuth credential providers path and no value set for the property itself. The latter usually only needed when the global path would be inappropriate for the usage at hand. Having to set the URI at the individual property level could lead to a proliferation of credential stores and/or difficulty in keeping redundant URIs in sync across multiple properties. 2. I am missing where you are setting the value as the credential.provider.path in conf so that conf.getPassword will find it (maybe it is there and I am just not seeing it) 3. it appears that ZKUtil.BadAuthFormatException is also thrown from getZKAuthInfos but is missing from javadoc (was previously as well) 4. credential provider docs would also need to be updated to reflect this new usage - see http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Attachments: HADOOP-15157.000.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the value of these properties can also be > CredentialProvider URI-s (such as a jceks or localjceks URI). This allows > users to point to an encrypted store containing the authentication info. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311600#comment-16311600 ] wujinhu commented on HADOOP-15027: -- Updated the patch! Thanks [~Sammi] for the review. Here are my updates for your comments: 1 & 2 fixed 3. store.close() will not throw any exception, OSSClient.shutdown() catches Exception. 4. Yes, fs.oss.max.total.tasks is the max queue length used for read ahead. 5. fsDataInputStream.seek() is not missed, you can see it in 007.patch 6. I have added another test case for AliyunOSSFileReaderTask and will continue to add more test cases. Please help to review again. Thanks > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311586#comment-16311586
]
genericqa commented on HADOOP-15027:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904619/HADOOP-15027.009.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 023a52d6d7ab 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 45a4719 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13921/testReport/ |
| Max. process+thread count | 340 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13921/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to
> Aliyun OSS performance
>
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: HADOOP-15027.009.patch > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch, > HADOOP-15027.009.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311493#comment-16311493
]
genericqa commented on HADOOP-15027:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 9s{color} | {color:orange} hadoop-tools/hadoop-aliyun: The patch generated
3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-aliyun in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904604/HADOOP-15027.008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3e2253252546 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2a48b35 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13920/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aliyun.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13920/testReport/ |
| Max. process+thread count | 314 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-aliyun U: hadoop-tools/hadoop-aliyun |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13920/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: HADOOP-15027.008.patch > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: (was: HADOOP-15027.008.patch) > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergo Repas updated HADOOP-15157: - Status: Patch Available (was: Open) > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Attachments: HADOOP-15157.000.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the value of these properties can also be > CredentialProvider URI-s (such as a jceks or localjceks URI). This allows > users to point to an encrypted store containing the authentication info. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311345#comment-16311345
]
genericqa commented on HADOOP-15027:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m
20s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15027 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904599/HADOOP-15027.008.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13918/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to
> Aliyun OSS performance
> --
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch,
> HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch
>
>
> Currently, AliyunOSSInputStream uses single thread to read data from
> AliyunOSS, so we can do some refactoring by using multi-thread pre-read to
> improve read performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wujinhu updated HADOOP-15027: - Attachment: HADOOP-15027.008.patch > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch, HADOOP-15027.008.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
[ https://issues.apache.org/jira/browse/HADOOP-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergo Repas updated HADOOP-15157: - Attachment: HADOOP-15157.000.patch > Zookeeper authentication related properties to support CredentialProviders > -- > > Key: HADOOP-15157 > URL: https://issues.apache.org/jira/browse/HADOOP-15157 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Gergo Repas >Assignee: Gergo Repas >Priority: Minor > Attachments: HADOOP-15157.000.patch > > > The hadoop.zk.auth and ha.zookeeper.auth properties currently support either > a plain-text authentication info (in scheme:value format), or a > @/path/to/file notation which points to a plain-text file. > This ticket proposes that the value of these properties can also be > CredentialProvider URI-s (such as a jceks or localjceks URI). This allows > users to point to an encrypted store containing the authentication info. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15157) Zookeeper authentication related properties to support CredentialProviders
Gergo Repas created HADOOP-15157: Summary: Zookeeper authentication related properties to support CredentialProviders Key: HADOOP-15157 URL: https://issues.apache.org/jira/browse/HADOOP-15157 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Gergo Repas Assignee: Gergo Repas Priority: Minor The hadoop.zk.auth and ha.zookeeper.auth properties currently support either a plain-text authentication info (in scheme:value format), or a @/path/to/file notation which points to a plain-text file. This ticket proposes that the value of these properties can also be CredentialProvider URI-s (such as a jceks or localjceks URI). This allows users to point to an encrypted store containing the authentication info. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[
https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311155#comment-16311155
]
SammiChen commented on HADOOP-15027:
Thanks [~wujinhu] for working on it. some comments,
1. DEFAULT_MAX_TOTAL_TASKS = 128; the naming pattern is not consistent
with others. Put the default as suffix.
2. take care of all the checkstyle issues
3. {quote}
store.close();
boundedThreadPool.shutdown();
{quote}
will store.close throw any exception so that boundedThreadPool.shutdown
be skipped?
4. "fs.oss.max.total.tasks" is the maximum of waiting queue length, right?
5. It seems fsDataInputStream.seek() is missed between the two asserts.
{quote}
assertTrue("expected position at:" + 0 + ", but got:"
+ fsDataInputStream.getPos(), fsDataInputStream.getPos() == 0);
assertTrue("expected position at:" + 1048576 + ", but got:"
+ in.getExpectNextPos(),
in.getExpectNextPos() == 1048576);
{quote}
6. Can we add more test cases to cover the failure cases, to verify the error
handling functions correctly.
> AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to
> Aliyun OSS performance
> --
>
> Key: HADOOP-15027
> URL: https://issues.apache.org/jira/browse/HADOOP-15027
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/oss
>Affects Versions: 3.0.0
>Reporter: wujinhu
>Assignee: wujinhu
> Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch,
> HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch,
> HADOOP-15027.006.patch, HADOOP-15027.007.patch
>
>
> Currently, AliyunOSSInputStream uses single thread to read data from
> AliyunOSS, so we can do some refactoring by using multi-thread pre-read to
> improve read performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Summary: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance (was: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performa) > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performa
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Summary: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performa (was: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance) > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performa > --- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HADOOP-14246:
--
Fix Version/s: 2.8.4
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha4, 2.8.4
>
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14246) Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets
[
https://issues.apache.org/jira/browse/HADOOP-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1630#comment-1630
]
Brahma Reddy Battula commented on HADOOP-14246:
---
Thanks [~vinayrpet] for heading up.I too feel, this should land.
Pushed to {{branch-2.8}},since it's clean merge,Ran tests locally.
There are conflicts for {{branch-2.7}},[~rkanter] will you upload the patch for
{{branch-2.7}} Or shall I resolve the conflicts?
> Authentication Tokens should use SecureRandom instead of Random and 256 bit
> secrets
> ---
>
> Key: HADOOP-14246
> URL: https://issues.apache.org/jira/browse/HADOOP-14246
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.9.0
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0-alpha4
>
> Attachments: HADOOP-14246.001.patch
>
>
> {{RandomSignerSecretProvider}} and {{ZKSignerSecretProvider}} currently use a
> {{long}} generated by {{Random}} (which is then converted to a {{String}} and
> is 160 bits) for secrets.
> We should improve this to use 256 bit secrets generated by {{SecureRandom}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15027) AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance
[ https://issues.apache.org/jira/browse/HADOOP-15027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HADOOP-15027: --- Summary: AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to Aliyun OSS performance (was: AliyunOSS: Improvements for Hadoop read from AliyunOSS) > AliyunOSS: Support multi-thread pre-read to improve read from Hadoop to > Aliyun OSS performance > -- > > Key: HADOOP-15027 > URL: https://issues.apache.org/jira/browse/HADOOP-15027 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/oss >Affects Versions: 3.0.0 >Reporter: wujinhu >Assignee: wujinhu > Attachments: HADOOP-15027.001.patch, HADOOP-15027.002.patch, > HADOOP-15027.003.patch, HADOOP-15027.004.patch, HADOOP-15027.005.patch, > HADOOP-15027.006.patch, HADOOP-15027.007.patch > > > Currently, AliyunOSSInputStream uses single thread to read data from > AliyunOSS, so we can do some refactoring by using multi-thread pre-read to > improve read performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
