[jira] [Updated] (HADOOP-12754) Client.handleSaslConnectionFailure() uses wrong user in exception text
[
https://issues.apache.org/jira/browse/HADOOP-12754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12754:
Attachment: HADOOP-12754-001.patch
switch to log current user, i.e. whoever is actually trying to make the call
that just failed
> Client.handleSaslConnectionFailure() uses wrong user in exception text
> --
>
> Key: HADOOP-12754
> URL: https://issues.apache.org/jira/browse/HADOOP-12754
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc, security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
> Attachments: HADOOP-12754-001.patch
>
>
> {{Client.handleSaslConnectionFailure()}} includes the user in SASL failure
> messages, but it calls {{UGI.getLoginUser()}} for its text. If there's an
> auth problem in a {{doAs()}} context, this exception is fundamentally
> misleading
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126898#comment-15126898
]
Terry Siu commented on HADOOP-3733:
---
Howdy! Given the lack of activity/comments for almost a year, can I assume this
issue is dead and regenerating the key is the acceptable solution? One of my
buckets has '/' in the secrete key and I just hit this issue using s3a. Tried
setting fs.s3a.access.key and fs.s3a.secret.key config in the CLI and no luck.
Anyone got a workaround other than key regeneration using s3a?
> "s3:" URLs break when Secret Key contains a slash, even if encoded
> --
>
> Key: HADOOP-3733
> URL: https://issues.apache.org/jira/browse/HADOOP-3733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 0.17.1, 2.0.2-alpha
>Reporter: Stuart Sierra
>Priority: Minor
> Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733.patch,
> hadoop-3733.patch
>
>
> When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line,
> distcp fails if the SECRET contains a slash, even when the slash is
> URL-encoded as %2F.
> Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
> And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> And your bucket is called "mybucket"
> You can URL-encode the Secret KKey as
> Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> But this doesn't work:
> {noformat}
> $ bin/hadoop distcp file:///source
> s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
> 08/07/09 15:05:22 INFO util.CopyFiles:
> destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket:
> mybucket
> org.jets3t.service.S3ServiceException: S3 HEAD request failed.
> ResponseCode=403, ResponseMessage=Forbidden
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
> ...
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message:
> encoding="UTF-8"?>SignatureDoesNotMatchThe
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.
> at
> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-12699:
---
Attachment: HADOOP-12699.08.patch
Had offline discussion with [~asuresh], [~andrew.wang] and [~zhz]. The
conclusion is that the scenario is not supported, and we should just fix the
unittest to make sure new key is generated after draining the queue. We should
also update docs about this behavior due to caching, to set user expectation.
Patch 08 fixes the unit tests.
I didn't find a good place to explicitly document this, so just put it on the
javadoc. Please advice if there's a better place I overlooked.
Thanks all for the discussions and reviews.
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2,
> HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10969) RawLocalFileSystem.setPermission throws Exception on windows
[
https://issues.apache.org/jira/browse/HADOOP-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126925#comment-15126925
]
ASF GitHub Bot commented on HADOOP-10969:
-
Github user velo commented on the pull request:
https://github.com/apache/incubator-tinkerpop/pull/207#issuecomment-178164276
the windows profile is to work around
https://issues.apache.org/jira/browse/HADOOP-10969
and
https://issues.apache.org/jira/browse/SPARK-6961
> RawLocalFileSystem.setPermission throws Exception on windows
>
>
> Key: HADOOP-10969
> URL: https://issues.apache.org/jira/browse/HADOOP-10969
> Project: Hadoop Common
> Issue Type: Bug
> Environment: hadoop 2.3.0, Windows Environment, Development using
> Eclipse, Lenevo Laptop
>Reporter: Venkatesh
>Priority: Blocker
>
> I'm an application developer. We recently moved from CDH4.7 to CDH5.1. The
> hadoop version have been from 1.x to 2.x. In order to perform development on
> Eclipse (on WINDOWS), the following class was created
> public class WindowsLocalFileSystem extends LocalFileSystem {
> public WindowsLocalFileSystem() {
> super();
> }
> @Override
> public boolean mkdirs(Path f, FsPermission permission) throws
> IOException {
> final boolean result = super.mkdirs(f);
> this.setPermission(f, permission);
> return result;
>
> }
> @Override
> public void setPermission(Path p, FsPermission permission)
> throws IOException {
> try {
> super.setPermission(p, permission);
> } catch (final IOException e) {
> System.err.println("Cant help it, hence ignoring
> IOException setting persmission for path \"" + p +
>"\": " + e.getMessage());
> }
> }
> }
> This class was used in MapReduce Job as
> if (RUN_LOCAL) {
> conf.set("fs.default.name", "file:///");
> conf.set("mapred.job.tracker", "local");
> conf.set("fs.file.impl",
>
> "org.scif.bdp.mrjobs.WindowsLocalFileSystem");
> conf.set(
> "io.serializations",
>
> "org.apache.hadoop.io.serializer.JavaSerialization,"
> +
> "org.apache.hadoop.io.serializer.WritableSerialization");
> }
> It worked fine on CDH4.7. Now the same code when compiled on CDH5.1 works but
> when I try to execute it throws the following stacktrace
> Exception in thread "main" java.lang.NullPointerException
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:451)
> at org.apache.hadoop.util.Shell.run(Shell.java:424)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:656)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:745)
> at org.apache.hadoop.util.Shell.execCommand(Shell.java:728)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:633)
> at
> org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:467)
> at
> com.scif.bdp.common.WindowsLocalFileSystem.setPermission(WindowsLocalFileSystem.java:26)
> at
> com.scif.bdp.common.WindowsLocalFileSystem.mkdirs(WindowsLocalFileSystem.java:17)
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1313)
> at com.scif.bdp.mrjobs.DeDup.run(DeDup.java:55)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at com.scif.bdp.mrjobs.DeDup.main(DeDup.java:59)
> (Note DeDup is my MR class to remove duplicates)
> Upon investigation the only change I saw was the change in method
> .setPermission(). It invokes Native.POSIX.chmod as against Native.chmod
--
This message was sent by Atlassian JIRA
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127042#comment-15127042
]
Hadoop QA commented on HADOOP-12699:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
15s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
59s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
21s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 8s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s
{color} | {color:green} hadoop-kms in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 14s {color}
| {color:red} hadoop-kms in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_91 Failed junit tests | hadoop.crypto.key.kms.server.TestKMS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785579/HADOOP-12699.08.patch
|
| JIRA Issue | HADOOP-12699 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
[jira] [Commented] (HADOOP-12754) Client.handleSaslConnectionFailure() uses wrong user in exception text
[
https://issues.apache.org/jira/browse/HADOOP-12754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126880#comment-15126880
]
Steve Loughran commented on HADOOP-12754:
-
While I'm at it, why does that code try and relogin as the current user
*without checking to see if the caller is the current user?* Again, this is of
limited use as all it could do is encounter KDC connectivity problems and
needless delays.
HADOOP-6706 is the source of that code...I don't see any discussion of that
topic there
> Client.handleSaslConnectionFailure() uses wrong user in exception text
> --
>
> Key: HADOOP-12754
> URL: https://issues.apache.org/jira/browse/HADOOP-12754
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ipc, security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>
> {{Client.handleSaslConnectionFailure()}} includes the user in SASL failure
> messages, but it calls {{UGI.getLoginUser()}} for its text. If there's an
> auth problem in a {{doAs()}} context, this exception is fundamentally
> misleading
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12753) S3A jUnit tests failing if using HTTP proxy
[ https://issues.apache.org/jira/browse/HADOOP-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoran Rajic updated HADOOP-12753: - Attachment: HADOOP-12753-s3a-jUnits.diff Fix attached -- summary of changes: * have to "clean up" the general HTTP proxy settings before the "proxy tests", so they don't interfere with the individual jUnit test environment * additionally, add a JavaDoc to a jUnit test, pointing out which parameters combination could cause the test to hang or fail --- Testing notes: * the jUnit test listed in Description field is now passing: ~/work/hadoop_git/hadoop-tools/hadoop-aws(branch-2.8)$ mvn clean test -Dtest=TestS3AConfiguration [...] --- T E S T S --- Running org.apache.hadoop.fs.s3a.TestS3AConfiguration Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.906 sec - in org.apache.hadoop.fs.s3a.TestS3AConfiguration Results : Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 > S3A jUnit tests failing if using HTTP proxy > --- > > Key: HADOOP-12753 > URL: https://issues.apache.org/jira/browse/HADOOP-12753 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 > Environment: Test hardware: > * issue identified while running jUnit tests on MacBook Pro > PREREQUISITE: > * have a functional HTTP proxy running (ie. SQUID HTTP proxy @ > http://127.0.0.1:3128) > * S3A test configuration (relevant options): > > > fs.s3a.endpoint > > > > > fs.s3a.proxy.host > 127.0.0.1 > > > fs.s3a.proxy.port > 3128 > > > fs.s3a.access.key > _REAL_AWS_S3_ACCESS_KEY > > > fs.s3a.secret.key > _REAL_AWS_S3_SECRET_KEY > >Reporter: Zoran Rajic >Priority: Minor > Attachments: HADOOP-12753-s3a-jUnits.diff > > Original Estimate: 24h > Remaining Estimate: 24h > > Some companies have restricted firewalls and don't allow the "direct access" > to Web-pages or Internet, so their employees have to use the SOCKS or the > HTTP proxy. > This also means that the engineers who develop and test the Hadoop code and > s3a:// -filesystem inside the company firewalls, need to access the > "external" Amazon S3 service via the HTTP proxy. > Unfortunately, there are S3A jUnit test failures when run with the HTTP proxy > (see Environment field above). > -- > Details: > * steps to reproduce: > cd hadoop_git/hadoop-tools/hadoop-aws > mvn clean test -Dtest=TestS3AConfiguration > [...] > Running org.apache.hadoop.fs.s3a.TestS3AConfiguration > Tests run: 5, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 2.737 sec > <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration > > TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration) > Time elapsed: 0.737 sec <<< FAILURE! > java.lang.AssertionError: Expected a connection error for proxy server > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:130) > TestProxyPortWithoutHost(org.apache.hadoop.fs.s3a.TestS3AConfiguration) > Time elapsed: 0.624 sec <<< ERROR! > com.amazonaws.AmazonClientException: Unable to execute HTTP request: > Connection to http://127.0.0.1:1 refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at > java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) > at > java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) > at > java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:589) > at > org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) > at > org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) > at > org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) > at > org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > at >
[jira] [Updated] (HADOOP-12744) Refactoring of checksum failure report related codes
[
https://issues.apache.org/jira/browse/HADOOP-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12744:
---
Attachment: HADOOP-12744-v2.patch
Refined the patch some bit.
> Refactoring of checksum failure report related codes
>
>
> Key: HADOOP-12744
> URL: https://issues.apache.org/jira/browse/HADOOP-12744
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12744-v1.patch, HADOOP-12744-v2.patch
>
>
> This was from discussion with [~jingzhao] in HDFS-9646. There is some
> duplicate codes between client and datanode sides:
> {code}
> private void addCorruptedBlock(ExtendedBlock blk, DatanodeInfo node,
> Map corruptionMap) {
> Set dnSet = corruptionMap.get(blk);
> if (dnSet == null) {
> dnSet = new HashSet<>();
> corruptionMap.put(blk, dnSet);
> }
> if (!dnSet.contains(node)) {
> dnSet.add(node);
> }
> }
> {code}
> This would resolve the duplication and also simplify the codes some bit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12755) Fix typo in defaultFS warning message
[ https://issues.apache.org/jira/browse/HADOOP-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-12755: - Attachment: HADOOP-12755.001.patch Patch attached, pretty trivial. [~eddyxu] mind reviewing? > Fix typo in defaultFS warning message > - > > Key: HADOOP-12755 > URL: https://issues.apache.org/jira/browse/HADOOP-12755 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Attachments: HADOOP-12755.001.patch > > > As noted by [~fjk] on HADOOP-12043, the warning message misspells the config > key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12755) Fix typo in defaultFS warning message
[ https://issues.apache.org/jira/browse/HADOOP-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-12755: - Status: Patch Available (was: Open) > Fix typo in defaultFS warning message > - > > Key: HADOOP-12755 > URL: https://issues.apache.org/jira/browse/HADOOP-12755 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Attachments: HADOOP-12755.001.patch > > > As noted by [~fjk] on HADOOP-12043, the warning message misspells the config > key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12744) Refactoring of checksum failure report related codes
[
https://issues.apache.org/jira/browse/HADOOP-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12744:
---
Description:
This was from discussion with [~jingzhao] in HDFS-9646. There is some duplicate
codes between client and datanode sides:
{code}
private void addCorruptedBlock(ExtendedBlock blk, DatanodeInfo node,
Map corruptionMap) {
Set dnSet = corruptionMap.get(blk);
if (dnSet == null) {
dnSet = new HashSet<>();
corruptionMap.put(blk, dnSet);
}
if (!dnSet.contains(node)) {
dnSet.add(node);
}
}
{code}
This would resolve the duplication and also simplify the codes some bit.
was:This was from discussion with [~jingzhao] in HDFS-9646. Jing, wish you
don't mind I'm handling it here. I planned to refactor the mentioned codes
along with other codes related to the work in HDFS-9694, but I found the result
patch there is too large, so separate it out here.
> Refactoring of checksum failure report related codes
>
>
> Key: HADOOP-12744
> URL: https://issues.apache.org/jira/browse/HADOOP-12744
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12744-v1.patch
>
>
> This was from discussion with [~jingzhao] in HDFS-9646. There is some
> duplicate codes between client and datanode sides:
> {code}
> private void addCorruptedBlock(ExtendedBlock blk, DatanodeInfo node,
> Map corruptionMap) {
> Set dnSet = corruptionMap.get(blk);
> if (dnSet == null) {
> dnSet = new HashSet<>();
> corruptionMap.put(blk, dnSet);
> }
> if (!dnSet.contains(node)) {
> dnSet.add(node);
> }
> }
> {code}
> This would resolve the duplication and also simplify the codes some bit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127239#comment-15127239
]
Zhe Zhang commented on HADOOP-12041:
Thanks for the great work Kai. Patch LGTM overall. A few minor issues:
# {{makeValidIndexes}} should be static and maybe moved to {{CodecUtil}}
# The reported {{checkstyle}} issue looks valid.
# When you say the new coder is compatible with ISA-L coder, you mean I can use
new Java coder to decode data encoded with ISA-L, right?
# {{gftbls}} is hard to understand by name, does it mean {{gfTables}}?
# Any reason to allocate {{encodeMatrix}}, {{decodeMatrix}}, and
{{invertMatrix}} as 1D arrays but use them as matrices? Can we use 2D arrays?
# The below is not easy to understand. Why don't we need to prepare
{{decodeMatrix}} if the cached indexes haven't changed?
{code}
if (Arrays.equals(this.cachedErasedIndexes, erasedIndexes) &&
Arrays.equals(this.validIndexes, tmpValidIndexes)) {
return; // Optimization. Nothing to do
}
{code}
# {{RSUtil2#genReedSolomonMatrix}} is unused
# Patch is already very large, I think we should add {{package-info}}
separately.
# So about the incompatibility between HDFS-RAID coder and the new Java coder:
is it because they use different GF matrices?
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch,
> HADOOP-12041-v6.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HADOOP-12755) Fix typo in defaultFS warning message
[ https://issues.apache.org/jira/browse/HADOOP-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang moved HDFS-9736 to HADOOP-12755: Affects Version/s: (was: 2.8.0) 2.8.0 Target Version/s: 2.8.0 (was: 2.8.0) Key: HADOOP-12755 (was: HDFS-9736) Project: Hadoop Common (was: Hadoop HDFS) > Fix typo in defaultFS warning message > - > > Key: HADOOP-12755 > URL: https://issues.apache.org/jira/browse/HADOOP-12755 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > > As noted by [~fjk] on HADOOP-12043, the warning message misspells the config > key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12043) Display warning if defaultFs is not set when running fs commands.
[
https://issues.apache.org/jira/browse/HADOOP-12043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127277#comment-15127277
]
Andrew Wang commented on HADOOP-12043:
--
Thanks for the find [~fjk], I filed HADOOP-12755 to fix the typo.
> Display warning if defaultFs is not set when running fs commands.
> -
>
> Key: HADOOP-12043
> URL: https://issues.apache.org/jira/browse/HADOOP-12043
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.7.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 2.8.0
>
> Attachments: HDFS-8322.000.patch, HDFS-8322.001.patch,
> HDFS-8322.002.patch, HDFS-8322.003.patch, HDFS-8322.003.patch,
> HDFS-8322.004.patch, HDFS-8322.005.patch, HDFS-8322.006.patch
>
>
> Using {{LocalFileSystem}} is rarely the intention of running {{hadoop fs
> -ls}}.
> This JIRA proposes displaying a warning message if hadoop fs -ls is showing
> the local filesystem or using default fs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12755) Fix typo in defaultFS warning message
[ https://issues.apache.org/jira/browse/HADOOP-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127286#comment-15127286 ] Lei (Eddy) Xu commented on HADOOP-12755: +1. It is a trivial fix. Thanks [~andrew.wang]! > Fix typo in defaultFS warning message > - > > Key: HADOOP-12755 > URL: https://issues.apache.org/jira/browse/HADOOP-12755 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Trivial > Attachments: HADOOP-12755.001.patch > > > As noted by [~fjk] on HADOOP-12043, the warning message misspells the config > key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12747) support wildcard in libjars argument
[ https://issues.apache.org/jira/browse/HADOOP-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127227#comment-15127227 ] Gera Shegalov commented on HADOOP-12747: bq. It is true that today one can pass in a directory for -files and also for -libjars. In case of MR, the entire directory (including all files and directories recursively) does get copied over and localized to nodes. For libjars, however, as you observed, the classpath basically doesn't work if you meant it as a list of jars as it simply references the directory. On the other hand, if you meant it as a real directory root (consisting of class files), it still works correctly. My [solution|https://issues.apache.org/jira/browse/HADOOP-12747?focusedCommentId=15123058=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15123058] (tested many times with production pipelines I own) using dir uploads is obviously geared to the former case and not the case of exploded archive as you can see from the value of application classpath. was just suggesting to automate it via this property or mapreduce.job.classpath.files. bq. When it comes to classpaths (after which libjars is modeled), directory and directory/* are different as you're undoubtedly aware. directory/* is specifically interpreted as the list of jars in that directory by the JVM. IMO it would be good to maintain that definition for libjars. That would lead to a consistent expectation. I don't know whether libjars () is modeled after CLASSPATH. But I think there should be a separation of concerns: syntax vs how it's implemented. In the end, I am saying let us not bloat configuration regardless whether it's 'libs/*' or 'libs/'. bq. Also, I learned of this interesting nugget while looking at GenericOptionsParser: the value of libjars is added to the client classpath: This is the code we had discussed with [~ianoc]. This a very brittle code because you have no control when frameworks start using GOP. E.g., Scalding needs scala before it comes to GOP. Once you start asking people to put more stuff on HADOOP_CLASSPATH to bootstrap anyways why do this in libjars, as well? With [Pants|https://pantsbuild.github.io/build_dictionary.html] we don't need it at all becuase the jar manifest already includes all jars on the classpath see MAPREDUCE-6128. Maybe we should deprecate the client-side effect of libjars and not try to add more in this JIRA. Regarding YARN-1492, since it's a recent feature, it should accommodate for directory uploads anyways. It can negotiate at the directory level (e.g., recursive checksum of files/dirs). But that's not a subject for this JIRA. > support wildcard in libjars argument > > > Key: HADOOP-12747 > URL: https://issues.apache.org/jira/browse/HADOOP-12747 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Attachments: HADOOP-12747.01.patch, HADOOP-12747.02.patch > > > There is a problem when a user job adds too many dependency jars in their > command line. The HADOOP_CLASSPATH part can be addressed, including using > wildcards (\*). But the same cannot be done with the -libjars argument. Today > it takes only fully specified file paths. > We may want to consider supporting wildcards as a way to help users in this > situation. The idea is to handle it the same way the JVM does it: \* expands > to the list of jars in that directory. It does not traverse into any child > directory. > Also, it probably would be a good idea to do it only for libjars (i.e. don't > do it for -files and -archives). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127206#comment-15127206
]
Xiao Chen commented on HADOOP-12699:
The failure in {{TestACLs}} seems unrelated to this patch. I don't understand
the NPEs in there though...
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2,
> HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12744) Refactoring of checksum failure report related codes
[
https://issues.apache.org/jira/browse/HADOOP-12744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127178#comment-15127178
]
Kai Zheng commented on HADOOP-12744:
[~zhz], would you help convert this to a HDFS one and give it a review? Thanks!
> Refactoring of checksum failure report related codes
>
>
> Key: HADOOP-12744
> URL: https://issues.apache.org/jira/browse/HADOOP-12744
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12744-v1.patch, HADOOP-12744-v2.patch
>
>
> This was from discussion with [~jingzhao] in HDFS-9646. There is some
> duplicate codes between client and datanode sides:
> {code}
> private void addCorruptedBlock(ExtendedBlock blk, DatanodeInfo node,
> Map corruptionMap) {
> Set dnSet = corruptionMap.get(blk);
> if (dnSet == null) {
> dnSet = new HashSet<>();
> corruptionMap.put(blk, dnSet);
> }
> if (!dnSet.contains(node)) {
> dnSet.add(node);
> }
> }
> {code}
> This would resolve the duplication and also simplify the codes some bit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12041:
---
Attachment: HADOOP-12041-v7.patch
Updated the patch according to above discussion.
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch,
> HADOOP-12041-v6.patch, HADOOP-12041-v7.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12755) Fix typo in defaultFS warning message
[
https://issues.apache.org/jira/browse/HADOOP-12755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127424#comment-15127424
]
Hadoop QA commented on HADOOP-12755:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
37s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
30s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 43s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 39s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag |
| | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785641/HADOOP-12755.001.patch
|
| JIRA Issue | HADOOP-12755 |
| Optional Tests | asflicense
[jira] [Created] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
shimingfei created HADOOP-12756: --- Summary: Incorporate Aliyun OSS file system implementation Key: HADOOP-12756 URL: https://issues.apache.org/jira/browse/HADOOP-12756 Project: Hadoop Common Issue Type: New Feature Components: fs Reporter: shimingfei Assignee: shimingfei Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to access data laid on OSS storage from user’s Hadoop/Spark application, because of no original support for OSS in Hadoop. This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop applications can read/write data from OSS without any code change. Narrowing the gap between user’s APP and data storage, like what have been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12747) support wildcard in libjars argument
[
https://issues.apache.org/jira/browse/HADOOP-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127312#comment-15127312
]
Sangjin Lee commented on HADOOP-12747:
--
{quote}
My solution (tested many times with production pipelines I own) using dir
uploads is obviously geared to the former case and not the case of exploded
archive as you can see from the value of application classpath. was just
suggesting to automate it via this property or mapreduce.job.classpath.files.
{quote}
OK, could you elaborate a little on your proposal? Currently directories are
already allowed in libjars. It's that the classpath on the task side is added
as the directory name (i.e. {{directory}}) and not as a set of jars (i.e.
{{directory/\*}}). Therefore jars are not recognized as part of the classpath.
Does the proposal then entail adding the classpath as {{directory/*}} instead
of {{directory}}? Strictly speaking, it would be a re-interpretation (or a
change of behavior) for a directory entry for libjars. Today, if you add a
directory of classfiles to libjars, that works correctly. I can imagine a
scenario where someone submits a job out of a dev environment and includes a
source build directory as a libjars argument. Again, I don't know whether it is
an advertised feature/behavior, but at least I can see that working today, and
if we change it this way we would re-interpret that to mean something else.
Also, what should be uploaded and localized? Should everything be uploaded or
only jars be uploaded? Note that anything other than jars would not be used for
the classpath.
{quote}
I don't know whether libjars () is modeled after
CLASSPATH. But I think there should be a separation of concerns: syntax vs how
it's implemented.
{quote}
I agree completely. What I was arguing for is that it would be helpful for
users if we adopt a consistent *syntax* between classpath and libjars. This
patch is just one way of implementing that syntax. It could also be implemented
by accepting {{directory/\*}} as a directory, uploading only jar files to hdfs,
and adding {{directory/\*}} to the classpath. Again, in that case, IMO we still
need to leave {{directory}} as libjars argument alone as it works today.
> support wildcard in libjars argument
>
>
> Key: HADOOP-12747
> URL: https://issues.apache.org/jira/browse/HADOOP-12747
> Project: Hadoop Common
> Issue Type: New Feature
> Components: util
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
> Attachments: HADOOP-12747.01.patch, HADOOP-12747.02.patch
>
>
> There is a problem when a user job adds too many dependency jars in their
> command line. The HADOOP_CLASSPATH part can be addressed, including using
> wildcards (\*). But the same cannot be done with the -libjars argument. Today
> it takes only fully specified file paths.
> We may want to consider supporting wildcards as a way to help users in this
> situation. The idea is to handle it the same way the JVM does it: \* expands
> to the list of jars in that directory. It does not traverse into any child
> directory.
> Also, it probably would be a good idea to do it only for libjars (i.e. don't
> do it for -files and -archives).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127327#comment-15127327
]
Kai Zheng commented on HADOOP-12041:
Thanks Zhe for the careful review. The comments are very helpful.
bq. makeValidIndexes should be static and maybe moved to CodecUtil
How about adding {{CoderUtil}} for coder implementation? This is something I
wanted to do but get missed. As we're going to have various raw erasure coders,
we'll need a general utility class to embrace such. {{CodecUtil}} can focus on
caller interface side.
bq. The reported checkstyle issue looks valid.
Sure, pending the fixup for your comments. :)
bq. When you say the new coder is compatible with ISA-L coder, you mean I can
use new Java coder to decode data encoded with ISA-L, right?
Exactly. Probably that's why we're here. As you may have already noted, there
does be some thin HDFS client that's not equipped with native hadoop library,
so pure Java solution in good performance would be needed in case the ISA-L
coder isn't available.
bq. gftbls is hard to understand by name, does it mean gfTables?
Ah, yes. Will rename it.
bq. Any reason to allocate encodeMatrix, decodeMatrix, and invertMatrix as 1D
arrays but use them as matrices? Can we use 2D arrays?
I borrow the approach from the ISA-L library as the whole is. I thought it goes
like that for better performance, because the mentioned matrices are highly
frequently accessed during encoding/decoding computing thus better to be
compact together so able to remain in the cache.
bq. The below is not easy to understand. Why don't we need to prepare
decodeMatrix if the cached indexes haven't changed?
I thought better to add some comment for easier understanding. The decodeMatrix
is only relevant to the schema and erased indexes. The schema is bounded during
initialization phase and won't change; the erased indexes however can change
during different calls, but if not changed, the decodeMatrix won't need to be
recomputed. As indicated in HDFS side, we're very likely calling a decoder
repeatedly for a corrupt block group which is of the fixed erasure indexes, the
optimization trick here would make some sense.
bq. RSUtil2#genReedSolomonMatrix is unused
OK, let me remove it for now and add it later when support the mode of encode
matrix generation.
bq. Patch is already very large, I think we should add package-info separately.
Yeah, sure. I think we can add them when doing the coder rename, if we can bear
the checking style issue complaining the lack of them.
bq. So about the incompatibility between HDFS-RAID coder and the new Java
coder: is it because they use different GF matrices?
I think so. The implementation approach is also quite different. As you see, in
this new Java coder (same in the ISA-L library), encode/decode matrix is
previously calculated, the encoding and decoding are unified in the same way
that purely does the matrix operation against the input data bytes. The benefit
is, the decoder and encoder can be optimized together and are of the same high
performance.
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch,
> HADOOP-12041-v6.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127351#comment-15127351
]
Wei-Chiu Chuang commented on HADOOP-12699:
--
The NPE is unrelated to Hadoop at all. It's in Apache MINA.
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2,
> HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shimingfei updated HADOOP-12756: Attachment: OSS integration.pdf abstract for the work > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: shimingfei > Attachments: OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HADOOP-12041:
---
Attachment: HADOOP-12041-v8.patch
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch,
> HADOOP-12041-v6.patch, HADOOP-12041-v7.patch, HADOOP-12041-v8.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127569#comment-15127569
]
Kai Zheng commented on HADOOP-12041:
Patch updated addressing an unused import issue.
bq. Missing package-info.java file.
This will be addressed separately as discussed above.
> Implement another Reed-Solomon coder in pure Java
> -
>
> Key: HADOOP-12041
> URL: https://issues.apache.org/jira/browse/HADOOP-12041
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Kai Zheng
> Attachments: HADOOP-12041-v1.patch, HADOOP-12041-v2.patch,
> HADOOP-12041-v3.patch, HADOOP-12041-v4.patch, HADOOP-12041-v5.patch,
> HADOOP-12041-v6.patch, HADOOP-12041-v7.patch, HADOOP-12041-v8.patch
>
>
> Currently existing Java RS coders based on {{GaloisField}} implementation
> have some drawbacks or limitations:
> * The decoder computes not really erased units unnecessarily (HADOOP-11871);
> * The decoder requires parity units + data units order for the inputs in the
> decode API (HADOOP-12040);
> * Need to support or align with native erasure coders regarding concrete
> coding algorithms and matrix, so Java coders and native coders can be easily
> swapped in/out and transparent to HDFS (HADOOP-12010);
> * It's unnecessarily flexible but incurs some overhead, as HDFS erasure
> coding is totally a byte based data system, we don't need to consider other
> symbol size instead of 256.
> This desires to implement another RS coder in pure Java, in addition to the
> existing {{GaliosField}} from HDFS-RAID. The new Java RS coder will be
> favored and used by default to resolve the related issues. The old HDFS-RAID
> originated coder will still be there for comparing, and converting old data
> from HDFS-RAID systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127350#comment-15127350
]
Andrew Wang commented on HADOOP-12699:
--
Thanks for all the work here [~xiaochen], LGTM overall, just doc notes
basically:
* Need a {{}} tag to get a line break in Javadoc
* Using single line comment in TestKMS rather than {{/**}} means that the
{{@see}} doesn't work, so need to make it into actual Javadoc if you want this
to work.
I'll also note a little discrepancy in the unit test vs. actual usage, which is
that we're using a CryptoExtension in the test rather than a KMSClientProvider.
KMSClientProvider has another level of caching via ValueQueue, so it makes our
story to users even more complicated.
The KMS documentation is available here:
{{hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm}} and has a
cache section. We could update it to mention the server and client side
caching, and what kind of behavior can be expected. Basically, I as a user
might have the following questions which the docs should answer:
* What caches are present on the system? KMS and client. NN could be called out
specifically as a KMS client.
* What config keys control these caches?
* What is my window of staleness after I roll a key? I'd expect server cache
timeout + client cache timeout, but I don't know how eagerly Guava caches
expire items.
* Anything else you can think of
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.repro.2,
> HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127368#comment-15127368 ] Jing Zhao commented on HADOOP-8779: --- Based on the discussion in HDFS-9184, should we continue the work here as a long-term solution? To have token regardless of authentication type can also make multi-tenancy in HDFS easier to build: maybe we can utilize token to indicate the io priority of a job, or more generally the resource that a job can use within HDFS. > Use tokens regardless of authentication type > > > Key: HADOOP-8779 > URL: https://issues.apache.org/jira/browse/HADOOP-8779 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, security >Affects Versions: 2.0.2-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > Security is a combination of authentication and authorization (tokens). > Authorization may be granted independently of the authentication model. > Tokens should be used regardless of simple or kerberos authentication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127540#comment-15127540 ] Junping Du commented on HADOOP-11683: - Move non-critical issue out of 2.6.4 to 2.6.5. > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch, > HADOOP-11683.003.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11683) Need a plugin API to translate long principal names to local OS user names arbitrarily
[ https://issues.apache.org/jira/browse/HADOOP-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-11683: Target Version/s: 2.6.5 (was: 2.6.4) > Need a plugin API to translate long principal names to local OS user names > arbitrarily > -- > > Key: HADOOP-11683 > URL: https://issues.apache.org/jira/browse/HADOOP-11683 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.6.0 >Reporter: Sunny Cheung >Assignee: roger mak > Attachments: HADOOP-11683.001.patch, HADOOP-11683.002.patch, > HADOOP-11683.003.patch > > > We need a plugin API to translate long principal names (e.g. > john@example.com) to local OS user names (e.g. user123456) arbitrarily. > For some organizations the name translation is straightforward (e.g. > john@example.com to john_doe), and the hadoop.security.auth_to_local > configurable mapping is sufficient to resolve this (see HADOOP-6526). > However, in some other cases the name translation is arbitrary and cannot be > generalized by a set of translation rules easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8779) Use tokens regardless of authentication type
[ https://issues.apache.org/jira/browse/HADOOP-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127374#comment-15127374 ] Li Lu commented on HADOOP-8779: --- Agreed. IIUC tokens generally represent authorizations. We may want to further separate authentication (who am I) and authorization (what can I do) in Hadoop. Having tokens as a fundamental concept for all authentication types may help to support HDFS multi-tenancy and integration with YARN. > Use tokens regardless of authentication type > > > Key: HADOOP-8779 > URL: https://issues.apache.org/jira/browse/HADOOP-8779 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, security >Affects Versions: 2.0.2-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > Security is a combination of authentication and authorization (tokens). > Authorization may be granted independently of the authentication model. > Tokens should be used regardless of simple or kerberos authentication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-12666:
---
Attachment: HADOOP-12666-003.patch
> Support Windows Azure Data Lake - as a file system in Hadoop
>
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch,
> HADOOP-12666-1.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
> Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as
> input or output.
>
> ADL is ultra-high capacity, Optimized for massive throughput with rich
> management and security features. More details available at
> https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs
> specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to
> provide an implementation for accessing ADL storage; the scheme ADL is used
> for accessing it over HTTPS. We use the URI scheme:
> {code}adl:///path/to/file{code}
> to address individual Files/Folders. Tests are implemented mostly using a
> Contract implementation for the ADL functionality, with an option to test
> against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work
> can be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
> [~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
> [~ansaiprasanna] [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the
> current public preview release. Various different customer and test workloads
> have been run against clusters with such configurations for quite some time.
> The current version reflects to the version of the code tested and used in
> our production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-12666:
---
Status: In Progress (was: Patch Available)
> Support Windows Azure Data Lake - as a file system in Hadoop
>
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch,
> HADOOP-12666-1.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
> Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as
> input or output.
>
> ADL is ultra-high capacity, Optimized for massive throughput with rich
> management and security features. More details available at
> https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs
> specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to
> provide an implementation for accessing ADL storage; the scheme ADL is used
> for accessing it over HTTPS. We use the URI scheme:
> {code}adl:///path/to/file{code}
> to address individual Files/Folders. Tests are implemented mostly using a
> Contract implementation for the ADL functionality, with an option to test
> against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work
> can be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
> [~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
> [~ansaiprasanna] [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the
> current public preview release. Various different customer and test workloads
> have been run against clusters with such configurations for quite some time.
> The current version reflects to the version of the code tested and used in
> our production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work logged] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12666?focusedWorklogId=23421=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-23421
]
Vishwajeet Dusane logged work on HADOOP-12666:
--
Author: Vishwajeet Dusane
Created on: 01/Feb/16 10:48
Start Date: 12/Feb/16 10:47
Worklog Time Spent: 336h
Issue Time Tracking
---
Worklog Id: (was: 23421)
Time Spent: 336h
Remaining Estimate: 0h (was: 336h)
> Support Windows Azure Data Lake - as a file system in Hadoop
>
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch,
> HADOOP-12666-1.patch
>
> Original Estimate: 336h
> Time Spent: 336h
> Remaining Estimate: 0h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
> Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as
> input or output.
>
> ADL is ultra-high capacity, Optimized for massive throughput with rich
> management and security features. More details available at
> https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs
> specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to
> provide an implementation for accessing ADL storage; the scheme ADL is used
> for accessing it over HTTPS. We use the URI scheme:
> {code}adl:///path/to/file{code}
> to address individual Files/Folders. Tests are implemented mostly using a
> Contract implementation for the ADL functionality, with an option to test
> against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work
> can be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
> [~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
> [~ansaiprasanna] [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the
> current public preview release. Various different customer and test workloads
> have been run against clusters with such configurations for quite some time.
> The current version reflects to the version of the code tested and used in
> our production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126203#comment-15126203 ] Bolke de Bruin commented on HADOOP-12751: - [~steve_l] will keep your comments in mind and update the patch. As for our case (yes enterprise customer) we don't need '/' support in usernames so I can re-add it. I will run without it for a week and report back. For now we only found an issue with Hive that has a separate and different code path also sanitizing user names with '@'. > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 >Reporter: Bolke de Bruin >Priority: Critical > Labels: kerberos > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12563) Updated utility to create/modify token files
[ https://issues.apache.org/jira/browse/HADOOP-12563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126507#comment-15126507 ] Steve Loughran commented on HADOOP-12563: - # Looking at the DtFetcher interface -it needs to take a configuration. How else can it know what to work with, if there have been any runtime config options above the -site.xml values? # The change to using a protobuf persistent format is significant enough it should be called out into its own patch. That doesn't mean there's anything wrong with it (it makes a lot of sense), it's just that it's a significant enough change that more people should be commenting on it. > Updated utility to create/modify token files > > > Key: HADOOP-12563 > URL: https://issues.apache.org/jira/browse/HADOOP-12563 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Matthew Paduano > Attachments: HADOOP-12563.01.patch, HADOOP-12563.02.patch, > HADOOP-12563.03.patch, HADOOP-12563.04.patch, HADOOP-12563.05.patch, > HADOOP-12563.06.patch, example_dtutil_commands_and_output.txt, > generalized_token_case.pdf > > > hdfs fetchdt is missing some critical features and is geared almost > exclusively towards HDFS operations. Additionally, the token files that are > created use Java serializations which are hard/impossible to deal with in > other languages. It should be replaced with a better utility in common that > can read/write protobuf-based token files, has enough flexibility to be used > with other services, and offers key functionality such as append and rename. > The old version file format should still be supported for backward > compatibility, but will be effectively deprecated. > A follow-on JIRA will deprecrate fetchdt. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11875) [JDK8] Renaming _ as a one-character identifier to another identifier
[ https://issues.apache.org/jira/browse/HADOOP-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126543#comment-15126543 ] Allen Wittenauer commented on HADOOP-11875: --- bq. In HDFS, switching from JSP to JMX + HTML5 has shown various improvement on user experience, productivity and security. This isn't nearly so cut and dried. We have users who refuse to update because of the loss of features in the UI (they are sticking to 2.4.1's "legacy mode"), never mind the destruction of long term APIs and the "tossed over the fence" WebHDFS authentication that we had to rework to even make security function if one was using a custom plug-in. The switch took a lot of effort to even get to stable and it's still not what I would consider feature complete compared to the old API. So at least our experience for both beginner and advanced users, that update was somewhere in the middle and definitely not a solid win. > [JDK8] Renaming _ as a one-character identifier to another identifier > - > > Key: HADOOP-11875 > URL: https://issues.apache.org/jira/browse/HADOOP-11875 > Project: Hadoop Common > Issue Type: Bug >Reporter: Tsuyoshi Ozawa > Labels: newbie > Attachments: build_error_dump.txt > > > From JDK8, _ as a one-character identifier is disallowed. Currently Web UI > uses it. We should fix them to compile with JDK8. > https://bugs.openjdk.java.net/browse/JDK-8061549 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12752) Improve diagnostics/use of envvar/sysprop credential propagation
[
https://issues.apache.org/jira/browse/HADOOP-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126549#comment-15126549
]
Hadoop QA commented on HADOOP-12752:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 30s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 24s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Timed out junit tests |
org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126187#comment-15126187 ] Steve Loughran commented on HADOOP-12751: - ps, regarding the patch, don't catch exceptions and convert to fails. just throw it all the way up. And for the codepath that expects a failure, have its failure path (getShortName() returns something), include what gets returned. Think "If I were trying to debug this from nothing but a jenkins run, what information would I like —and what information is my test case currently losing?" see: https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 >Reporter: Bolke de Bruin >Priority: Critical > Labels: kerberos > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12751) While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple
[ https://issues.apache.org/jira/browse/HADOOP-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126184#comment-15126184 ] Steve Loughran commented on HADOOP-12751: - # pretty much all production enterprise customers run Hadoop in Kerberos mode. I know that as Kerberos related problems seem to often reach me. I don't want any more. # Kerberos and Hadoop integration is a pain point, with a combination of a security infrastructure brittle to host, network, DNS and clock config, meaningless error messages coming from the JVM libs, and our own UGI code not doing anything to help. # no, methods don't get renamed, on the basis of (a) "short name" is a concept in Hadoop (specifically, the bit before the /) and (b) things outside hadoop core will be using it. For that reason, I'd be tempted to leave the / check in, even if @ is addressed. > While using kerberos Hadoop incorrectly assumes names with '@' to be > non-simple > --- > > Key: HADOOP-12751 > URL: https://issues.apache.org/jira/browse/HADOOP-12751 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 >Reporter: Bolke de Bruin >Priority: Critical > Labels: kerberos > Attachments: 0001-HADOOP-12751-leave-user-validation-to-os.patch, > 0002-HADOOP-12751-leave-user-validation-to-os.patch, > 0003-HADOOP-12751-leave-user-validation-to-os.patch, > 0004-HADOOP-12751-leave-user-validation-to-os.patch > > > In the scenario of a trust between two directories, eg. FreeIPA (ipa.local) > and Active Directory (ad.local) users can be made available on the OS level > by something like sssd. The trusted users will be of the form 'user@ad.local' > while other users are will not contain the domain. Executing 'id -Gn > user@ad.local' will successfully return the groups the user belongs to if > configured correctly. > However, it is assumed by Hadoop that users of the format with '@' cannot be > correct. This code is in KerberosName.java and seems to be a validator if the > 'auth_to_local' rules are applied correctly. > In my opinion this should be removed or changed to a different kind of check > or maybe logged as a warning while still proceeding, as the current behavior > limits integration possibilities with other standard tools. > Workaround are difficult to apply (by having a rewrite by system tools to for > example user_ad_local) due to down stream consequences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
[
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126901#comment-15126901
]
Terry Siu commented on HADOOP-3733:
---
Just to clarify my comment above, I'm using Hive to create a table overlay on
an existing S3 folder and when I specify the location s3a::@/ where has a '/', I get:
FAILED: IllegalArgumentException The bucketName parameter must be specified
which I know the / in the is messing up Hive.
> "s3:" URLs break when Secret Key contains a slash, even if encoded
> --
>
> Key: HADOOP-3733
> URL: https://issues.apache.org/jira/browse/HADOOP-3733
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 0.17.1, 2.0.2-alpha
>Reporter: Stuart Sierra
>Priority: Minor
> Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733.patch,
> hadoop-3733.patch
>
>
> When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line,
> distcp fails if the SECRET contains a slash, even when the slash is
> URL-encoded as %2F.
> Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
> And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> And your bucket is called "mybucket"
> You can URL-encode the Secret KKey as
> Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> But this doesn't work:
> {noformat}
> $ bin/hadoop distcp file:///source
> s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
> 08/07/09 15:05:22 INFO util.CopyFiles:
> destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket:
> mybucket
> org.jets3t.service.S3ServiceException: S3 HEAD request failed.
> ResponseCode=403, ResponseMessage=Forbidden
> at
> org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
> ...
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.fs.s3.S3Exception:
> org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message:
> encoding="UTF-8"?>SignatureDoesNotMatchThe
> request signature we calculated does not match the signature you provided.
> Check your key and signing method.
> at
> org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12579) Deprecate and remove WriteableRPCEngine
[
https://issues.apache.org/jira/browse/HADOOP-12579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127834#comment-15127834
]
Kai Zheng commented on HADOOP-12579:
[~ste...@apache.org], do you agree RPC#waitForProxy can be retired as well as
commented above? Thanks!
> Deprecate and remove WriteableRPCEngine
> ---
>
> Key: HADOOP-12579
> URL: https://issues.apache.org/jira/browse/HADOOP-12579
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Haohui Mai
> Attachments: HADOOP-12579-v1.patch
>
>
> The {{WriteableRPCEninge}} depends on Java's serialization mechanisms for RPC
> requests. Without proper checks, it has be shown that it can lead to security
> vulnerabilities such as remote code execution (e.g., COLLECTIONS-580,
> HADOOP-12577).
> The current implementation has migrated from {{WriteableRPCEngine}} to
> {{ProtobufRPCEngine}} now. This jira proposes to deprecate
> {{WriteableRPCEngine}} in branch-2 and to remove it in trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12699) TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
[
https://issues.apache.org/jira/browse/HADOOP-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-12699:
---
Attachment: HADOOP-12699.09.patch
Thanks a lot [~andrew.wang] for the additional review, and educations on
javadoc!
Patch 9 fixes the javadocs, and adds the documents into the {{index.md.vm}}.
Please see if the English I put in there are understandable and reasonable to
you.
P.S. I've googled, but didn't find how to compile the docs. How do you usually
check the docs' format? I'm using a plugin in my IDE, but would love to double
check in a browser somehow. (I wanted to put the link to expireAfterAccess to
be more accurate, but since the link has brackets in it the md doesn't seem to
work with it
"http://docs.guava-libraries.googlecode.com/git-history/release/javadoc/com/google/common/cache/CacheBuilder.html#expireAfterAccess(long,
java.util.concurrent.TimeUnit)")
Also, looks to me the test uses {{KMSClientProvider}} too, and
{{CryptoExtension}} is an interface. I verified by debugging, I can step in
TestKMS -> LoadBalancingKMSClientProvider -> KMSClientProvider... So I modified
the test to expect rollover to be observable after both caches are drain.
(Still weird that disabling the async on EagerKGKPCE along would pass the
test maybe it's just low probability?) Please correct me if I misunderstood.
Thanks again!
> TestKMS#testKMSProvider intermittently fails during 'test rollover draining'
>
>
> Key: HADOOP-12699
> URL: https://issues.apache.org/jira/browse/HADOOP-12699
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HADOOP-12699.01.patch, HADOOP-12699.02.patch,
> HADOOP-12699.03.patch, HADOOP-12699.04.patch, HADOOP-12699.06.patch,
> HADOOP-12699.07.patch, HADOOP-12699.08.patch, HADOOP-12699.09.patch,
> HADOOP-12699.repro.2, HADOOP-12699.repro.patch
>
>
> I've seen several failures of testKMSProvider, all failed in the following
> snippet:
> {code}
> // test rollover draining
> KeyProviderCryptoExtension kpce = KeyProviderCryptoExtension.
> createKeyProviderCryptoExtension(kp);
> .
> EncryptedKeyVersion ekv1 = kpce.generateEncryptedKey("k6");
> kpce.rollNewVersion("k6");
> EncryptedKeyVersion ekv2 = kpce.generateEncryptedKey("k6");
> Assert.assertNotEquals(ekv1.getEncryptionKeyVersionName(),
> ekv2.getEncryptionKeyVersionName());
> {code}
> with error message
> {quote}Values should be different. Actual: k6@0{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12736) TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out
[
https://issues.apache.org/jira/browse/HADOOP-12736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HADOOP-12736:
Target Version/s: 2.6.4
> TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out
>
>
> Key: HADOOP-12736
> URL: https://issues.apache.org/jira/browse/HADOOP-12736
> Project: Hadoop Common
> Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Fix For: 2.8.0, 2.7.3, 2.6.4
>
> Attachments: HADOOP-12736.01.patch
>
>
> Saw this test failure today, the {{@Test(timeout=500)}} seems too aggressive
> to me.
> {noformat}
> testThreadDumpAndDeadlocks(org.apache.hadoop.test.TestTimedOutTestsListener)
> Time elapsed: 0.521 sec <<< ERROR!
> java.lang.Exception: test timed out after 500 milliseconds
> at
> jdk.internal.org.objectweb.asm.ByteVector.putShort(ByteVector.java:147)
> at
> jdk.internal.org.objectweb.asm.ClassWriter.toByteArray(ClassWriter.java:942)
> at
> java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCodeBytes(InvokerBytecodeGenerator.java:727)
> at
> java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(InvokerBytecodeGenerator.java:618)
> at java.lang.invoke.LambdaForm.compileToBytecode(LambdaForm.java:654)
> at java.lang.invoke.LambdaForm.prepare(LambdaForm.java:635)
> at java.lang.invoke.MethodHandle.(MethodHandle.java:461)
> at java.lang.invoke.BoundMethodHandle.(BoundMethodHandle.java:56)
> at
> java.lang.invoke.SimpleMethodHandle.(SimpleMethodHandle.java:37)
> at java.lang.invoke.SimpleMethodHandle.make(SimpleMethodHandle.java:41)
> at java.lang.invoke.LambdaForm.createIdentityForms(LambdaForm.java:1778)
> at java.lang.invoke.LambdaForm.(LambdaForm.java:1833)
> at
> java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(DirectMethodHandle.java:222)
> at
> java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:187)
> at
> java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:176)
> at java.lang.invoke.DirectMethodHandle.make(DirectMethodHandle.java:83)
> at
> java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:1656)
> at
> java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(MethodHandles.java:1613)
> at
> java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(MethodHandles.java:1798)
> at
> java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1747)
> at
> java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:477)
> at java.lang.UNIXProcess$Platform.get(UNIXProcess.java:155)
> at java.lang.UNIXProcess.(UNIXProcess.java:168)
> at java.lang.ProcessImpl.start(ProcessImpl.java:130)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:868)
> at org.apache.hadoop.util.Shell.run(Shell.java:838)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1117)
> at org.apache.hadoop.util.Shell.checkIsBashSupported(Shell.java:716)
> at org.apache.hadoop.util.Shell.(Shell.java:705)
> at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)
> at
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDump(TimedOutTestsListener.java:98)
> at
> org.apache.hadoop.test.TimedOutTestsListener.buildThreadDiagnosticString(TimedOutTestsListener.java:73)
> at
> org.apache.hadoop.test.TimedOutTestsListener.testFailure(TimedOutTestsListener.java:62)
> at
> org.apache.hadoop.test.TestTimedOutTestsListener.testThreadDumpAndDeadlocks(TestTimedOutTestsListener.java:163)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12706) TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally
[ https://issues.apache.org/jira/browse/HADOOP-12706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-12706: Target Version/s: 2.6.4 > TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out > occasionally > --- > > Key: HADOOP-12706 > URL: https://issues.apache.org/jira/browse/HADOOP-12706 > Project: Hadoop Common > Issue Type: Bug > Components: test >Reporter: Jason Lowe >Assignee: Sangjin Lee > Fix For: 2.7.3, 2.6.4 > > Attachments: HADOOP-12706.002.patch, HADOOP-12706.01.patch, > HADOOP-12706.03.patch > > > TestLocalFsFCStatistics has been failing sometimes, and when it fails it > appears to be from FCStatisticsBaseTest.testStatisticsThreadLocalDataCleanUp. > The test is timing out when it fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12107) long running apps may have a huge number of StatisticsData instances under FileSystem
[
https://issues.apache.org/jira/browse/HADOOP-12107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HADOOP-12107:
Target Version/s: 2.8.0, 2.6.4 (was: 2.8.0)
> long running apps may have a huge number of StatisticsData instances under
> FileSystem
> -
>
> Key: HADOOP-12107
> URL: https://issues.apache.org/jira/browse/HADOOP-12107
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Fix For: 2.8.0, 2.7.3, 2.6.4
>
> Attachments: HADOOP-12107.001.patch, HADOOP-12107.002.patch,
> HADOOP-12107.003.patch, HADOOP-12107.004.patch, HADOOP-12107.005.patch
>
>
> We observed with some of our apps (non-mapreduce apps that use filesystems)
> that they end up accumulating a huge memory footprint coming from
> {{FileSystem$Statistics$StatisticsData}} (in the {{allData}} list of
> {{Statistics}}).
> Although the thread reference from {{StatisticsData}} is a weak reference,
> and thus can get cleared once a thread goes away, the actual
> {{StatisticsData}} instances in the list won't get cleared until any of these
> following methods is called on {{Statistics}}:
> - {{getBytesRead()}}
> - {{getBytesWritten()}}
> - {{getReadOps()}}
> - {{getLargeReadOps()}}
> - {{getWriteOps()}}
> - {{toString()}}
> It is quite possible to have an application that interacts with a filesystem
> but does not call any of these methods on the {{Statistics}}. If such an
> application runs for a long time and has a large amount of thread churn, the
> memory footprint will grow significantly.
> The current workaround is either to limit the thread churn or to invoke these
> operations occasionally to pare down the memory. However, this is still a
> deficiency with {{FileSystem$Statistics}} itself in that the memory is
> controlled only as a side effect of those operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127645#comment-15127645 ] Cheng Hao commented on HADOOP-12756: +1 This is critical for AliYun users when integrated with MapReduce/Spark etc. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: shimingfei >Assignee: shimingfei > Attachments: OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
Rohith Sharma K S created HADOOP-12757:
--
Summary: Findbug compilation fails for 'Kafka Library support'
Key: HADOOP-12757
URL: https://issues.apache.org/jira/browse/HADOOP-12757
Project: Hadoop Common
Issue Type: Bug
Reporter: Rohith Sharma K S
Findbug compilation is failing for 'Kafka Library support'
{noformat}
[INFO] Apache Hadoop Amazon Web Services support .. SUCCESS [ 12.731 s]
[INFO] Apache Hadoop Azure support SUCCESS [ 14.972 s]
[INFO] Apache Hadoop Client ... SUCCESS [ 0.051 s]
[INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 0.045 s]
[INFO] Apache Hadoop Scheduler Load Simulator . SUCCESS [ 15.146 s]
[INFO] Apache Hadoop Tools Dist ... SUCCESS [ 0.045 s]
[INFO] Apache Hadoop Kafka Library support FAILURE [ 0.263 s]
[INFO] Apache Hadoop Tools SKIPPED
[INFO] Apache Hadoop Distribution . SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 21:40 min
[INFO] Finished at: 2016-02-02T10:16:57+05:30
[INFO] Final Memory: 159M/941M
[INFO]
[ERROR] Could not find resource
'/home/root1/workspace/hadoop-trunk/hadoop-tools/hadoop-kafka/dev-support/findbugs-exclude.xml'.
-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ResourceNotFoundException
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
[
https://issues.apache.org/jira/browse/HADOOP-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-12757:
---
Attachment: HADOOP-12757.01.patch
Attaching a patch to fix pom.xml.
> Findbug compilation fails for 'Kafka Library support'
> -
>
> Key: HADOOP-12757
> URL: https://issues.apache.org/jira/browse/HADOOP-12757
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: HADOOP-12757.01.patch
>
>
> Findbug compilation is failing for 'Kafka Library support'
> {noformat}
> [INFO] Apache Hadoop Amazon Web Services support .. SUCCESS [ 12.731
> s]
> [INFO] Apache Hadoop Azure support SUCCESS [ 14.972
> s]
> [INFO] Apache Hadoop Client ... SUCCESS [ 0.051
> s]
> [INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Scheduler Load Simulator . SUCCESS [ 15.146
> s]
> [INFO] Apache Hadoop Tools Dist ... SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Kafka Library support FAILURE [ 0.263
> s]
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 21:40 min
> [INFO] Finished at: 2016-02-02T10:16:57+05:30
> [INFO] Final Memory: 159M/941M
> [INFO]
>
> [ERROR] Could not find resource
> '/home/root1/workspace/hadoop-trunk/hadoop-tools/hadoop-kafka/dev-support/findbugs-exclude.xml'.
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ResourceNotFoundException
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
[
https://issues.apache.org/jira/browse/HADOOP-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-12757:
---
Status: Patch Available (was: Open)
> Findbug compilation fails for 'Kafka Library support'
> -
>
> Key: HADOOP-12757
> URL: https://issues.apache.org/jira/browse/HADOOP-12757
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: HADOOP-12757.01.patch
>
>
> Findbug compilation is failing for 'Kafka Library support'
> {noformat}
> [INFO] Apache Hadoop Amazon Web Services support .. SUCCESS [ 12.731
> s]
> [INFO] Apache Hadoop Azure support SUCCESS [ 14.972
> s]
> [INFO] Apache Hadoop Client ... SUCCESS [ 0.051
> s]
> [INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Scheduler Load Simulator . SUCCESS [ 15.146
> s]
> [INFO] Apache Hadoop Tools Dist ... SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Kafka Library support FAILURE [ 0.263
> s]
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 21:40 min
> [INFO] Finished at: 2016-02-02T10:16:57+05:30
> [INFO] Final Memory: 159M/941M
> [INFO]
>
> [ERROR] Could not find resource
> '/home/root1/workspace/hadoop-trunk/hadoop-tools/hadoop-kafka/dev-support/findbugs-exclude.xml'.
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ResourceNotFoundException
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127548#comment-15127548
]
Hadoop QA commented on HADOOP-12041:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 59s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 34s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 40s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 40s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-common-project/hadoop-common: patch generated 2
new + 8 unchanged - 1 fixed = 10 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 17s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 15s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785663/HADOOP-12041-v7.patch
|
| JIRA Issue | HADOOP-12041 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs
[jira] [Commented] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127582#comment-15127582
]
Vishwajeet Dusane commented on HADOOP-12666:
Not seeing in the report any Javadoc warning related to this patch.
> Support Windows Azure Data Lake - as a file system in Hadoop
>
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch,
> HADOOP-12666-1.patch
>
> Original Estimate: 336h
> Time Spent: 336h
> Remaining Estimate: 0h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
> Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as
> input or output.
>
> ADL is ultra-high capacity, Optimized for massive throughput with rich
> management and security features. More details available at
> https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs
> specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to
> provide an implementation for accessing ADL storage; the scheme ADL is used
> for accessing it over HTTPS. We use the URI scheme:
> {code}adl:///path/to/file{code}
> to address individual Files/Folders. Tests are implemented mostly using a
> Contract implementation for the ADL functionality, with an option to test
> against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work
> can be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
> [~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
> [~ansaiprasanna] [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the
> current public preview release. Various different customer and test workloads
> have been run against clusters with such configurations for quite some time.
> The current version reflects to the version of the code tested and used in
> our production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12041) Implement another Reed-Solomon coder in pure Java
[
https://issues.apache.org/jira/browse/HADOOP-12041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127674#comment-15127674
]
Hadoop QA commented on HADOOP-12041:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
49s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
38s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 0s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
39s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 30s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 50s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 111m 35s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.security.ssl.TestReloadingX509TrustManager |
| | hadoop.fs.shell.find.TestAnd |
| | hadoop.io.compress.TestCodecPool |
| | hadoop.fs.shell.find.TestPrint |
| | hadoop.fs.shell.find.TestIname |
| | hadoop.fs.shell.find.TestName |
| | hadoop.ipc.TestIPC |
| JDK v1.7.0_91 Failed junit tests |
[jira] [Assigned] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
[
https://issues.apache.org/jira/browse/HADOOP-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA reassigned HADOOP-12757:
--
Assignee: Akira AJISAKA
> Findbug compilation fails for 'Kafka Library support'
> -
>
> Key: HADOOP-12757
> URL: https://issues.apache.org/jira/browse/HADOOP-12757
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
>
> Findbug compilation is failing for 'Kafka Library support'
> {noformat}
> [INFO] Apache Hadoop Amazon Web Services support .. SUCCESS [ 12.731
> s]
> [INFO] Apache Hadoop Azure support SUCCESS [ 14.972
> s]
> [INFO] Apache Hadoop Client ... SUCCESS [ 0.051
> s]
> [INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Scheduler Load Simulator . SUCCESS [ 15.146
> s]
> [INFO] Apache Hadoop Tools Dist ... SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Kafka Library support FAILURE [ 0.263
> s]
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 21:40 min
> [INFO] Finished at: 2016-02-02T10:16:57+05:30
> [INFO] Final Memory: 159M/941M
> [INFO]
>
> [ERROR] Could not find resource
> '/home/root1/workspace/hadoop-trunk/hadoop-tools/hadoop-kafka/dev-support/findbugs-exclude.xml'.
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ResourceNotFoundException
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
[
https://issues.apache.org/jira/browse/HADOOP-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HADOOP-12757:
---
Target Version/s: 3.0.0
> Findbug compilation fails for 'Kafka Library support'
> -
>
> Key: HADOOP-12757
> URL: https://issues.apache.org/jira/browse/HADOOP-12757
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Akira AJISAKA
> Attachments: HADOOP-12757.01.patch
>
>
> Findbug compilation is failing for 'Kafka Library support'
> {noformat}
> [INFO] Apache Hadoop Amazon Web Services support .. SUCCESS [ 12.731
> s]
> [INFO] Apache Hadoop Azure support SUCCESS [ 14.972
> s]
> [INFO] Apache Hadoop Client ... SUCCESS [ 0.051
> s]
> [INFO] Apache Hadoop Mini-Cluster . SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Scheduler Load Simulator . SUCCESS [ 15.146
> s]
> [INFO] Apache Hadoop Tools Dist ... SUCCESS [ 0.045
> s]
> [INFO] Apache Hadoop Kafka Library support FAILURE [ 0.263
> s]
> [INFO] Apache Hadoop Tools SKIPPED
> [INFO] Apache Hadoop Distribution . SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 21:40 min
> [INFO] Finished at: 2016-02-02T10:16:57+05:30
> [INFO] Final Memory: 159M/941M
> [INFO]
>
> [ERROR] Could not find resource
> '/home/root1/workspace/hadoop-trunk/hadoop-tools/hadoop-kafka/dev-support/findbugs-exclude.xml'.
> -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ResourceNotFoundException
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12757) Findbug compilation fails for 'Kafka Library support'
[
https://issues.apache.org/jira/browse/HADOOP-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127753#comment-15127753
]
Hadoop QA commented on HADOOP-12757:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 0s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hadoop-kafka in the patch passed with JDK v1.8.0_66.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hadoop-kafka in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785709/HADOOP-12757.01.patch
|
| JIRA Issue | HADOOP-12757 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux f9c8bf3a1a24 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 1cd55e0 |
| Default Java | 1.7.0_91 |
| Multi-JDK versions |
[jira] [Updated] (HADOOP-12666) Support Windows Azure Data Lake - as a file system in Hadoop
[
https://issues.apache.org/jira/browse/HADOOP-12666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishwajeet Dusane updated HADOOP-12666:
---
Status: Patch Available (was: In Progress)
> Support Windows Azure Data Lake - as a file system in Hadoop
>
>
> Key: HADOOP-12666
> URL: https://issues.apache.org/jira/browse/HADOOP-12666
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, tools
>Reporter: Vishwajeet Dusane
>Assignee: Vishwajeet Dusane
> Attachments: HADOOP-12666-002.patch, HADOOP-12666-003.patch,
> HADOOP-12666-1.patch
>
> Original Estimate: 336h
> Time Spent: 336h
> Remaining Estimate: 0h
>
> h2. Description
> This JIRA describes a new file system implementation for accessing Windows
> Azure Data Lake Store (ADL) from within Hadoop. This would enable existing
> Hadoop applications such has MR, HIVE, Hbase etc.., to use ADL store as
> input or output.
>
> ADL is ultra-high capacity, Optimized for massive throughput with rich
> management and security features. More details available at
> https://azure.microsoft.com/en-us/services/data-lake-store/
> h2. High level design
> ADL file system exposes RESTful interfaces compatible with WebHdfs
> specification 2.7.1.
> At a high level, the code here extends the SWebHdfsFileSystem class to
> provide an implementation for accessing ADL storage; the scheme ADL is used
> for accessing it over HTTPS. We use the URI scheme:
> {code}adl:///path/to/file{code}
> to address individual Files/Folders. Tests are implemented mostly using a
> Contract implementation for the ADL functionality, with an option to test
> against a real ADL storage if configured.
> h2. Credits and history
> This has been ongoing work for a while, and the early version of this work
> can be seen in. Credit for this work goes to the team: [~vishwajeet.dusane],
> [~snayak], [~srevanka], [~kiranch], [~chakrab], [~omkarksa], [~snvijaya],
> [~ansaiprasanna] [~jsangwan]
> h2. Test
> Besides Contract tests, we have used ADL as the additional file system in the
> current public preview release. Various different customer and test workloads
> have been run against clusters with such configurations for quite some time.
> The current version reflects to the version of the code tested and used in
> our production environment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12752) Improve diagnostics/use of envvar/sysprop credential propagation
[
https://issues.apache.org/jira/browse/HADOOP-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12752:
Attachment: HADOOP-12752-001.patch
Patch-001
# log-> slf4j,
# log @ debug details about HADOOP_TOKEN_FILE_LOCATION load,
# explicitly raise FNFE if it's missing, mentioning the env var as the source
of trouble
> Improve diagnostics/use of envvar/sysprop credential propagation
>
>
> Key: HADOOP-12752
> URL: https://issues.apache.org/jira/browse/HADOOP-12752
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
> Attachments: HADOOP-12752-001.patch
>
>
> * document the system property {{hadoop.token.files}}.
> * document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
> * When UGI inits tokens off that or the env var , log this fact
> * when trying to load a file referenced in the env var (a) trim it and (b)
> check for it existing, failing with a message referring to the ENV var as
> well as the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12752) Improve diagnostics/use of envvar/sysprop credential propagation
[
https://issues.apache.org/jira/browse/HADOOP-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12752:
Assignee: Steve Loughran
Priority: Minor (was: Major)
Description:
* document the system property {{hadoop.token.files}}.
* document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
* When UGI inits tokens off that or the env var , log this fact
* when trying to load a file referenced in the env var (a) trim it and (b)
check for it existing, failing with a message referring to the ENV var as well
as the file.
was:
* document the system property {{hadoop.token.files}}.
* document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
* When UGI inits tokens off that or the env var , log this fact
* when trying to load a file referenced in the env var (a) trim it and (b)
check for it existing, failing with a message referring to the ENV var as well
as the file.
> Improve diagnostics/use of envvar/sysprop credential propagation
>
>
> Key: HADOOP-12752
> URL: https://issues.apache.org/jira/browse/HADOOP-12752
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-12752-001.patch
>
>
> * document the system property {{hadoop.token.files}}.
> * document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
> * When UGI inits tokens off that or the env var , log this fact
> * when trying to load a file referenced in the env var (a) trim it and (b)
> check for it existing, failing with a message referring to the ENV var as
> well as the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12752) Improve diagnostics/use of envvar/sysprop credential propagation
[
https://issues.apache.org/jira/browse/HADOOP-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12752:
Status: Patch Available (was: Open)
> Improve diagnostics/use of envvar/sysprop credential propagation
>
>
> Key: HADOOP-12752
> URL: https://issues.apache.org/jira/browse/HADOOP-12752
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-12752-001.patch
>
>
> * document the system property {{hadoop.token.files}}.
> * document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
> * When UGI inits tokens off that or the env var , log this fact
> * when trying to load a file referenced in the env var (a) trim it and (b)
> check for it existing, failing with a message referring to the ENV var as
> well as the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12752) Improve diagnostics/use of envvar/sysprop credential propagation
[
https://issues.apache.org/jira/browse/HADOOP-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126393#comment-15126393
]
Steve Loughran commented on HADOOP-12752:
-
YARN-4653 documents YARN's use of the env var and has a checklist for app
developers to follow.
> Improve diagnostics/use of envvar/sysprop credential propagation
>
>
> Key: HADOOP-12752
> URL: https://issues.apache.org/jira/browse/HADOOP-12752
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: security
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>
> * document the system property {{hadoop.token.files}}.
> * document the env var {{HADOOP_TOKEN_FILE_LOCATION}}.
> * When UGI inits tokens off that or the env var , log this fact
> * when trying to load a file referenced in the env var (a) trim it and (b)
> check for it existing, failing with a message referring to the ENV var as
> well as the file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11875) [JDK8] Renaming _ as a one-character identifier to another identifier
[ https://issues.apache.org/jira/browse/HADOOP-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126413#comment-15126413 ] Steve Loughran commented on HADOOP-11875: - # Hamlet is easy to use from Java code. What do you want, us to switch to JSP? # It's used in downstream YARN apps, which is not just done out of laziness, but to handle the proxy renaming stuff, plus escaping of text. Because of #2, Hamlet is going to have to stay around, even if something to replace it is added alongside (or just pulled in from elsewhere) > [JDK8] Renaming _ as a one-character identifier to another identifier > - > > Key: HADOOP-11875 > URL: https://issues.apache.org/jira/browse/HADOOP-11875 > Project: Hadoop Common > Issue Type: Bug >Reporter: Tsuyoshi Ozawa > Labels: newbie > Attachments: build_error_dump.txt > > > From JDK8, _ as a one-character identifier is disallowed. Currently Web UI > uses it. We should fix them to compile with JDK8. > https://bugs.openjdk.java.net/browse/JDK-8061549 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9621) Document/analyze current Hadoop security model
[ https://issues.apache.org/jira/browse/HADOOP-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126415#comment-15126415 ] Steve Loughran commented on HADOOP-9621: # the doc is still there, you just need to request read access # someone could export it from google docs to .md and then for site; illustrations would have to go in as png files. Be nice if the plantuml (presumably) spec files were in source too, for maintenance > Document/analyze current Hadoop security model > -- > > Key: HADOOP-9621 > URL: https://issues.apache.org/jira/browse/HADOOP-9621 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Brian Swan >Priority: Minor > Labels: documentation > Attachments: HadoopSecurityAnalysis-20130612.pdf, > HadoopSecurityAnalysis-20130614.pdf, HadoopSecurityAnalysis-20130624.pdf, > ThreatsforToken-basedAuthN-20130619.pdf > > Original Estimate: 336h > Remaining Estimate: 336h > > In light of the proposed changes to Hadoop security in Hadoop-9533 and > Hadoop-9392, having a common, detailed understanding (in the form of a > document) of the benefits/drawbacks of the current security model and how it > works would be useful. The document should address all security principals, > their authentication mechanisms, and handling of shared secrets through the > lens of the following principles: Minimize attack surface area, Establish > secure defaults, Principle of Least privilege, Principle of Defense in depth, > Fail securely, Don’t trust services, Separation of duties, Avoid security by > obscurity, Keep security simple, Fix security issues correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12664) UGI auto-renewer does not verify kinit availability during initialization
[
https://issues.apache.org/jira/browse/HADOOP-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126420#comment-15126420
]
Steve Loughran commented on HADOOP-12664:
-
UGI @ error does log, BTW
> UGI auto-renewer does not verify kinit availability during initialization
> -
>
> Key: HADOOP-12664
> URL: https://issues.apache.org/jira/browse/HADOOP-12664
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Ray Chiang
>Priority: Minor
> Labels: supportability
> Attachments: HADOOP-12664.001.patch
>
>
> UGI auto-renewer does not verify that {{hadoop.kerberos.kinit.command}} is in
> the path during initialization. If not available, the auto-renewal thread
> will hit an error during TGT renewal. We recently saw a case where it
> manifests as transient errors during client program execution which can be
> hard to track down without UGI logging.
> It seems like {{kinit}} availability should be verified during initialization
> to make the behavior more predictable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9621) Document/analyze current Hadoop security model
[ https://issues.apache.org/jira/browse/HADOOP-9621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126426#comment-15126426 ] Larry McCay commented on HADOOP-9621: - I can look into top converting it. Couple questions first: 1. would it be better in wiki (I believe there is a plantuml plugin too)? 2. if so, could we still put a link in the navigation bar of the site that points to the wiki? 3. if not, how would we like to link it from the Hadoop site navigation bar - I assume it would be a top level and standalone set of docs and not part of any other part of the security docs. > Document/analyze current Hadoop security model > -- > > Key: HADOOP-9621 > URL: https://issues.apache.org/jira/browse/HADOOP-9621 > Project: Hadoop Common > Issue Type: Task > Components: security >Reporter: Brian Swan >Priority: Minor > Labels: documentation > Attachments: HadoopSecurityAnalysis-20130612.pdf, > HadoopSecurityAnalysis-20130614.pdf, HadoopSecurityAnalysis-20130624.pdf, > ThreatsforToken-basedAuthN-20130619.pdf > > Original Estimate: 336h > Remaining Estimate: 336h > > In light of the proposed changes to Hadoop security in Hadoop-9533 and > Hadoop-9392, having a common, detailed understanding (in the form of a > document) of the benefits/drawbacks of the current security model and how it > works would be useful. The document should address all security principals, > their authentication mechanisms, and handling of shared secrets through the > lens of the following principles: Minimize attack surface area, Establish > secure defaults, Principle of Least privilege, Principle of Defense in depth, > Fail securely, Don’t trust services, Separation of duties, Avoid security by > obscurity, Keep security simple, Fix security issues correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11875) [JDK8] Renaming _ as a one-character identifier to another identifier
[ https://issues.apache.org/jira/browse/HADOOP-11875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126433#comment-15126433 ] Haohui Mai commented on HADOOP-11875: - bq. Hamlet is easy to use from Java code. What do you want, us to switch to JSP? In HDFS, switching from JSP to JMX + HTML5 has shown various improvement on user experience, productivity and security. bq. It's used in downstream YARN apps, which is not just done out of laziness, but to handle the proxy renaming stuff, plus escaping of text. Can you be more concrete on what needs to be fixed? > [JDK8] Renaming _ as a one-character identifier to another identifier > - > > Key: HADOOP-11875 > URL: https://issues.apache.org/jira/browse/HADOOP-11875 > Project: Hadoop Common > Issue Type: Bug >Reporter: Tsuyoshi Ozawa > Labels: newbie > Attachments: build_error_dump.txt > > > From JDK8, _ as a one-character identifier is disallowed. Currently Web UI > uses it. We should fix them to compile with JDK8. > https://bugs.openjdk.java.net/browse/JDK-8061549 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12664) UGI auto-renewer does not verify kinit availability during initialization
[
https://issues.apache.org/jira/browse/HADOOP-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126422#comment-15126422
]
Steve Loughran commented on HADOOP-12664:
-
Oh, and HADOOP-12752 includes a switch to the slf4j API, for easier logging and
efficient enough you can strip out .isInfoEnabled checks
> UGI auto-renewer does not verify kinit availability during initialization
> -
>
> Key: HADOOP-12664
> URL: https://issues.apache.org/jira/browse/HADOOP-12664
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Ray Chiang
>Priority: Minor
> Labels: supportability
> Attachments: HADOOP-12664.001.patch
>
>
> UGI auto-renewer does not verify that {{hadoop.kerberos.kinit.command}} is in
> the path during initialization. If not available, the auto-renewal thread
> will hit an error during TGT renewal. We recently saw a case where it
> manifests as transient errors during client program execution which can be
> hard to track down without UGI logging.
> It seems like {{kinit}} availability should be verified during initialization
> to make the behavior more predictable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-12668) Modify HDFS embeded jetty server logic in HttpServer2.java to exclude weak Ciphers through ssl-server.conf
[
https://issues.apache.org/jira/browse/HADOOP-12668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126625#comment-15126625
]
Zhe Zhang commented on HADOOP-12668:
I added a break point and printed the values in {{availableCiphers}}. I think
the main point of the unit test is to verify that once we add one of the below
to {{excludeCiphers}}, they won't appear in {{availableCiphers}} anymore. Then
we know the added config knob is taking effect.
{code}
0 = "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA"
1 = "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA"
2 = "TLS_RSA_WITH_AES_128_CBC_SHA"
3 = "TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA"
4 = "TLS_ECDH_RSA_WITH_AES_128_CBC_SHA"
5 = "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
6 = "TLS_DHE_DSS_WITH_AES_128_CBC_SHA"
7 = "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA"
8 = "TLS_ECDHE_RSA_WITH_RC4_128_SHA"
9 = "SSL_RSA_WITH_RC4_128_SHA"
10 = "TLS_ECDH_ECDSA_WITH_RC4_128_SHA"
11 = "TLS_ECDH_RSA_WITH_RC4_128_SHA"
12 = "TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA"
13 = "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA"
14 = "SSL_RSA_WITH_3DES_EDE_CBC_SHA"
15 = "TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA"
16 = "TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA"
17 = "SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA"
18 = "SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"
19 = "SSL_RSA_WITH_RC4_128_MD5"
20 = "TLS_EMPTY_RENEGOTIATION_INFO_SCSV"
{code}
> Modify HDFS embeded jetty server logic in HttpServer2.java to exclude weak
> Ciphers through ssl-server.conf
> --
>
> Key: HADOOP-12668
> URL: https://issues.apache.org/jira/browse/HADOOP-12668
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Affects Versions: 2.7.1
>Reporter: Vijay Singh
>Assignee: Vijay Singh
>Priority: Critical
> Labels: common, ha, hadoop, hdfs, security
> Attachments: Hadoop-12668.006.patch, Hadoop-12668.007.patch, test.log
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently Embeded jetty Server used across all hadoop services is configured
> through ssl-server.xml file from their respective configuration section.
> However, the SSL/TLS protocol being used for this jetty servers can be
> downgraded to weak cipher suites. This code changes aims to add following
> functionality:
> 1) Add logic in hadoop common (HttpServer2.java and associated interfaces) to
> spawn jetty servers with ability to exclude weak cipher suites. I propose we
> make this though ssl-server.xml and hence each service can choose to disable
> specific ciphers.
> 2) Modify DFSUtil.java used by HDFS code to supply new parameter
> ssl.server.exclude.cipher.list for hadoop-common code, so it can exclude the
> ciphers supplied through this key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
