[jira] [Commented] (HADOOP-14976) Allow overriding HADOOP_SHELL_EXECNAME
[
https://issues.apache.org/jira/browse/HADOOP-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277244#comment-16277244
]
Chen Liang commented on HADOOP-14976:
-
Thanks [~arpitagarwal] for working on this and [~aw] for the discussion! +1 on
v004 patch.
> Allow overriding HADOOP_SHELL_EXECNAME
> --
>
> Key: HADOOP-14976
> URL: https://issues.apache.org/jira/browse/HADOOP-14976
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HADOOP-14976.01.patch, HADOOP-14976.02.patch,
> HADOOP-14976.03.patch, HADOOP-14976.04.patch
>
>
> Some Hadoop shell scripts infer their own name using this bit of shell magic:
> {code}
> 18 MYNAME="${BASH_SOURCE-$0}"
> 19 HADOOP_SHELL_EXECNAME="${MYNAME##*/}"
> {code}
> e.g. see the
> [hdfs|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs#L18]
> script.
> The inferred shell script name is later passed to _hadoop-functions.sh_ which
> uses it to construct the names of some environment variables. E.g. when
> invoking _hdfs datanode_, the options variable name is inferred as follows:
> {code}
> # HDFS + DATANODE + OPTS -> HDFS_DATANODE_OPTS
> {code}
> This works well if the calling script name is standard {{hdfs}} or {{yarn}}.
> If a distribution renames the script to something like foo.bar, , then the
> variable names will be inferred as {{FOO.BAR_DATANODE_OPTS}}. This is not a
> valid bash variable name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15086) NativeAzureFileSystem.rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277256#comment-16277256 ] Shixiong Zhu commented on HADOOP-15086: --- Looks like we can use conditional-headers to implement an atomic file rename on Azure blob storage https://docs.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations ? I think it's not necessary to introduce object-store specific committers when a storage already supports atomic operations. > NativeAzureFileSystem.rename is not atomic > -- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15086) NativeAzureFileSystem.rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277260#comment-16277260 ] Shixiong Zhu commented on HADOOP-15086: --- In addition, I probably was not clear. I created this ticket is just for atomic file rename. > NativeAzureFileSystem.rename is not atomic > -- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277408#comment-16277408 ] Xiao Chen edited comment on HADOOP-15012 at 12/4/17 8:10 PM: - FYI - I plan to backport this to branch-2 and branch-3.0 (after taking care of HADOOP-15056) if no objections. was (Author: xiaochen): FYI - I plan to backport this to branch-2 and branch-3.0 if no objections. > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.1.0 > > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15012) Add readahead, dropbehind, and unbuffer to StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277408#comment-16277408 ] Xiao Chen commented on HADOOP-15012: FYI - I plan to backport this to branch-2 and branch-3.0 if no objections. > Add readahead, dropbehind, and unbuffer to StreamCapabilities > - > > Key: HADOOP-15012 > URL: https://issues.apache.org/jira/browse/HADOOP-15012 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 2.9.0 >Reporter: John Zhuge >Assignee: John Zhuge > Fix For: 3.1.0 > > > A split from HADOOP-14872 to track changes that enhance StreamCapabilities > class with READAHEAD, DROPBEHIND, and UNBUFFER capability. > Discussions and code reviews are done in HADOOP-14872. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277231#comment-16277231
]
Daryn Sharp commented on HADOOP-10768:
--
Is hmac still needed if using aes? Will try to give initial comments this
afternoon.
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> Optimize Hadoop RPC encryption performance.pdf
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277537#comment-16277537 ] Andrew Wang commented on HADOOP-15058: -- [~mackrorysd], maybe you could review? This script is self-contained, so I think it's not too bad to read. > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14872) CryptoInputStream should implement unbuffer
[
https://issues.apache.org/jira/browse/HADOOP-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277409#comment-16277409
]
Xiao Chen commented on HADOOP-14872:
FYI - I plan to backport this to branch-2 and branch-3.0 if no objections.
> CryptoInputStream should implement unbuffer
> ---
>
> Key: HADOOP-14872
> URL: https://issues.apache.org/jira/browse/HADOOP-14872
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, security
>Affects Versions: 2.6.4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Fix For: 3.1.0
>
> Attachments: HADOOP-14872.001.patch, HADOOP-14872.002.patch,
> HADOOP-14872.003.patch, HADOOP-14872.004.patch, HADOOP-14872.005.patch,
> HADOOP-14872.006.patch, HADOOP-14872.007.patch, HADOOP-14872.008.patch,
> HADOOP-14872.009.patch, HADOOP-14872.010.patch, HADOOP-14872.011.patch,
> HADOOP-14872.012.patch, HADOOP-14872.013.patch
>
>
> Discovered in IMPALA-5909.
> Opening an encrypted HDFS file returns a chain of wrapped input streams:
> {noformat}
> HdfsDataInputStream
> CryptoInputStream
> DFSInputStream
> {noformat}
> If an application such as Impala or HBase calls HdfsDataInputStream#unbuffer,
> FSDataInputStream#unbuffer will be called:
> {code:java}
> try {
> ((CanUnbuffer)in).unbuffer();
> } catch (ClassCastException e) {
> throw new UnsupportedOperationException("this stream does not " +
> "support unbuffering.");
> }
> {code}
> If the {{in}} class does not implement CanUnbuffer, UOE will be thrown. If
> the application is not careful, tons of UOEs will show up in logs.
> In comparison, opening an non-encrypted HDFS file returns this chain:
> {noformat}
> HdfsDataInputStream
> DFSInputStream
> {noformat}
> DFSInputStream implements CanUnbuffer.
> It is good for CryptoInputStream to implement CanUnbuffer for 2 reasons:
> * Release buffer, cache, or any other resource when instructed
> * Able to call its wrapped DFSInputStream unbuffer
> * Avoid the UOE described above. Applications may not handle the UOE very
> well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277616#comment-16277616 ] Sean Mackrory commented on HADOOP-15058: Looks good to me! +1 > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277713#comment-16277713 ] Hudson commented on HADOOP-15058: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13321 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13321/]) HADOOP-15058. create-release site build outputs dummy shaded jars due to (mackrorysd: rev 5533648241b11f547ac10b64a126f3ae7a2f54c4) * (edit) dev-support/bin/create-release > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 3.0.0, 3.1.0, 3.0.1 > > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15080: --- Status: Patch Available (was: Open) Just "submitting" to get a Pre-Commit run - I'm not proposing we move forward with this until we've considered less drastic options. > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080.001.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277555#comment-16277555 ] Sean Mackrory commented on HADOOP-15058: [~andrew.wang] Sure I'll take a look. At first glance I agree with the approach. > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15058: --- Resolution: Fixed Fix Version/s: 3.0.1 3.1.0 3.0.0 Status: Resolved (was: Patch Available) Committed to trunk, branch-3.0 and branch-3.0.0. > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 3.0.0, 3.1.0, 3.0.1 > > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277748#comment-16277748 ] Íñigo Goiri commented on HADOOP-1: -- [~luky] thanks for this work; looks like a big improvement to the current FTP and SFTP implementations. I tried to go through the discussion in the JIRA but I'm pretty sure I missed some. My main question is the transition between the old FTP/SFTP filesystems and the current one. Should we make the new ones as defaults? Probably, the safest bet would be to release this and once tested in the wild, make the changes to point to this implementation by default and maybe mark the old ones as deprecated. > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.11.patch, > HADOOP-1.12.patch, HADOOP-1.13.patch, HADOOP-1.2.patch, > HADOOP-1.3.patch, HADOOP-1.4.patch, HADOOP-1.5.patch, > HADOOP-1.6.patch, HADOOP-1.7.patch, HADOOP-1.8.patch, > HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Mackrory updated HADOOP-15080:
---
Attachment: HADOOP-15080.001.patch
Just to make sure we have at least one legal option to proceed here, attaching
a patch that removes the aliyun-oss module by reverting all of the commits that
affect hadoop-tools/hadoop-aliyun. Of course if we can just replace the
dependency with another option, I'm obviously all for that instead. Just
putting this up here if there's no other option in the near future.
A couple of Maven version updates and some more general test modifications
(6a52b5e14495c5b2e0257aec65e61acd43aef309,
0d5c8ed8e0176909ee4d68b2969a882c21c0df55) were not reverted directly (hence the
conflicts below), and one other removal from the hadoop-cloud-storage module
was needed to compile (see HADOOP-13687). Here's a summary of all the other
commit messages to document what would need to be added back if we can figure
this out later on a slower timeframe:
{quote}
Revert "HADOOP-14787. AliyunOSS: Implement the `createNonRecursive`
operator."
This reverts commit 27ab5f7385c70f16fd593edc336c573c69f19331.
Revert "HADOOP-14194. Aliyun OSS should not use empty endpoint as default.
Contributed by Genmao Yu"
This reverts commit 267e19a09f366a965b30c8d4dc75e377b0d92fff.
Revert "HADOOP-14466. Remove useless document from
TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang."
This reverts commit 6e6e96583f1ee3c8a8775480f41c3adcdd5e2c45.
Revert "HADOOP-14458. Add missing imports to
TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu."
This reverts commit 07e60f85d87ca9a585d351a308ee0ecfa9293750.
Revert "HADOOP-14192. Aliyun OSS FileSystem contract test should implement
getTestBaseDir(). Contributed by Mingliang Liu"
This reverts commit 86035c16447eb0ad7989975639f83bc4b4b26a27.
Revert "HADOOP-14072. AliyunOSS: Failed to read from stream when seek
beyond the download size. Contributed by Genmao Yu"
This reverts commit cd3e59a3dcc69f68711777d448da5228a55846b3.
Revert "HADOOP-14069. AliyunOSS: listStatus returns wrong file info.
Contributed by Fei Hui"
This reverts commit 01be4503c3b053d2cff0b179774dabfd267877db.
Conflicts:
hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java
Revert "HADOOP-13768. AliyunOSS: handle the failure in the batch delete
operation `deleteDirs`. Contributed by Genmao Yu"
This reverts commit 5b151290ae2916dc04d6a4338085fcefafa21982.
Revert "HADOOP-14065. AliyunOSS: oss directory filestatus should use meta
time. Contributed by Fei Hui"
This reverts commit a8a594b4c89319bef294534755f0e4ed6198ec88.
Revert "HADOOP-13723. AliyunOSSInputStream#read() should update read bytes
stat correctly. Contributed by Mingliang Liu"
This reverts commit d9f73f1b7cd893a7d88baa9bfd1b809a5dec9e59.
Revert "HADOOP-13701. AbstractContractRootDirectoryTest can fail when
handling delete "/". Contributed by Genmao Yu"
This reverts commit c31b5e61b1f09949548116309218a2b3e9c0beda.
Revert "HADOOP-13634. Some configuration in doc has been outdated.
Contributed by Genmao Yu"
This reverts commit 26d5df390cf976dcc1d17fc68d0fed789dc34e84.
Revert "HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by
Genmao Yu"
This reverts commit 22af6f8db3a44cd51514b4851b99adcfad42751d.
Revert "HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and
TestOSSContractRootDir. Contributed by Genmao Yu"
This reverts commit 08b37603d9c0be67c4e0790c1ad266551ef21f5e.
Revert "HADOOP-13609. Refine credential provider related codes for
AliyunOss integration. Contributed by Genmao Yu"
This reverts commit 9cd47602576cd01a905e27642b685905a88eee72.
Revert "HADOOP-13610. Clean up AliyunOss integration tests. Contributed by
Genmao Yu"
This reverts commit a1940464a498d1e662e5c3843f2d31ce63ec726b.
Conflicts:
hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestOSSFileSystemContract.java
Revert "HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed
by Genmao Yu."
This reverts commit e671a0f52b5488b8453e1a3258ea5e6477995648.
Revert "HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu."
This reverts commit d33e928fbeb1764a724c8f3c051bb0d8be82bbff.
Conflicts:
hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestOSSFileSystemContract.java
Revert "HADOOP-13498. The number of multi-part upload part should not
bigger than 1. Contributed by Genmao Yu."
This reverts commit
[jira] [Commented] (HADOOP-15058) create-release site build outputs dummy shaded jars due to skipShade
[ https://issues.apache.org/jira/browse/HADOOP-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1624#comment-1624 ] Andrew Wang commented on HADOOP-15058: -- Thanks for reviewing Sean! Allen, if you have any more comments, I'd be happy to address in a follow-on. > create-release site build outputs dummy shaded jars due to skipShade > > > Key: HADOOP-15058 > URL: https://issues.apache.org/jira/browse/HADOOP-15058 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Blocker > Fix For: 3.0.0, 3.1.0, 3.0.1 > > Attachments: HADOOP-15058.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277782#comment-16277782 ] Aaron Fabbri commented on HADOOP-14475: --- Latest patch looks good to me. Tested w/ and w/o s3guard in us west 2 (and I did also see HADOOP-15079). I'll remove the one checkstyle unused import line and then commit this evening. > Metrics of S3A don't print out when enable it in Hadoop metrics property file > -- > > Key: HADOOP-14475 > URL: https://issues.apache.org/jira/browse/HADOOP-14475 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 > Environment: uname -a > Linux client01 4.4.0-74-generic #95-Ubuntu SMP Wed Apr 12 09:50:34 UTC 2017 > x86_64 x86_64 x86_64 GNU/Linux > cat /etc/issue > Ubuntu 16.04.2 LTS \n \l >Reporter: Yonger >Assignee: Yonger > Attachments: HADOOP-14475-003.patch, HADOOP-14475.002.patch, > HADOOP-14475.005.patch, HADOOP-14475.006.patch, HADOOP-14475.008.patch, > HADOOP-14475.009.patch, HADOOP-14475.010.patch, HADOOP-14475.011.patch, > HADOOP-14475.012.patch, HADOOP-14475.013.patch, HADOOP-14475.014.patch, > HADOOP-14475.015.patch, HADOOP-14475.016.patch, HADOOP-14475.017.patch, > HADOOP-14775.007.patch, failsafe-report-s3a-it.html, > failsafe-report-s3a-scale.html, failsafe-report-scale.html, > failsafe-report-scale.zip, s3a-metrics.patch1, stdout.zip > > > *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink > #*.sink.file.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #*.sink.influxdb.url=http:/xx > #*.sink.influxdb.influxdb_port=8086 > #*.sink.influxdb.database=hadoop > #*.sink.influxdb.influxdb_username=hadoop > #*.sink.influxdb.influxdb_password=hadoop > #*.sink.ingluxdb.cluster=c1 > *.period=10 > #namenode.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > #S3AFileSystem.sink.influxdb.class=org.apache.hadoop.metrics2.sink.influxdb.InfluxdbSink > S3AFileSystem.sink.file.filename=s3afilesystem-metrics.out > I can't find the out put file even i run a MR job which should be used s3. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277783#comment-16277783 ] Andrew Wang commented on HADOOP-15080: -- Based on today's activity on LEGAL-349, this dependency needs to be removed. What's the next step? [~Sammi] do you have a patch to either replace json-lib or revert out Aliyun OSS? > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15071) s3a troubleshooting docs to add a couple more failure modes
[ https://issues.apache.org/jira/browse/HADOOP-15071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277823#comment-16277823 ] Aaron Fabbri commented on HADOOP-15071: --- +1 LGTM. > s3a troubleshooting docs to add a couple more failure modes > --- > > Key: HADOOP-15071 > URL: https://issues.apache.org/jira/browse/HADOOP-15071 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.8.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-15071-001.patch > > > I've got some more troubleshooting entries to add -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15080: --- Attachment: (was: HADOOP-15080.001.patch) > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14985) Remove subversion related code from VersionInfoMojo.java
[
https://issues.apache.org/jira/browse/HADOOP-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278056#comment-16278056
]
Hudson commented on HADOOP-14985:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13323 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13323/])
HADOOP-14985. Remove subversion related code from VersionInfoMojo.java.
(aajisaka: rev 9f1bdafedb60f83598819c6a682f659d6e168eb0)
* (edit)
hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/versioninfo/VersionInfoMojo.java
> Remove subversion related code from VersionInfoMojo.java
>
>
> Key: HADOOP-14985
> URL: https://issues.apache.org/jira/browse/HADOOP-14985
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Ajay Kumar
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HADOOP-14985.001.patch, HADOOP-14985.002.patch
>
>
> When building Apache Hadoop, we can see the following message:
> {noformat}
> [WARNING] [svn, info] failed with error code 1
> {noformat}
> We have migrated to the code base from svn to git, so the message is useless.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Genmao Yu updated HADOOP-15039:
---
Attachment: HADOOP-15039.004.patch
> Move SemaphoredDelegatingExecutor to hadoop-common
> --
>
> Key: HADOOP-15039
> URL: https://issues.apache.org/jira/browse/HADOOP-15039
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/oss, fs/s3
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>Priority: Minor
> Attachments: HADOOP-15039.001.patch, HADOOP-15039.002.patch,
> HADOOP-15039.003.patch, HADOOP-15039.004.patch
>
>
> Detailed discussions in HADOOP-14999 and HADOOP-15027.
> share {{SemaphoredDelegatingExecutor}} and move it to {{hadoop-common}}.
> cc [~ste...@apache.org]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277964#comment-16277964 ] SammiChen commented on HADOOP-15080: Hi [~andrew.wang], I'm working with Aliyun oss team to find the solutions. The good news is that the json-lib library is only used in oss sdk test functions. Currently it's marked as "compile dependency" while actually it only requires "test dependency". So basically two solutions, 1. A new oss sdk which marks "json-lib" as "test dependency". As test dependency is intransitive. Hadoop will not been impacted. 2. A new oss sdk, use "jersey-json" 1.9 version. "jersey-json" is also "test dependency". Solution 2 is preferred for it solve the problem once-at-all. It will takes 2~3 days to prepare the new oss sdk. Do you have any concerns about solution 2? In the meantime, I agree we should prepare a patch to revert out Aliyun OSS support to not block the release. But if we still have time, may I ask a favor to hold the revert for several days? Also thanks [~mackrorysd] for prepare the patch. > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.001.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277975#comment-16277975
]
genericqa commented on HADOOP-15080:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
48s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.0.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
27s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
30s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-tools
hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} branch-3.0.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 45s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
hadoop-cloud-storage-project/hadoop-cloud-storage {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-tools-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 47m
30s{color} | {color:green} hadoop-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-cloud-storage in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278041#comment-16278041
]
genericqa commented on HADOOP-15080:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 19 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.0.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
42s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
50s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-cloud-storage-project/hadoop-cloud-storage hadoop-tools
hadoop-tools/hadoop-tools-dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} branch-3.0.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} branch-3.0.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 38s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-tools/hadoop-tools-dist hadoop-tools
hadoop-cloud-storage-project/hadoop-cloud-storage {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
18s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-tools-dist in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m
49s{color} | {color:green} hadoop-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-cloud-storage in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not
[jira] [Commented] (HADOOP-3) Output directories are not cleaned up before the reduces run
[ https://issues.apache.org/jira/browse/HADOOP-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277989#comment-16277989 ] Hudson commented on HADOOP-3: - FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4170 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4170/]) HBASE-18942 hbase-hadoop2-compat module ignores hadoop-3 profile (apurtell: rev 6d69fd9c7871ffc3878eefa307da0e49bb7205f5) * (edit) hbase-hadoop2-compat/pom.xml > Output directories are not cleaned up before the reduces run > > > Key: HADOOP-3 > URL: https://issues.apache.org/jira/browse/HADOOP-3 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Minor > Fix For: 0.1.0 > > Attachments: clean-out-dir.patch, noclobber.patch > > > The output directory for the reduces is not cleaned up and therefore if you > can see left overs from previous runs, if they had more reduces. For example, > if you run the application once with reduces=10 and then rerun with > reduces=8, your output directory will have frag0 to frag9 with the > first 8 fragments from the second run and the last 2 fragments from the first > run. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14976) Set HADOOP_SHELL_EXECNAME explicitly in scripts
[
https://issues.apache.org/jira/browse/HADOOP-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278033#comment-16278033
]
Hudson commented on HADOOP-14976:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13322 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13322/])
HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts. (arp: rev
e00c7f78c1c00467319ce5b92e4a3ebc691d246e)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/scripts/hadoop_shell_execname.bats
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* (edit) hadoop-mapreduce-project/bin/mapred
* (edit) hadoop-common-project/hadoop-common/src/main/bin/hadoop
* (edit) hadoop-yarn-project/hadoop-yarn/bin/yarn
* (add)
hadoop-common-project/hadoop-common/src/test/scripts/hadoop_shell_execname.bats
> Set HADOOP_SHELL_EXECNAME explicitly in scripts
> ---
>
> Key: HADOOP-14976
> URL: https://issues.apache.org/jira/browse/HADOOP-14976
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.1.0
>
> Attachments: HADOOP-14976.01.patch, HADOOP-14976.02.patch,
> HADOOP-14976.03.patch, HADOOP-14976.04.patch
>
>
> Some Hadoop shell scripts infer their own name using this bit of shell magic:
> {code}
> 18 MYNAME="${BASH_SOURCE-$0}"
> 19 HADOOP_SHELL_EXECNAME="${MYNAME##*/}"
> {code}
> e.g. see the
> [hdfs|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs#L18]
> script.
> The inferred shell script name is later passed to _hadoop-functions.sh_ which
> uses it to construct the names of some environment variables. E.g. when
> invoking _hdfs datanode_, the options variable name is inferred as follows:
> {code}
> # HDFS + DATANODE + OPTS -> HDFS_DATANODE_OPTS
> {code}
> This works well if the calling script name is standard {{hdfs}} or {{yarn}}.
> If a distribution renames the script to something like foo.bar, , then the
> variable names will be inferred as {{FOO.BAR_DATANODE_OPTS}}. This is not a
> valid bash variable name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[
https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277862#comment-16277862
]
genericqa commented on HADOOP-15080:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HADOOP-15080 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15080 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900584/HADOOP-15080.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13780/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Cat-X dependency on org.json via derived json-lib
> -
>
> Key: HADOOP-15080
> URL: https://issues.apache.org/jira/browse/HADOOP-15080
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/oss
>Affects Versions: 3.0.0-beta1
>Reporter: Chris Douglas
>Priority: Blocker
> Attachments: HADOOP-15080.001.patch
>
>
> The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library
> (from which json-lib may be derived) is released under a
> [category-x|https://www.apache.org/legal/resolved.html#json] license.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15080: --- Attachment: (was: HADOOP-15080-branch-3.0.0.patch) > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.001.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15080: --- Attachment: HADOOP-15080-branch-3.0.0.001.patch > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.001.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14985) Remove subversion related code from VersionInfoMojo.java
[
https://issues.apache.org/jira/browse/HADOOP-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-14985:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.1
2.9.1
2.10.0
3.1.0
Status: Resolved (was: Patch Available)
Committed this to trunk, branch-3.0, branch-2, and branch-2.9. Thanks
[~ajayydv] for the contribution!
> Remove subversion related code from VersionInfoMojo.java
>
>
> Key: HADOOP-14985
> URL: https://issues.apache.org/jira/browse/HADOOP-14985
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Ajay Kumar
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.1
>
> Attachments: HADOOP-14985.001.patch, HADOOP-14985.002.patch
>
>
> When building Apache Hadoop, we can see the following message:
> {noformat}
> [WARNING] [svn, info] failed with error code 1
> {noformat}
> We have migrated to the code base from svn to git, so the message is useless.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15080) Cat-X dependency on org.json via derived json-lib
[ https://issues.apache.org/jira/browse/HADOOP-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HADOOP-15080: --- Attachment: HADOOP-15080-branch-3.0.0.patch > Cat-X dependency on org.json via derived json-lib > - > > Key: HADOOP-15080 > URL: https://issues.apache.org/jira/browse/HADOOP-15080 > Project: Hadoop Common > Issue Type: Bug > Components: fs/oss >Affects Versions: 3.0.0-beta1 >Reporter: Chris Douglas >Priority: Blocker > Attachments: HADOOP-15080-branch-3.0.0.patch > > > The OSS SDK has a dependency on json-lib. In LEGAL-245, the org.json library > (from which json-lib may be derived) is released under a > [category-x|https://www.apache.org/legal/resolved.html#json] license. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14976) Allow overriding HADOOP_SHELL_EXECNAME
[
https://issues.apache.org/jira/browse/HADOOP-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277950#comment-16277950
]
Allen Wittenauer commented on HADOOP-14976:
---
+1 lgtm
> Allow overriding HADOOP_SHELL_EXECNAME
> --
>
> Key: HADOOP-14976
> URL: https://issues.apache.org/jira/browse/HADOOP-14976
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HADOOP-14976.01.patch, HADOOP-14976.02.patch,
> HADOOP-14976.03.patch, HADOOP-14976.04.patch
>
>
> Some Hadoop shell scripts infer their own name using this bit of shell magic:
> {code}
> 18 MYNAME="${BASH_SOURCE-$0}"
> 19 HADOOP_SHELL_EXECNAME="${MYNAME##*/}"
> {code}
> e.g. see the
> [hdfs|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs#L18]
> script.
> The inferred shell script name is later passed to _hadoop-functions.sh_ which
> uses it to construct the names of some environment variables. E.g. when
> invoking _hdfs datanode_, the options variable name is inferred as follows:
> {code}
> # HDFS + DATANODE + OPTS -> HDFS_DATANODE_OPTS
> {code}
> This works well if the calling script name is standard {{hdfs}} or {{yarn}}.
> If a distribution renames the script to something like foo.bar, , then the
> variable names will be inferred as {{FOO.BAR_DATANODE_OPTS}}. This is not a
> valid bash variable name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14976) Set HADOOP_SHELL_EXECNAME explicitly in scripts
[
https://issues.apache.org/jira/browse/HADOOP-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14976:
---
Summary: Set HADOOP_SHELL_EXECNAME explicitly in scripts (was: Allow
overriding HADOOP_SHELL_EXECNAME)
> Set HADOOP_SHELL_EXECNAME explicitly in scripts
> ---
>
> Key: HADOOP-14976
> URL: https://issues.apache.org/jira/browse/HADOOP-14976
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HADOOP-14976.01.patch, HADOOP-14976.02.patch,
> HADOOP-14976.03.patch, HADOOP-14976.04.patch
>
>
> Some Hadoop shell scripts infer their own name using this bit of shell magic:
> {code}
> 18 MYNAME="${BASH_SOURCE-$0}"
> 19 HADOOP_SHELL_EXECNAME="${MYNAME##*/}"
> {code}
> e.g. see the
> [hdfs|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs#L18]
> script.
> The inferred shell script name is later passed to _hadoop-functions.sh_ which
> uses it to construct the names of some environment variables. E.g. when
> invoking _hdfs datanode_, the options variable name is inferred as follows:
> {code}
> # HDFS + DATANODE + OPTS -> HDFS_DATANODE_OPTS
> {code}
> This works well if the calling script name is standard {{hdfs}} or {{yarn}}.
> If a distribution renames the script to something like foo.bar, , then the
> variable names will be inferred as {{FOO.BAR_DATANODE_OPTS}}. This is not a
> valid bash variable name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14976) Set HADOOP_SHELL_EXECNAME explicitly in scripts
[
https://issues.apache.org/jira/browse/HADOOP-14976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HADOOP-14976:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Thank you for the review and suggestions Allen.
Also, thanks Chen.
> Set HADOOP_SHELL_EXECNAME explicitly in scripts
> ---
>
> Key: HADOOP-14976
> URL: https://issues.apache.org/jira/browse/HADOOP-14976
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.1.0
>
> Attachments: HADOOP-14976.01.patch, HADOOP-14976.02.patch,
> HADOOP-14976.03.patch, HADOOP-14976.04.patch
>
>
> Some Hadoop shell scripts infer their own name using this bit of shell magic:
> {code}
> 18 MYNAME="${BASH_SOURCE-$0}"
> 19 HADOOP_SHELL_EXECNAME="${MYNAME##*/}"
> {code}
> e.g. see the
> [hdfs|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs#L18]
> script.
> The inferred shell script name is later passed to _hadoop-functions.sh_ which
> uses it to construct the names of some environment variables. E.g. when
> invoking _hdfs datanode_, the options variable name is inferred as follows:
> {code}
> # HDFS + DATANODE + OPTS -> HDFS_DATANODE_OPTS
> {code}
> This works well if the calling script name is standard {{hdfs}} or {{yarn}}.
> If a distribution renames the script to something like foo.bar, , then the
> variable names will be inferred as {{FOO.BAR_DATANODE_OPTS}}. This is not a
> valid bash variable name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14985) Remove subversion related code from VersionInfoMojo.java
[
https://issues.apache.org/jira/browse/HADOOP-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278031#comment-16278031
]
Akira Ajisaka commented on HADOOP-14985:
+1, committing this.
> Remove subversion related code from VersionInfoMojo.java
>
>
> Key: HADOOP-14985
> URL: https://issues.apache.org/jira/browse/HADOOP-14985
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14985.001.patch, HADOOP-14985.002.patch
>
>
> When building Apache Hadoop, we can see the following message:
> {noformat}
> [WARNING] [svn, info] failed with error code 1
> {noformat}
> We have migrated to the code base from svn to git, so the message is useless.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15039) Move SemaphoredDelegatingExecutor to hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16278162#comment-16278162
]
genericqa commented on HADOOP-15039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 2s{color} | {color:orange} root: The patch generated 1 new + 8 unchanged -
3 fixed = 9 total (was 11) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
38s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
39s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
33s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15039 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900617/HADOOP-15039.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 19af32fe1e63 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9f1bdaf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13783/artifact/out/diff-checkstyle-root.txt
|
| Test
[jira] [Commented] (HADOOP-3) Output directories are not cleaned up before the reduces run
[ https://issues.apache.org/jira/browse/HADOOP-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277883#comment-16277883 ] Hudson commented on HADOOP-3: - FAILURE: Integrated in Jenkins build HBase-1.4 #1049 (See [https://builds.apache.org/job/HBase-1.4/1049/]) HBASE-18942 hbase-hadoop2-compat module ignores hadoop-3 profile (apurtell: rev 9c8c0e08a66ced7dffb232facc51079e1de8a722) * (edit) hbase-hadoop2-compat/pom.xml > Output directories are not cleaned up before the reduces run > > > Key: HADOOP-3 > URL: https://issues.apache.org/jira/browse/HADOOP-3 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.1.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Minor > Fix For: 0.1.0 > > Attachments: clean-out-dir.patch, noclobber.patch > > > The output directory for the reduces is not cleaned up and therefore if you > can see left overs from previous runs, if they had more reduces. For example, > if you run the application once with reduces=10 and then rerun with > reduces=8, your output directory will have frag0 to frag9 with the > first 8 fragments from the second run and the last 2 fragments from the first > run. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276486#comment-16276486 ] Adam Sotona commented on HADOOP-1: -- Hi guys, this new ftp implementation actually saved several my projects. Original FTP filesystem implementation didn't work for way too many problems (passwords, proxies, performance, timeouts, etc...). My current use cases are: - Linux-based Hadoop clusters behind Http proxy - public or password-protected FTP servers with tons of medium to big files - use with the standalone copy (for servers with limited connections) and with distcp (for load-balanced servers where I can afford heavy parallelisation) And it works! Please speedup integration of this patch. Thank you, Adam > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.11.patch, > HADOOP-1.12.patch, HADOOP-1.13.patch, HADOOP-1.2.patch, > HADOOP-1.3.patch, HADOOP-1.4.patch, HADOOP-1.5.patch, > HADOOP-1.6.patch, HADOOP-1.7.patch, HADOOP-1.8.patch, > HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13837) Always get unable to kill error message even the hadoop process was successfully killed
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276464#comment-16276464
]
genericqa commented on HADOOP-13837:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
11s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-13837 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900444/HADOOP-13837.05.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux dffc115af861 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 37ca416 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13775/testReport/ |
| Max. process+thread count | 343 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13775/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Always get unable to kill error message even the hadoop process was
> successfully killed
> ---
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, HADOOP-13837.05.patch,
> check_proc.sh
>
>
> *Reproduce steps*
> # Setup a hadoop cluster
> # Stop resource manager : yarn --daemon stop resourcemanager
> # Stop node manager : yarn --daemon stop nodemanager
> WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill
> with kill -9
> ERROR: Unable to kill 20325
> it always gets "Unable to kill " error message, this gives user
> impression there is something wrong with the node manager process because it
> was not able to be
[jira] [Commented] (HADOOP-13837) Always get unable to kill error message even the hadoop process was successfully killed
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276454#comment-16276454
]
genericqa commented on HADOOP-13837:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
3s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-13837 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900444/HADOOP-13837.05.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux d5d6bb24eafb 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 37ca416 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13774/testReport/ |
| Max. process+thread count | 315 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13774/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Always get unable to kill error message even the hadoop process was
> successfully killed
> ---
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, HADOOP-13837.05.patch,
> check_proc.sh
>
>
> *Reproduce steps*
> # Setup a hadoop cluster
> # Stop resource manager : yarn --daemon stop resourcemanager
> # Stop node manager : yarn --daemon stop nodemanager
> WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill
> with kill -9
> ERROR: Unable to kill 20325
> it always gets "Unable to kill " error message, this gives user
> impression there is something wrong with the node manager process because it
> was not able to be
[jira] [Commented] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch
[
https://issues.apache.org/jira/browse/HADOOP-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276729#comment-16276729
]
Steve Loughran commented on HADOOP-15079:
-
OK, so I did break it then :(
> ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after
> OutputCommitter patch
> ---
>
> Key: HADOOP-15079
> URL: https://issues.apache.org/jira/browse/HADOOP-15079
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Sean Mackrory
>Priority: Critical
>
> I see this test failing with "object_delete_requests expected:<1> but
> was:<2>". I printed stack traces whenever this metric was incremented, and
> found the root cause to be that innerMkdirs is now causing two calls to
> delete fake directories when it previously caused only one. It is called once
> inside createFakeDirectory, and once directly inside innerMkdirs later:
> {code}
> at
> org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1625)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:2634)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.finishedWrite(S3AFileSystem.java:2599)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1498)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$11(S3AFileSystem.java:2684)
> at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:108)
> at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:259)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:255)
> at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:230)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2682)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2657)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2021)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1956)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2305)
> at
> org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
> at
> org.apache.hadoop.fs.s3a.ITestS3AFileOperationCost.testFakeDirectoryDeletion(ITestS3AFileOperationCost.java:209)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74
> {code}
> {code}
> at
> org.apache.hadoop.fs.s3a.S3AInstrumentation.incrementCounter(S3AInstrumentation.java:454)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.incrementStatistic(S3AFileSystem.java:1108)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$8(S3AFileSystem.java:1369)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:313)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:279)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:1366)
> at
>
[jira] [Updated] (HADOOP-10054) ViewFsFileStatus.toString() is broken
[
https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HADOOP-10054:
Attachment: HADOOP-10054.001.patch
> ViewFsFileStatus.toString() is broken
> -
>
> Key: HADOOP-10054
> URL: https://issues.apache.org/jira/browse/HADOOP-10054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.0.5-alpha
>Reporter: Paul Han
>Assignee: Hanisha Koneru
>Priority: Minor
> Attachments: HADOOP-10054.001.patch
>
>
> ViewFsFileStatus.toString is broken. Following code snippet :
> {code}
> FileStatus stat= somefunc(); // somefunc() returns an instance of
> ViewFsFileStatus
> System.out.println("path:" + stat.getPath());
> System.out.println(stat.toString());
> {code}
> produces the output:
> {code}
> path:viewfs://x.com/user/X/tmp-48
> ViewFsFileStatus{path=null; isDirectory=false; length=0; replication=0;
> blocksize=0; modification_time=0; access_time=0; owner=; group=;
> permission=rw-rw-rw-; isSymlink=false}
> {code}
> Note that "path=null" is not correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10054) ViewFsFileStatus.toString() is broken
[
https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanisha Koneru updated HADOOP-10054:
Status: Patch Available (was: Open)
> ViewFsFileStatus.toString() is broken
> -
>
> Key: HADOOP-10054
> URL: https://issues.apache.org/jira/browse/HADOOP-10054
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.0.5-alpha
>Reporter: Paul Han
>Assignee: Hanisha Koneru
>Priority: Minor
> Attachments: HADOOP-10054.001.patch
>
>
> ViewFsFileStatus.toString is broken. Following code snippet :
> {code}
> FileStatus stat= somefunc(); // somefunc() returns an instance of
> ViewFsFileStatus
> System.out.println("path:" + stat.getPath());
> System.out.println(stat.toString());
> {code}
> produces the output:
> {code}
> path:viewfs://x.com/user/X/tmp-48
> ViewFsFileStatus{path=null; isDirectory=false; length=0; replication=0;
> blocksize=0; modification_time=0; access_time=0; owner=; group=;
> permission=rw-rw-rw-; isSymlink=false}
> {code}
> Note that "path=null" is not correct.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14565) Azure: Add Authorization support to ADLS
[
https://issues.apache.org/jira/browse/HADOOP-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276915#comment-16276915
]
genericqa commented on HADOOP-14565:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HADOOP-14565 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14565 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880108/HADOOP_14565__Added_authorizer_functionality_to_ADL_driver.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13778/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Azure: Add Authorization support to ADLS
>
>
> Key: HADOOP-14565
> URL: https://issues.apache.org/jira/browse/HADOOP-14565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: Ryan Waters
>Assignee: Sivaguru Sankaridurg
> Attachments:
> HADOOP_14565__Added_authorizer_functionality_to_ADL_driver.patch
>
>
> This task is meant to add an Authorizer interface to be used by the ADLS
> driver in a similar way to the one used by WASB. The primary difference in
> functionality being that the implementation of this Authorizer will be
> provided by an external jar. This class will be specified through
> configuration using "adl.external.authorization.class".
> If this configuration is provided, an instance of the provided class will be
> created and all file system calls will be passed through the authorizer,
> allowing implementations to determine if the file path and access type
> (create, open, delete, etc.) being requested is valid. If the requested
> implementation class is not found or it fails to initialize, it will fail
> initialization of the ADL driver. If no configuration is provided, calls to
> the authorizer will be skipped and the driver will behave as it did
> previously.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15087) Write directly without creating temp directory to avoid rename
[ https://issues.apache.org/jira/browse/HADOOP-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276837#comment-16276837 ] Steve Loughran commented on HADOOP-15087: - The key flaw with the existing committers is not that they are are slow, it is in the absence of consistent file listings rename() can miss things to copy, so can deliver bad answers. You can't safely use the normal output committers against AWS S3 without S3Guard, though other implementations (yours too?) can be have differently. Have you played with the S3A committers? I think we can outdo stocator with better failure semantics, though I've got to benchmark it properly. Why don't you have a go there? Be good to see your patch though, as it'd be something to line up all commit strategies, "class/broken", "s3a staging", "s3a magic", "stocator". FWIW, Teragen is a meaningless benchmark except as a stress test of a cluster and bootstrap to terasort tests; it doesn't resemble any real workloads. TCP-DS is the one to play with. See also: http://steveloughran.blogspot.co.uk/2017/09/stocator-high-performance-object-store.html > Write directly without creating temp directory to avoid rename > --- > > Key: HADOOP-15087 > URL: https://issues.apache.org/jira/browse/HADOOP-15087 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Yonger > > Rename in workloads like Teragen/Terasort who use Hadoop default > outputcommitters really hurt performance a lot. > Stocator announce it doesn't create the temporary directories any all, and > still preserves Hadoop's fault tolerance. I add a switch when creating file > via integrating it's code into s3a, I got 5x performance gain in Teragen and > 15% performance improvement in Terasort. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15087) S3A to support writing directly to the destination dir without creating temp directory to avoid rename
[ https://issues.apache.org/jira/browse/HADOOP-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15087: Summary: S3A to support writing directly to the destination dir without creating temp directory to avoid rename (was: Write directly without creating temp directory to avoid rename ) > S3A to support writing directly to the destination dir without creating temp > directory to avoid rename > --- > > Key: HADOOP-15087 > URL: https://issues.apache.org/jira/browse/HADOOP-15087 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Yonger > > Rename in workloads like Teragen/Terasort who use Hadoop default > outputcommitters really hurt performance a lot. > Stocator announce it doesn't create the temporary directories any all, and > still preserves Hadoop's fault tolerance. I add a switch when creating file > via integrating it's code into s3a, I got 5x performance gain in Teragen and > 15% performance improvement in Terasort. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14444) New implementation of ftp and sftp filesystems
[ https://issues.apache.org/jira/browse/HADOOP-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276848#comment-16276848 ] Steve Loughran commented on HADOOP-1: - Good to hear Adam. Lukas: announce on common-dev and ask for help testing & reviewing > New implementation of ftp and sftp filesystems > -- > > Key: HADOOP-1 > URL: https://issues.apache.org/jira/browse/HADOOP-1 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: Lukas Waldmann >Assignee: Lukas Waldmann > Attachments: HADOOP-1.10.patch, HADOOP-1.11.patch, > HADOOP-1.12.patch, HADOOP-1.13.patch, HADOOP-1.2.patch, > HADOOP-1.3.patch, HADOOP-1.4.patch, HADOOP-1.5.patch, > HADOOP-1.6.patch, HADOOP-1.7.patch, HADOOP-1.8.patch, > HADOOP-1.9.patch, HADOOP-1.patch > > > Current implementation of FTP and SFTP filesystems have severe limitations > and performance issues when dealing with high number of files. Mine patch > solve those issues and integrate both filesystems such a way that most of the > core functionality is common for both and therefore simplifying the > maintainability. > The core features: > * Support for HTTP/SOCKS proxies > * Support for passive FTP > * Support for explicit FTPS (SSL/TLS) > * Support of connection pooling - new connection is not created for every > single command but reused from the pool. > For huge number of files it shows order of magnitude performance improvement > over not pooled connections. > * Caching of directory trees. For ftp you always need to list whole directory > whenever you ask information about particular file. > Again for huge number of files it shows order of magnitude performance > improvement over not cached connections. > * Support of keep alive (NOOP) messages to avoid connection drops > * Support for Unix style or regexp wildcard glob - useful for listing a > particular files across whole directory tree > * Support for reestablishing broken ftp data transfers - can happen > surprisingly often -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14294) Rename ADLS mountpoint properties
[
https://issues.apache.org/jira/browse/HADOOP-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14294:
Parent Issue: HADOOP-14764 (was: HADOOP-14112)
> Rename ADLS mountpoint properties
> -
>
> Key: HADOOP-14294
> URL: https://issues.apache.org/jira/browse/HADOOP-14294
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Rohan Garg
>
> Follow up to HADOOP-14038. Rename the prefix of
> {{dfs.adls..mountpoint}} and {{dfs.adls..hostname}} to
> {{fs.adl.}}.
> Borrow code from
> https://issues.apache.org/jira/secure/attachment/12857500/HADOOP-14038.006.patch
> and add a few unit tests.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13983) Print better error when accessing a non-existent ADL store
[
https://issues.apache.org/jira/browse/HADOOP-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13983:
Parent Issue: HADOOP-14764 (was: HADOOP-14112)
> Print better error when accessing a non-existent ADL store
> --
>
> Key: HADOOP-13983
> URL: https://issues.apache.org/jira/browse/HADOOP-13983
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Atul Sikaria
>Priority: Minor
> Labels: supportability
>
> The error message when accessing a non-existent store is not user friendly:
> {noformat}
> $ hdfs dfs -ls adl://STORE.azuredatalakestore.net/
> ls: Operation GETFILESTATUS failed with exception
> java.net.UnknownHostException : STORE.azuredatalakestore.net
> {noformat}
> Hadoop is configured with a valid SPI but {{STORE}} does not exist.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14175) NPE when ADL store URI contains underscore
[
https://issues.apache.org/jira/browse/HADOOP-14175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14175:
Parent Issue: HADOOP-14764 (was: HADOOP-14112)
> NPE when ADL store URI contains underscore
> --
>
> Key: HADOOP-14175
> URL: https://issues.apache.org/jira/browse/HADOOP-14175
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Xiaobing Zhou
>Priority: Minor
> Labels: newbie, supportability
>
> Please note the underscore {{_}} in the store name {{jzhuge_adls}}. Same NPE
> wherever the underscore in the URI.
> {noformat}
> $ bin/hadoop fs -ls adl://jzhuge_adls.azuredatalakestore.net/
> -ls: Fatal internal error
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.adl.AdlFileSystem.initialize(AdlFileSystem.java:145)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3257)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3306)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3274)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
> at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:249)
> at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:232)
> at
> org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:103)
> at org.apache.hadoop.fs.shell.Command.run(Command.java:176)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:326)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:389)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14112) Über-jira adl:// Azure Data Lake Phase I: Stabilization
[ https://issues.apache.org/jira/browse/HADOOP-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-14112. - Resolution: Fixed Fix Version/s: 2.9.0 3.0.0 Marking as closed in 2.9.0/3.0.0; all new items should go under ADL phase II, HADOOP-14764 > Über-jira adl:// Azure Data Lake Phase I: Stabilization > --- > > Key: HADOOP-14112 > URL: https://issues.apache.org/jira/browse/HADOOP-14112 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: Steve Loughran >Assignee: John Zhuge > Fix For: 3.0.0, 2.9.0 > > > Uber JIRA to track things needed for Azure Data Lake to be considered stable > and adl:// ready for wide use. > Based on the experience with other object stores, the things which usually > surface once a stabilizing FS is picked up and used are > * handling of many GB files, up and down, be it: efficiency of read, when the > writes take place, file leakage, time for close() and filesystem shutdown > * resilience to transient failures > * reporting of problems/diagnostics > * security option tuning > * race conditions > * differences between implementation and what actual applications expect -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15086) NativeAzureFileSystem.rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15086. - Resolution: Won't Fix > NativeAzureFileSystem.rename is not atomic > -- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
[
https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276822#comment-16276822
]
Steve Loughran commented on HADOOP-15038:
-
This is something we should bring up on the common-dev list.
# hadoop-cloud-core sounds nice
# I've been doing some work on cloudup locally (including a 2.7.x build); I'll
need to submit a new patch
# I've also been doing some other small patches factoring out common code from
stores (HADOOP-14943), where again, this stuff can be shared
Essentially: we've been copying and pasting stuff between versions, and it's
reached the limits of maintenance. See
[https://hortonworks.com/blog/history-apache-hadoops-support-amazon-s3/] for
the illustration of that pasting.
Things I'd like to see in there
* the core "mimic a filesystem" functions
* standard statistic collection & names
* retry logic of S3A.Invoker
* Support for marshalling login secrets as filesystem delegation tokens (see
HADOOP-14556). This is needed for users to submit their own credentials &
encryption keys to shared query engines
* Any CLI utility for listing/viewing things (see "hadoop s3guard
bucket-info"),
* Any CLI utility we can do for diagnostics. Your support team will love you
for this.
* Any more integration tests we can do beyond the basic abstract contract &
distcp tests. We now have a variant of the [Hadoop MR protocol test
suite|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitProtocol.java]
and [MR
Job|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitMRJob.java]
One thing to consider though, the Ozone store under HDFS could be looking at
some of this stuff too, ultimately. Which means that hadoop-common.jar is the
right place to put this stuff, at least until there's a compelling reason to
split it out. (Except: do that, things start to depend on it, and splitting it
becomes impossible
Thoughts?
BTW, if you haven't noticed, I've got a module designed to do some integration
with and testing under Apache Spark:
https://github.com/hortonworks-spark/cloud-integration . At some point I hope
to submit to Apache Bahir, for now it's a bit too unstable.
> Abstract MetadataStore in S3Guard into a common module.
> ---
>
> Key: HADOOP-15038
> URL: https://issues.apache.org/jira/browse/HADOOP-15038
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Affects Versions: 3.0.0-beta1
>Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}}
> into a common module.
> Based on this work, other filesystem or object store can implement their own
> metastore for optimization (known issues like consistency problem and
> metadata operation performance). [~ste...@apache.org] and other guys have
> done many base and great works in {{S3Guard}}. It is very helpful to start
> work. I did some perf test in HADOOP-14098, and started related work for
> Aliyun OSS. Indeed there are still works to do for {{S3Guard}}, like
> metadata cache inconsistent with S3 and so on. It also will be a problem for
> other object store. However, we can do these works in parallel.
> [~ste...@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15088) BufferedInputStream.skip function can return 0 when the file is corrupted, causing an infinite loop
[
https://issues.apache.org/jira/browse/HADOOP-15088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276846#comment-16276846
]
Steve Loughran commented on HADOOP-15088:
-
Probably a common bug we've copied and pasted around. Why not add a new IOUtil
operation to do this, as other streams may do the same, especially the object
stores
> BufferedInputStream.skip function can return 0 when the file is corrupted,
> causing an infinite loop
> ---
>
> Key: HADOOP-15088
> URL: https://issues.apache.org/jira/browse/HADOOP-15088
> Project: Hadoop Common
> Issue Type: Bug
> Components: streaming
>Affects Versions: 2.5.0
>Reporter: John Doe
>
> When a file is corrupted, the skip function can return 0, causing an infinite
> loop.
> Here is the code:
> {code:java}
> private boolean slowReadUntilMatch(Pattern markPattern, boolean includePat,
> DataOutputBuffer outBufOrNull) throws
> IOException {
> ...
> for (long skiplen = endPos; skiplen > 0; ) {
> skiplen -= bin_.skip(skiplen);
> }
> ...
> }
> {code}
> Similar bugs are
> [Hadoop-8614|https://issues.apache.org/jira/browse/HADOOP-8614],
> [Yarn-2905|https://issues.apache.org/jira/browse/YARN-2905],
> [Yarn-163|https://issues.apache.org/jira/browse/YARN-163],
> [MAPREDUCE-6990|https://issues.apache.org/jira/browse/MAPREDUCE-6990]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14258) Verify and document ADLS client mount table feature
[
https://issues.apache.org/jira/browse/HADOOP-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14258:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-14764
> Verify and document ADLS client mount table feature
> ---
>
> Key: HADOOP-14258
> URL: https://issues.apache.org/jira/browse/HADOOP-14258
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Priority: Minor
>
> ADLS connector supports a simple form of client mount table (chrooted) so
> that multiple clusters can share a single store as the default filesystem
> without sharing any directories. Verify and document this feature.
> How to setup:
> * Set property {{dfs.adls..hostname}} to
> {{.azuredatalakestore.net}}
> * Set property {{dfs.adls..mountpoint}} to {{}}
> * URI {{adl:///...}} will be translated to
> {{adl://.azuredatalakestore.net/}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15086) NativeAzureFileSystem.rename is not atomic
[ https://issues.apache.org/jira/browse/HADOOP-15086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276796#comment-16276796 ] Steve Loughran commented on HADOOP-15086: - I don't disagree with you about the existence of the problem, just don't think it's easily fixed. Essentially: blobstores tend not to have a rename() (or indeed: create(overwrite=false), delete(directory), and the things we do to mimic this in our connectors aren't atomic 1. We cover this in [Object Stores|https://hado op.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/introduction.html#Object_Stores_vs._Filesystems] 2. This is also common to: S3x, Swift, OSS, ADL, ... 3. By inference, the Hadoop FileOutputCommit protocol is not atomic on object stores either. 4. Compare with the requirements of rename() as covered in [rename()|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/filesystem/filesystem.html#boolean_renamePath_src_Path_d] There is actually special support in Azure for atomic rename of HBase directories; this is done with leasing, recovery and stuff. It manages exclusivity, but it is still not an O(1) operation. If you look at where we are going with this, the work is in moving to object-store specific committers which provide the commit semantics without relying on renames. HADOOP-13786 is the initial implementation of this for S3A, but the hooks put into FileOutputFormat are designed to support filesystem-specific committers for any store which implements one. I'm closing as a WONTFIX. Sorry. It's not that we don't want to, it's just directory operations are where the metaphor "object stores are like filesystems" fail if you look closely enough. (On a brighter note: wasb is consistent of both metadata and data) > NativeAzureFileSystem.rename is not atomic > -- > > Key: HADOOP-15086 > URL: https://issues.apache.org/jira/browse/HADOOP-15086 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 2.7.3 >Reporter: Shixiong Zhu > Attachments: RenameReproducer.java > > > When multiple threads rename files to the same target path, more than 1 > threads can succeed. It's because check and copy file in `rename` is not > atomic. > I would expect it's atomic just like HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14766) Cloudup: an object store high performance dfs put command
[
https://issues.apache.org/jira/browse/HADOOP-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14766:
Status: Open (was: Patch Available)
Cancelling as I've been doing work in the external branch; this patch is out of
date
> Cloudup: an object store high performance dfs put command
> -
>
> Key: HADOOP-14766
> URL: https://issues.apache.org/jira/browse/HADOOP-14766
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure, fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14766-001.patch, HADOOP-14766-002.patch
>
>
> {{hdfs put local s3a://path}} is suboptimal as it treewalks down down the
> source tree then, sequentially, copies up the file through copying the file
> (opened as a stream) contents to a buffer, writes that to the dest file,
> repeats.
> For S3A that hurts because
> * it;s doing the upload inefficiently: the file can be uploaded just by
> handling the pathname to the AWS xter manager
> * it is doing it sequentially, when some parallelised upload would work.
> * as the ordering of the files to upload is a recursive treewalk, it doesn't
> spread the upload across multiple shards.
> Better:
> * build the list of files to upload
> * upload in parallel, picking entries from the list at random and spreading
> across a pool of uploaders
> * upload straight from local file (copyFromLocalFile()
> * track IO load (files created/second) to estimate risk of throttling.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14985) Remove subversion related code from VersionInfoMojo.java
[
https://issues.apache.org/jira/browse/HADOOP-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HADOOP-14985:
Attachment: HADOOP-14985.002.patch
[~ajisakaa], Thanks for catching that. Addressed it in new patch.
> Remove subversion related code from VersionInfoMojo.java
>
>
> Key: HADOOP-14985
> URL: https://issues.apache.org/jira/browse/HADOOP-14985
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Akira Ajisaka
>Assignee: Ajay Kumar
>Priority: Minor
> Attachments: HADOOP-14985.001.patch, HADOOP-14985.002.patch
>
>
> When building Apache Hadoop, we can see the following message:
> {noformat}
> [WARNING] [svn, info] failed with error code 1
> {noformat}
> We have migrated to the code base from svn to git, so the message is useless.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15087) S3A to support writing directly to the destination dir without creating temp directory to avoid rename
[ https://issues.apache.org/jira/browse/HADOOP-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15087: Affects Version/s: 3.0.0 Issue Type: New Feature (was: Improvement) > S3A to support writing directly to the destination dir without creating temp > directory to avoid rename > --- > > Key: HADOOP-15087 > URL: https://issues.apache.org/jira/browse/HADOOP-15087 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 >Affects Versions: 3.0.0 >Reporter: Yonger > > Rename in workloads like Teragen/Terasort who use Hadoop default > outputcommitters really hurt performance a lot. > Stocator announce it doesn't create the temporary directories any all, and > still preserves Hadoop's fault tolerance. I add a switch when creating file > via integrating it's code into s3a, I got 5x performance gain in Teragen and > 15% performance improvement in Terasort. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14112) Über-jira adl:// Azure Data Lake Phase I: Stabilization
[ https://issues.apache.org/jira/browse/HADOOP-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276896#comment-16276896 ] Steve Loughran commented on HADOOP-14112: - Are all these things in 2.9 and 3.0? If so, let's move the rest to phase II and close this one > Über-jira adl:// Azure Data Lake Phase I: Stabilization > --- > > Key: HADOOP-14112 > URL: https://issues.apache.org/jira/browse/HADOOP-14112 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Affects Versions: 3.0.0-alpha2 >Reporter: Steve Loughran >Assignee: John Zhuge > > Uber JIRA to track things needed for Azure Data Lake to be considered stable > and adl:// ready for wide use. > Based on the experience with other object stores, the things which usually > surface once a stabilizing FS is picked up and used are > * handling of many GB files, up and down, be it: efficiency of read, when the > writes take place, file leakage, time for close() and filesystem shutdown > * resilience to transient failures > * reporting of problems/diagnostics > * security option tuning > * race conditions > * differences between implementation and what actual applications expect -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14304) AdlStoreClient#getExceptionFromResponse should not swallow defaultMessage
[
https://issues.apache.org/jira/browse/HADOOP-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14304:
Parent Issue: HADOOP-14764 (was: HADOOP-14112)
> AdlStoreClient#getExceptionFromResponse should not swallow defaultMessage
> -
>
> Key: HADOOP-14304
> URL: https://issues.apache.org/jira/browse/HADOOP-14304
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Assignee: Atul Sikaria
> Labels: newbie, supportability
>
> Discovered the issue in HADOOP-14292.
> In {{AdlStoreClient}}, {{enumerateDirectoryInternal}} called
> {{getExceptionFromResponse}} with {{defaultMessage}} set to {{"Error
> enumerating directory " + path}}. This useful message was swallowed at
> https://github.com/Azure/azure-data-lake-store-java/blob/2.1.4/src/main/java/com/microsoft/azure/datalake/store/ADLStoreClient.java#L1106.
> Actually {{getExceptionFromResponse}} swallows {{defaultMessage}} at several
> places. Suggest always displaying the {{defaultMessage}} in some way.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13982) Print better error when accessing a store without permission
[
https://issues.apache.org/jira/browse/HADOOP-13982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13982:
Parent Issue: HADOOP-14764 (was: HADOOP-14112)
> Print better error when accessing a store without permission
>
>
> Key: HADOOP-13982
> URL: https://issues.apache.org/jira/browse/HADOOP-13982
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>Assignee: Atul Sikaria
> Labels: supportability
>
> The error message when accessing a store without permission is not user
> friendly:
> {noformat}
> $ hdfs dfs -ls adl://STORE.azuredatalakestore.net/
> ls: Operation GETFILESTATUS failed with HTTP403 : null
> {noformat}
> Store {{STORE}} exists but Hadoop is configured with an SPI that does not
> have access to the store.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14565) Azure: Add Authorization support to ADLS
[ https://issues.apache.org/jira/browse/HADOOP-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14565: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14764 > Azure: Add Authorization support to ADLS > > > Key: HADOOP-14565 > URL: https://issues.apache.org/jira/browse/HADOOP-14565 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: Ryan Waters >Assignee: Sivaguru Sankaridurg > Attachments: > HADOOP_14565__Added_authorizer_functionality_to_ADL_driver.patch > > > This task is meant to add an Authorizer interface to be used by the ADLS > driver in a similar way to the one used by WASB. The primary difference in > functionality being that the implementation of this Authorizer will be > provided by an external jar. This class will be specified through > configuration using "adl.external.authorization.class". > If this configuration is provided, an instance of the provided class will be > created and all file system calls will be passed through the authorizer, > allowing implementations to determine if the file path and access type > (create, open, delete, etc.) being requested is valid. If the requested > implementation class is not found or it fails to initialize, it will fail > initialization of the ADL driver. If no configuration is provided, calls to > the authorizer will be skipped and the driver will behave as it did > previously. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14579) Add Kerberos and Delegation token support to ADLS client
[
https://issues.apache.org/jira/browse/HADOOP-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14579:
Issue Type: Sub-task (was: Improvement)
Parent: HADOOP-14764
> Add Kerberos and Delegation token support to ADLS client
>
>
> Key: HADOOP-14579
> URL: https://issues.apache.org/jira/browse/HADOOP-14579
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl
>Affects Versions: 2.8.0
>Reporter: Anandsagar Kothapalli
>Assignee: Santhosh G Nayak
>
> Current implementation of Azure storage client for Hadoop ({{ADLS}}) does not
> support Kerberos Authentication and FileSystem authorization, which makes it
> unusable in secure environments with multi user setup.
> To make {{ADLS}} client more suitable to run in Secure environments,
> HADOOP-14565 is under way for providing the authorization and fine grained
> access control support.
> This JIRA is created to add Kerberos and delegation token support to {{ADLS}}
> client to fetch Azure Storage SAS keys, which provides fine grained timed
> access to containers and blobs.
> For delegation token management, the proposal is it use the same REST service
> which being used to generate the SAS Keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14565) Azure: Add Authorization support to ADLS
[ https://issues.apache.org/jira/browse/HADOOP-14565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276911#comment-16276911 ] Steve Loughran commented on HADOOP-14565: - Based on the experience of WASB # I'd like as realistic testing of this as possible, ideally even bringing up some minimal REST endpoint to do it. Why? Ensures that the production codepath gets tested before integration test time # add lots of logging, diagnostics and timeouts here. See #1 > Azure: Add Authorization support to ADLS > > > Key: HADOOP-14565 > URL: https://issues.apache.org/jira/browse/HADOOP-14565 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Affects Versions: 2.8.0 >Reporter: Ryan Waters >Assignee: Sivaguru Sankaridurg > Attachments: > HADOOP_14565__Added_authorizer_functionality_to_ADL_driver.patch > > > This task is meant to add an Authorizer interface to be used by the ADLS > driver in a similar way to the one used by WASB. The primary difference in > functionality being that the implementation of this Authorizer will be > provided by an external jar. This class will be specified through > configuration using "adl.external.authorization.class". > If this configuration is provided, an instance of the provided class will be > created and all file system calls will be passed through the authorizer, > allowing implementations to determine if the file path and access type > (create, open, delete, etc.) being requested is valid. If the requested > implementation class is not found or it fails to initialize, it will fail > initialization of the ADL driver. If no configuration is provided, calls to > the authorizer will be skipped and the driver will behave as it did > previously. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14259) Verify viewfs works with ADLS
[
https://issues.apache.org/jira/browse/HADOOP-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-14259:
Issue Type: Sub-task (was: Test)
Parent: HADOOP-14764
> Verify viewfs works with ADLS
> -
>
> Key: HADOOP-14259
> URL: https://issues.apache.org/jira/browse/HADOOP-14259
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/adl, viewfs
>Affects Versions: 2.8.0
>Reporter: John Zhuge
>Priority: Minor
>
> Many clusters can share a single ADL store as the default filesystem. In
> order to prevent directories of the same names but from different clusters to
> collide, use viewfs over ADLS filesystem:
> * Set {{fs.defaultFS}} to {{viewfs://clusterX}} for cluster X
> * Set {{fs.defaultFS}} to {{viewfs://clusterY}} for cluster Y
> * The viewfs client mount table should have entry clusterX and ClusterY
> Tasks
> * Verify all filesystem operations work as expected, especially rename and
> concat
> * Verify homedir entry works
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13972) ADLS to support per-store configuration
[
https://issues.apache.org/jira/browse/HADOOP-13972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276919#comment-16276919
]
Steve Loughran commented on HADOOP-13972:
-
The S3A code should be pulled up to hadoop common for this; we don't want
different semantics of per4-bucket options
including the (as yet uncommitted) work on jceks support, where getPassword()
looks for the bucket keys ahead of any more general ones: HADOOP-14507
> ADLS to support per-store configuration
> ---
>
> Key: HADOOP-13972
> URL: https://issues.apache.org/jira/browse/HADOOP-13972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/adl
>Affects Versions: 3.0.0-alpha2
>Reporter: John Zhuge
>
> Useful when distcp needs to access 2 Data Lake stores with different SPIs.
> Of course, a workaround is to grant the same SPI access permission to both
> stores, but sometimes it might not be feasible.
> One idea is to embed the store name in the configuration property names,
> e.g., {{dfs.adls.oauth2..client.id}}. Per-store keys will be consulted
> first, then fall back to the global keys.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15089) ADLS to implement AbstractContractDistCpTest
Steve Loughran created HADOOP-15089: --- Summary: ADLS to implement AbstractContractDistCpTest Key: HADOOP-15089 URL: https://issues.apache.org/jira/browse/HADOOP-15089 Project: Hadoop Common Issue Type: Sub-task Components: test Affects Versions: 3.0.0 Reporter: Steve Loughran Priority: Minor ADLS doesn't implement the AbstractContractDistCpTest yet. It should -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12876) [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations
[ https://issues.apache.org/jira/browse/HADOOP-12876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12876: Summary: [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations (was: [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent opeations) > [Azure Data Lake] Support for process level FileStatus cache to optimize > GetFileStatus frequent operations > -- > > Key: HADOOP-12876 > URL: https://issues.apache.org/jira/browse/HADOOP-12876 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/adl, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > > Add support to cache GetFileStatus and ListStatus response locally for > limited period of time. Local cache for limited period of time would optimize > number of calls for GetFileStatus operation. > One of the example where local limited period cache would be useful - > terasort ListStatus on input directory follows with GetFileStatus operation > on each file within directory. For 2048 input files in a directory would save > 2048 GetFileStatus calls during start up (Using the ListStatus response to > cache FileStatus instances). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12876) [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent opeations
[ https://issues.apache.org/jira/browse/HADOOP-12876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276923#comment-16276923 ] Steve Loughran commented on HADOOP-12876: - This could be done with the code in HADOOP-15038 once it is made common > [Azure Data Lake] Support for process level FileStatus cache to optimize > GetFileStatus frequent opeations > - > > Key: HADOOP-12876 > URL: https://issues.apache.org/jira/browse/HADOOP-12876 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/adl, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > > Add support to cache GetFileStatus and ListStatus response locally for > limited period of time. Local cache for limited period of time would optimize > number of calls for GetFileStatus operation. > One of the example where local limited period cache would be useful - > terasort ListStatus on input directory follows with GetFileStatus operation > on each file within directory. For 2048 input files in a directory would save > 2048 GetFileStatus calls during start up (Using the ListStatus response to > cache FileStatus instances). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14234) ADLS to implement FileSystemContractBaseTest
[ https://issues.apache.org/jira/browse/HADOOP-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14234: Summary: ADLS to implement FileSystemContractBaseTest (was: Improve ADLS FileSystem tests with JUnit4) > ADLS to implement FileSystemContractBaseTest > > > Key: HADOOP-14234 > URL: https://issues.apache.org/jira/browse/HADOOP-14234 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl, test >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > HADOOP-14180 switches FileSystem contract tests to JUnit4 and makes various > enhancements. Improve ADLS FileSystem contract tests based on that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15090) add adl troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15090: Status: Patch Available (was: Open) > add adl troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-15090-001.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15090) add adl troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-15090: Attachment: HADOOP-15090-001.patch patch 001. My stack traces from playing with adl > add adl troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran > Attachments: HADOOP-15090-001.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15090) add adl troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-15090: --- Assignee: Steve Loughran > add adl troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-15090-001.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14234) ADLS to implement FileSystemContractBaseTest
[ https://issues.apache.org/jira/browse/HADOOP-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14234: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14764 > ADLS to implement FileSystemContractBaseTest > > > Key: HADOOP-14234 > URL: https://issues.apache.org/jira/browse/HADOOP-14234 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl, test >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > HADOOP-14180 switches FileSystem contract tests to JUnit4 and makes various > enhancements. Improve ADLS FileSystem contract tests based on that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14234) ADLS to implement FileSystemContractBaseTest
[ https://issues.apache.org/jira/browse/HADOOP-14234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276930#comment-16276930 ] Steve Loughran commented on HADOOP-14234: - All the tests which are there are junit4; what's missing seems to be an implementation of FileSystemContractBaseTest (at least in the branch-2 in front of me right now). Renaming JIRA > ADLS to implement FileSystemContractBaseTest > > > Key: HADOOP-14234 > URL: https://issues.apache.org/jira/browse/HADOOP-14234 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl, test >Affects Versions: 2.8.0 >Reporter: John Zhuge >Priority: Minor > > HADOOP-14180 switches FileSystem contract tests to JUnit4 and makes various > enhancements. Improve ADLS FileSystem contract tests based on that. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14437) AdlFileSystem.getAclStatus() returns unexpected ACL information
[ https://issues.apache.org/jira/browse/HADOOP-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276979#comment-16276979 ] Steve Loughran commented on HADOOP-14437: - I don't like flags: change the API to match what is expected Assuming the code to use this in in-house, is moving it in sync with ADL releases going to be hard? > AdlFileSystem.getAclStatus() returns unexpected ACL information > --- > > Key: HADOOP-14437 > URL: https://issues.apache.org/jira/browse/HADOOP-14437 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Reporter: Sailesh Mukil > Labels: acl, api-breaking > > The HDFS getAclStatus() returns ACL information of a file and how that > pertains to a Hadoop user/group. > However, the ADLS getAclStatus() returns ACL information of a file but does > not map them to Hadoop users/groups and instead maps them to the client ID of > the SPI. > The components built around Hadoop use this API with the expectation that the > ACL information returned will be mapped to Hadoop users/groups, i.e. the > components expect it to have the same behavior as other filesystems that > support this API. > If not, the components need to have logic to handle this case where if > they're talking to ADLS, they won't look for matching the Hadoop user/group, > but instead the client ID. And once AdlFileSystem changes its API to be able > to map to Hadoop users/groups, it will be a breaking change for all the > components. > Until such functionality is able to be provided, I would suggest that the > getAclStatus() be unsupported, just as it is in S3AFileSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10054) ViewFsFileStatus.toString() is broken
[
https://issues.apache.org/jira/browse/HADOOP-10054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277015#comment-16277015
]
genericqa commented on HADOOP-10054:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 44s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 8s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-10054 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900491/HADOOP-10054.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a34d472a981f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 37ca416 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13777/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13777/testReport/ |
| Max. process+thread count | 1683 (vs. ulimit of 5000) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Updated] (HADOOP-12876) [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations
[ https://issues.apache.org/jira/browse/HADOOP-12876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-12876: Issue Type: Sub-task (was: Improvement) Parent: HADOOP-14764 > [Azure Data Lake] Support for process level FileStatus cache to optimize > GetFileStatus frequent operations > -- > > Key: HADOOP-12876 > URL: https://issues.apache.org/jira/browse/HADOOP-12876 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/adl, tools >Reporter: Vishwajeet Dusane >Assignee: Vishwajeet Dusane > > Add support to cache GetFileStatus and ListStatus response locally for > limited period of time. Local cache for limited period of time would optimize > number of calls for GetFileStatus operation. > One of the example where local limited period cache would be useful - > terasort ListStatus on input directory follows with GetFileStatus operation > on each file within directory. For 2048 input files in a directory would save > 2048 GetFileStatus calls during start up (Using the ListStatus response to > cache FileStatus instances). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15090) add adl troubleshooting doc
Steve Loughran created HADOOP-15090: --- Summary: add adl troubleshooting doc Key: HADOOP-15090 URL: https://issues.apache.org/jira/browse/HADOOP-15090 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs/adl Affects Versions: 2.9.0 Reporter: Steve Loughran Add a troubleshooting section/doc to the ADL docs based on our experiences. this should not be a substitute for improving the diagnostics/fixing the error messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14985) Remove subversion related code from VersionInfoMojo.java
[
https://issues.apache.org/jira/browse/HADOOP-14985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16276950#comment-16276950
]
genericqa commented on HADOOP-14985:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} hadoop-maven-plugins: The patch generated 0 new + 31
unchanged - 26 fixed = 31 total (was 57) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-maven-plugins in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-14985 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900494/HADOOP-14985.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 57510e91edc6 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 37ca416 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13776/testReport/ |
| Max. process+thread count | 329 (vs. ulimit of 5000) |
| modules | C: hadoop-maven-plugins U: hadoop-maven-plugins |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13776/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Updated] (HADOOP-14437) AdlFileSystem.getAclStatus() returns unexpected ACL information
[ https://issues.apache.org/jira/browse/HADOOP-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14437: Issue Type: Sub-task (was: Bug) Parent: HADOOP-14764 > AdlFileSystem.getAclStatus() returns unexpected ACL information > --- > > Key: HADOOP-14437 > URL: https://issues.apache.org/jira/browse/HADOOP-14437 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/adl >Reporter: Sailesh Mukil > Labels: acl, api-breaking > > The HDFS getAclStatus() returns ACL information of a file and how that > pertains to a Hadoop user/group. > However, the ADLS getAclStatus() returns ACL information of a file but does > not map them to Hadoop users/groups and instead maps them to the client ID of > the SPI. > The components built around Hadoop use this API with the expectation that the > ACL information returned will be mapped to Hadoop users/groups, i.e. the > components expect it to have the same behavior as other filesystems that > support this API. > If not, the components need to have logic to handle this case where if > they're talking to ADLS, they won't look for matching the Hadoop user/group, > but instead the client ID. And once AdlFileSystem changes its API to be able > to map to Hadoop users/groups, it will be a breaking change for all the > components. > Until such functionality is able to be provided, I would suggest that the > getAclStatus() be unsupported, just as it is in S3AFileSystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15090) add adl troubleshooting doc
[
https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277050#comment-16277050
]
genericqa commented on HADOOP-15090:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15090 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900503/HADOOP-15090-001.patch
|
| Optional Tests | asflicense mvnsite |
| uname | Linux 65bcda58695d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 37ca416 |
| maven | version: Apache Maven 3.3.9 |
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13779/artifact/out/whitespace-eol.txt
|
| Max. process+thread count | 304 (vs. ulimit of 5000) |
| modules | C: hadoop-tools/hadoop-azure-datalake U:
hadoop-tools/hadoop-azure-datalake |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13779/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add adl troubleshooting doc
> ---
>
> Key: HADOOP-15090
> URL: https://issues.apache.org/jira/browse/HADOOP-15090
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/adl
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-15090-001.patch
>
>
> Add a troubleshooting section/doc to the ADL docs based on our experiences.
> this should not be a substitute for improving the diagnostics/fixing the
> error messages.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15090) add adl troubleshooting doc
[ https://issues.apache.org/jira/browse/HADOOP-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277094#comment-16277094 ] Vishwajeet Dusane commented on HADOOP-15090: Thanks [~ste...@apache.org] for initiating trouble shooting doc. +1 for the patch. > add adl troubleshooting doc > --- > > Key: HADOOP-15090 > URL: https://issues.apache.org/jira/browse/HADOOP-15090 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/adl >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-15090-001.patch > > > Add a troubleshooting section/doc to the ADL docs based on our experiences. > this should not be a substitute for improving the diagnostics/fixing the > error messages. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
