[jira] [Commented] (HADOOP-11293) Factor OSType out from Shell
[
https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310637#comment-14310637
]
Hadoop QA commented on HADOOP-11293:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697220/HADOOP-11293.005.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 47 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 14 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
hadoop-tools/hadoop-azure hadoop-tools/hadoop-streaming
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.cli.TestHDFSCLI
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5625//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5625//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5625//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5625//console
This message is automatically generated.
Factor OSType out from Shell
Key: HADOOP-11293
URL: https://issues.apache.org/jira/browse/HADOOP-11293
Project: Hadoop Common
Issue Type: Improvement
Components: fs, util
Affects Versions: 2.7.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch,
HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch,
HADOOP-11293.005.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch
Currently the code that detects the OS type is located in Shell.java. Code
that need to check OS type refers to Shell, even if no other stuff of Shell
is needed.
I am proposing to refactor OSType out to its own class, so to make the
OSType easier to access and the dependency cleaner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310635#comment-14310635 ] Kai Zheng commented on HADOOP-11541: bq.you need a +1 from someone else (not yourself). OK. It clarifies the process for me, thanks. I thought it has been relaxed somehow for the branch but not sure how I got the impression. Thanks for your +1. I'll get it in so am able to sort out other patches. Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310640#comment-14310640 ] Kai Zheng commented on HADOOP-11541: commit e031adecc5cb7414395c6708e8d6fb12dcf6f8d8 Author: Kai Zheng dran...@apache.org Date: Sun Feb 8 01:40:27 2015 +0800 HADOOP-11541. Raw XOR coder Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-3619) DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one of the interfaces is IPv6
[
https://issues.apache.org/jira/browse/HADOOP-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310701#comment-14310701
]
Hadoop QA commented on HADOOP-3619:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12655523/HADOOP-3619-v2.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5626//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5626//console
This message is automatically generated.
DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one
of the interfaces is IPv6
--
Key: HADOOP-3619
URL: https://issues.apache.org/jira/browse/HADOOP-3619
Project: Hadoop Common
Issue Type: Bug
Components: net
Reporter: Steve Loughran
Assignee: Dr. Martin Menzel
Labels: ipv6, patch
Attachments: HADOOP-3619-v2.patch
reverseDNS tries to split a host address string by ., and so fails if :
is the separator, as it is in IPv6. When it tries to access the parts of the
address, a stack trace is seen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310668#comment-14310668 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/831/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310671#comment-14310671 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Yarn-trunk #831 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/831/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-3619) DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one of the interfaces is IPv6
[ https://issues.apache.org/jira/browse/HADOOP-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-3619: --- Status: Patch Available (was: Open) submitting to see current patch state. DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one of the interfaces is IPv6 -- Key: HADOOP-3619 URL: https://issues.apache.org/jira/browse/HADOOP-3619 Project: Hadoop Common Issue Type: Bug Components: net Reporter: Steve Loughran Assignee: Dr. Martin Menzel Labels: ipv6, patch Attachments: HADOOP-3619-v2.patch reverseDNS tries to split a host address string by ., and so fails if : is the separator, as it is in IPv6. When it tries to access the parts of the address, a stack trace is seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11293: Status: Open (was: Patch Available) Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Affects Versions: 2.7.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11556) SwiftNativeFileSystem rename being interrupted during fs -put
[
https://issues.apache.org/jira/browse/HADOOP-11556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310682#comment-14310682
]
Steve Loughran commented on HADOOP-11556:
-
the partitioned bit of swift is something that is best described as
troublesome. It appears to violate some of the core semantics of a filesystem
(size of a file in a directory listing == size of file in stat() == amount of
data you can read), and things like rename get very confused.
Declaring swiftfs an object store with expensive renames via HADOOP-9565, along
with your fs -put improvements could somewhat improve this situation.
I believe that swift now supports 5GB files...adding support for that would
also help
SwiftNativeFileSystem rename being interrupted during fs -put
---
Key: HADOOP-11556
URL: https://issues.apache.org/jira/browse/HADOOP-11556
Project: Hadoop Common
Issue Type: Bug
Components: fs, fs/swift
Affects Versions: 2.6.0
Reporter: Lei (Eddy) Xu
When running {{hadoop fs -put bar swift://... }}, {{FsShell}} writes to
{{bar._COPYING_}} first, and then renames it to bar.
However, it fails when the file is partitioned uploaded.
{code}
15/02/06 21:00:49 DEBUG snative.SwiftNativeFileSystemStore: Source file
appears to be partitioned. copying file and deleting children
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.method.retry-handler =
org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@26bf4e60
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.connection.timeout = 15000
15/02/06 21:00:49 DEBUG params.DefaultHttpParams: Set parameter
http.socket.timeout = 6
15/02/06 21:00:49 DEBUG http.SwiftRestClient: COPY
https://snet-storage101.iad3.clouddrive.com/v1/MossoCloudFS_946265/foo/bar._COPYING_
X-Auth-Token: ...
Destination: foo/bar
User-Agent: Apache Hadoop Swift Client 3.0.0-SNAPSHOT from ... by root source
checksum ...
15/02/06 21:00:49 DEBUG httpclient.HttpConnection: Open connection to
snet-storage101.iad3.clouddrive.com:443
15/02/06 21:00:49 DEBUG wire.header: COPY
/v1/MossoCloudFS_946265/foo/bar._COPYING_ HTTP/1.1[\r][\n]
15/02/06 21:00:49 DEBUG httpclient.HttpMethodBase: Adding Host request header
15/02/06 21:00:49 DEBUG wire.header: X-Auth-Token: ...[\r][\n]
15/02/06 21:00:49 DEBUG wire.header: Destination: foo/bar[\r][\n]
15/02/06 21:00:49 DEBUG wire.header: User-Agent: Apache Hadoop Swift
Client 3.0.0-SNAPSHOT from ... by root source checksum ...[\r][\n]
15/02/06 21:00:49 DEBUG wire.header: Host:
snet-storage101.iad3.clouddrive.com[\r][\n]
15/02/06 21:00:49 DEBUG wire.header: Content-Length: 0[\r][\n]
15/02/06 21:00:49 DEBUG wire.header: [\r][\n]
15/02/06 21:00:49 DEBUG methods.EntityEnclosingMethod: Request body has not
been specified
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Closing the connection.
15/02/06 21:01:49 DEBUG httpclient.HttpMethodDirector: Method retry handler
returned false. Automatic recovery will not be attempted
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back
to connection manager.
15/02/06 21:01:49 DEBUG httpclient.HttpConnection: Releasing connection back
to connection manager.
{code}
The {{bar}} file eventually shows up in Swift. However, the {{bar._COPYING_}}
directories and the manifest file are not deleted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HADOOP-11561) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe moved MAPREDUCE-6218 to HADOOP-11561: --- Key: HADOOP-11561 (was: MAPREDUCE-6218) Project: Hadoop Common (was: Hadoop Map/Reduce) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order --- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Priority: Minor Labels: mapfile Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many MapFiles which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple MapFiles. That means, there should be a reader which can be supplied with one or more directories containing MapFiles, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11561) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310688#comment-14310688 ] Jens Rabe commented on HADOOP-11561: I posted this in the wrong category, as MapReduce already has a CompositeInputFormat which does exactly this task. With my feature request I wanted to address client functionality. It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order --- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Priority: Minor Labels: mapfile Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many MapFiles which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple MapFiles. That means, there should be a reader which can be supplied with one or more directories containing MapFiles, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-11561) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe reassigned HADOOP-11561: -- Assignee: Jens Rabe It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order --- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: mapfile Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many MapFiles which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple MapFiles. That means, there should be a reader which can be supplied with one or more directories containing MapFiles, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11293: Status: Patch Available (was: Open) Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Affects Versions: 2.7.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11541) Raw XOR coder
[ https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11541: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Raw XOR coder - Key: HADOOP-11541 URL: https://issues.apache.org/jira/browse/HADOOP-11541 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-EC Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch This will implement XOR codes by porting the codes from HDFS-RAID. The coder in the algorithm is needed by some high level codecs like LRC. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310664#comment-14310664 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * hadoop-common-project/hadoop-common/CHANGES.txt Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310661#comment-14310661 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/97/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Attachment: HADOOP-11561.patch It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order --- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: mapfile Attachments: HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many MapFiles which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple MapFiles. That means, there should be a reader which can be supplied with one or more directories containing MapFiles, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310727#comment-14310727 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310732#comment-14310732 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310735#comment-14310735 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #94 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/94/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * hadoop-common-project/hadoop-common/CHANGES.txt Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Summary: Join multiple files on the fly and read the records in order with a client (was: Join multiple files on the fly and read the records in order) Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11561) Join multiple files on the fly and read the records in order
[
https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310738#comment-14310738
]
Hadoop QA commented on HADOOP-11561:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697257/HADOOP-11561.patch
against trunk revision cfb829e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5627//console
This message is automatically generated.
Join multiple files on the fly and read the records in order
Key: HADOOP-11561
URL: https://issues.apache.org/jira/browse/HADOOP-11561
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jens Rabe
Assignee: Jens Rabe
Priority: Minor
Labels: composite
Attachments: HADOOP-11561.patch
Original Estimate: 96h
Remaining Estimate: 96h
In a scenario where there are many files which all share the same key/value
types, e.g., when dealing with measured data from sensors, it should be
possible to chain-load multiple files. That means, there should be a reader
which can be supplied with one or more directories containing files, and it
should be possible to read the records of all files in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Description: In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. (was: In a scenario where there are many MapFiles which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple MapFiles. That means, there should be a reader which can be supplied with one or more directories containing MapFiles, and it should be possible to read the records of all files in order.) Summary: Join multiple files on the fly and read the records in order (was: It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order) Join multiple files on the fly and read the records in order Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[
https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310762#comment-14310762
]
Hadoop QA commented on HADOOP-11561:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697259/HADOOP-11561.002.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5628//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5628//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5628//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5628//console
This message is automatically generated.
Join multiple files on the fly and read the records in order with a client
--
Key: HADOOP-11561
URL: https://issues.apache.org/jira/browse/HADOOP-11561
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jens Rabe
Assignee: Jens Rabe
Priority: Minor
Labels: composite
Attachments: HADOOP-11561.002.patch, HADOOP-11561.patch
Original Estimate: 96h
Remaining Estimate: 96h
In a scenario where there are many files which all share the same key/value
types, e.g., when dealing with measured data from sensors, it should be
possible to chain-load multiple files. That means, there should be a reader
which can be supplied with one or more directories containing files, and it
should be possible to read the records of all files in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Status: In Progress (was: Patch Available) FindBugs warnings are, again, unrelated, release audit warning will be fixed now. Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) It should be possible to chain-load multiple MapFiles on the fly and read the records in an ascending order
[
https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Rabe updated HADOOP-11561:
---
Labels: composite (was: mapfile)
Status: Patch Available (was: Open)
See [^HADOOP-11561.patch]. I added an inner class Reader to the
CompositeInputFormat which can be used to read and join multiple files on the
fly when reading data in a client application.
The same constraints as for the {{CompositeInputFormat}} apply as this reader
uses the format internally. To use the reader, do the following:
# Make sure all input files have the same key and value classes
# Make sure all records in the input files are sorted by the same sorting
criterion
# Prepare a {{Configuration}} object with at least {{mapreduce.join.expr}} and
{{mapreduce.join.comparator}} set. Consult {{CompositeInputFormat}} for details.
# Use the constructor {{CompositeInputFormat.Reader(Configuration)}} to
instantiate the reader
# Use its {{nextKeyValue}} to read a record and store it in the writables you
supplied, just like it is done with the {{SequenceFile}} and {{MapFile}}
readers.
It should be possible to chain-load multiple MapFiles on the fly and read the
records in an ascending order
---
Key: HADOOP-11561
URL: https://issues.apache.org/jira/browse/HADOOP-11561
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jens Rabe
Assignee: Jens Rabe
Priority: Minor
Labels: composite
Attachments: HADOOP-11561.patch
Original Estimate: 96h
Remaining Estimate: 96h
In a scenario where there are many MapFiles which all share the same
key/value types, e.g., when dealing with measured data from sensors, it
should be possible to chain-load multiple MapFiles. That means, there should
be a reader which can be supplied with one or more directories containing
MapFiles, and it should be possible to read the records of all files in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Status: Open (was: Patch Available) Outputted the patch the wrong way, will add another Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Status: Patch Available (was: In Progress) [^HADOOP-11561.003.patch] adds the Apache License header to the Test class. The findbugs warnings are unrelated. Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.003.patch, HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Attachment: HADOOP-11561.003.patch Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.003.patch, HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310773#comment-14310773 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310724#comment-14310724 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2029 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2029/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Status: Patch Available (was: Open) Outputted the patch again with the correct settings as [^HADOOP-11561.002.patch] Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Attachment: HADOOP-11561.002.patch Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.patch Original Estimate: 96h Remaining Estimate: 96h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310756#comment-14310756 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs * hadoop-common-project/hadoop-common/CHANGES.txt Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310759#comment-14310759 ] Hudson commented on HADOOP-11520: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #98 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/98/]) HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel) (stevel: rev 17165d3df9cc5b4a4f25946ae215f960ab62658e) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * hadoop-common-project/hadoop-common/CHANGES.txt Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310770#comment-14310770 ] Hudson commented on HADOOP-11485: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2048/]) HADOOP-11485. Pluggable shell integration (aw) (aw: rev 5c79439568ff0c73062cf09d87f1e739703c7dc0) * hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh * hadoop-assemblies/src/main/resources/assemblies/hadoop-yarn-dist.xml * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-assemblies/src/main/resources/assemblies/hadoop-dist.xml * hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh * hadoop-yarn-project/hadoop-yarn/shellprofile.d/yarn * hadoop-common-project/hadoop-common/src/main/conf/shellprofile.d/example * hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml * hadoop-yarn-project/hadoop-yarn/bin/yarn * hadoop-mapreduce-project/shellprofile.d/mapreduce * hadoop-hdfs-project/hadoop-hdfs/src/main/shellprofile.d/hdfs Pluggable shell integration --- Key: HADOOP-11485 URL: https://issues.apache.org/jira/browse/HADOOP-11485 Project: Hadoop Common Issue Type: New Feature Components: scripts Affects Versions: 3.0.0 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Labels: scripts, shell Fix For: 3.0.0 Attachments: HADOOP-11485-00.patch, HADOOP-11485-01.patch, HADOOP-11485-02.patch, HADOOP-11485-03.patch, HADOOP-11485-04.patch It would be useful to provide a way for core and non-core Hadoop components to plug into the shell infrastructure. This would allow us to pull the HDFS, MapReduce, and YARN shell functions out of hadoop-functions.sh. Additionally, it should let 3rd parties such as HBase influence things like classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work logged] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?focusedWorklogId=19157page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-19157 ] Jens Rabe logged work on HADOOP-11561: -- Author: Jens Rabe Created on: 07/Feb/15 15:25 Start Date: 07/Feb/15 14:30 Worklog Time Spent: 1h Issue Time Tracking --- Worklog Id: (was: 19157) Time Spent: 1h Remaining Estimate: 0h (was: 96h) Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.003.patch, HADOOP-11561.patch Original Estimate: 96h Time Spent: 1h Remaining Estimate: 0h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[ https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jens Rabe updated HADOOP-11561: --- Description: In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to join multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. (was: In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to chain-load multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order.) Join multiple files on the fly and read the records in order with a client -- Key: HADOOP-11561 URL: https://issues.apache.org/jira/browse/HADOOP-11561 Project: Hadoop Common Issue Type: Improvement Reporter: Jens Rabe Assignee: Jens Rabe Priority: Minor Labels: composite Attachments: HADOOP-11561.002.patch, HADOOP-11561.003.patch, HADOOP-11561.patch Original Estimate: 96h Time Spent: 1h Remaining Estimate: 0h In a scenario where there are many files which all share the same key/value types, e.g., when dealing with measured data from sensors, it should be possible to join multiple files. That means, there should be a reader which can be supplied with one or more directories containing files, and it should be possible to read the records of all files in order. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11561) Join multiple files on the fly and read the records in order with a client
[
https://issues.apache.org/jira/browse/HADOOP-11561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310786#comment-14310786
]
Hadoop QA commented on HADOOP-11561:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12697260/HADOOP-11561.003.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5629//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5629//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5629//console
This message is automatically generated.
Join multiple files on the fly and read the records in order with a client
--
Key: HADOOP-11561
URL: https://issues.apache.org/jira/browse/HADOOP-11561
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jens Rabe
Assignee: Jens Rabe
Priority: Minor
Labels: composite
Attachments: HADOOP-11561.002.patch, HADOOP-11561.003.patch,
HADOOP-11561.patch
Original Estimate: 96h
Time Spent: 1h
Remaining Estimate: 0h
In a scenario where there are many files which all share the same key/value
types, e.g., when dealing with measured data from sensors, it should be
possible to join multiple files. That means, there should be a reader which
can be supplied with one or more directories containing files, and it should
be possible to read the records of all files in order.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11562) Improvement for ConverterUtils
Bibin A Chundatt created HADOOP-11562:
-
Summary: Improvement for ConverterUtils
Key: HADOOP-11562
URL: https://issues.apache.org/jira/browse/HADOOP-11562
Project: Hadoop Common
Issue Type: Improvement
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11562) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/HADOOP-11562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated HADOOP-11562:
--
Affects Version/s: 2.6.0
Improvement for ConverterUtils
---
Key: HADOOP-11562
URL: https://issues.apache.org/jira/browse/HADOOP-11562
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11562) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/HADOOP-11562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated HADOOP-11562:
--
Status: Patch Available (was: Open)
Added check for application id and application attempt
Improvement for ConverterUtils
---
Key: HADOOP-11562
URL: https://issues.apache.org/jira/browse/HADOOP-11562
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: HADOOP-11562.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11560) Unable to compile Source code
[ https://issues.apache.org/jira/browse/HADOOP-11560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310793#comment-14310793 ] Gopi commented on HADOOP-11560: --- Thats correct I am trying to build Hadoop on Windows 7 (win32). I went thru HADOOP-11425 but I dont see a clear direction. Need to l know what else is to be setup so that I can complete build and start working on actual implementation. I am investing lot of time in building hadoop itself on window, I still have not started my actual implementation work. Please help. On Sat, Feb 7, 2015 at 1:36 AM, Brahma Reddy Battula (JIRA) j...@apache.org Unable to compile Source code - Key: HADOOP-11560 URL: https://issues.apache.org/jira/browse/HADOOP-11560 Project: Hadoop Common Issue Type: Bug Components: build Environment: Windows 7 Professional Reporter: Gopi I have done the Setup for all of the below: Hadoop source: C:\Practice\Hadoop\hadoop-2.6.0-src * Windows System 7 Professional * JDK 1.6 * Apache Maven 3.2.5 * ProtocolBuffer 2.5.0 (protoc-2.5.0-win32.zip [protoc.exe]) * CMake (cmake-3.1.2-win32-x86.exe) * Windows SDK 7.1 * Unix command-line tools installed Cygwin I am getting the below build error == [INFO] BUILD FAILURE [INFO] [INFO] Total time: 44.521 s [INFO] Finished at: 2015-02-07T00:35:14-05:00 [INFO] Final Memory: 54M/129M [INFO] [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216) at org.apache.maven.cli.MavenCli.main(MavenCli.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed. at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:345) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 19 more Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:589) at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:335) ... 21 more [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1]
[jira] [Updated] (HADOOP-11562) Improvement for ConverterUtils
[
https://issues.apache.org/jira/browse/HADOOP-11562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated HADOOP-11562:
--
Attachment: HADOOP-11562.patch
Improvement for ConverterUtils
---
Key: HADOOP-11562
URL: https://issues.apache.org/jira/browse/HADOOP-11562
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.6.0
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Minor
Attachments: HADOOP-11562.patch
yarn.cmd application -kill application_123
Format wrong given for application id or attempt. Exception will be thrown to
console with out any info
{quote}
15/02/07 22:18:01 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where
Exception in thread main java.util.NoSuchElementException
at
com.google.common.base.AbstractIterator.next(AbstractIterator.java:75)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:146)
at
org.apache.hadoop.yarn.util.ConverterUtils.toApplicationId(ConverterUtils.java:205)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.killApplication(ApplicationCLI.java:383)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:219)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{quote}
Need to add catch block for java.util.NoSuchElementException also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11560) Unable to compile Source code
[ https://issues.apache.org/jira/browse/HADOOP-11560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310838#comment-14310838 ] Brahma Reddy Battula commented on HADOOP-11560: --- Hello [~talk2gopijj], AFAIK, Directly we can't compile hadoop code in win32, you may need do some extra steps like following 1) Apply HADOOP-9922 patch. 2) install winsdk8.0 and visual studio-2013. I am not pretty sure,you can try.. In the future please use a mailing list. JIRA is for reporting a bug. Mailing list URL : http://hadoop.apache.org/mailing_lists.html Unable to compile Source code - Key: HADOOP-11560 URL: https://issues.apache.org/jira/browse/HADOOP-11560 Project: Hadoop Common Issue Type: Bug Components: build Environment: Windows 7 Professional Reporter: Gopi I have done the Setup for all of the below: Hadoop source: C:\Practice\Hadoop\hadoop-2.6.0-src * Windows System 7 Professional * JDK 1.6 * Apache Maven 3.2.5 * ProtocolBuffer 2.5.0 (protoc-2.5.0-win32.zip [protoc.exe]) * CMake (cmake-3.1.2-win32-x86.exe) * Windows SDK 7.1 * Unix command-line tools installed Cygwin I am getting the below build error == [INFO] BUILD FAILURE [INFO] [INFO] Total time: 44.521 s [INFO] Finished at: 2015-02-07T00:35:14-05:00 [INFO] Final Memory: 54M/129M [INFO] [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1(Exit value: 1) - [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project hadoop-common: Command execution failed. at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216) at org.apache.maven.cli.MavenCli.main(MavenCli.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: Command execution failed. at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:345) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 19 more Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 1(Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:346) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:149) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:589) at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:335) ... 21 more [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR]
[jira] [Commented] (HADOOP-3619) DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one of the interfaces is IPv6
[
https://issues.apache.org/jira/browse/HADOOP-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310878#comment-14310878
]
Steve Loughran commented on HADOOP-3619:
-1 as is, though with a couple of iterations it should be ready to go in.
h3. production code
# no need to explicitly use {{java.net.Inet4Address}}
{{java.net.Inet6Address}}; they are imported can be used directly.
# recommend: the naming exception to include a link to new hadoop wiki entry
(ideally https://wiki.apache.org/hadoop/DNS) which we can expand to cover DNS
issues for those users who don't know anything about them.
# the new ipv4/ipv6 methods should be private.
# {{getNibbleFromAddress()}} should be package scoped, tagged
{{@VisibleForTesting}}, and have a test which that the example result comes
from the example input. Given that there's no way to create a stub IPv6
address, the bytes-string generation logic could be pulled out and tested
standalone.
h3. Test case
# {{assertEquals()}} comparators are inverted: expected value comes first.
# We can't have tests that expect the external network to be there, not even
DNS. Is there a way to test this such that only localhost DNS lookup is needed?
# what happens to these tests on machines without IPv6 support? It's not
directly a test failure, just a machine/network configuration. The test needs
to recognise and handle this situation.
General
* hadoop style guide is 2 chars, no tabs, amongst other things.
see
https://github.com/steveloughran/formality/blob/master/styleguide/styleguide.md
for a side project of mine to document what is generally expected.
DNS.getHosts triggers an ArrayIndexOutOfBoundsException in reverseDNS if one
of the interfaces is IPv6
--
Key: HADOOP-3619
URL: https://issues.apache.org/jira/browse/HADOOP-3619
Project: Hadoop Common
Issue Type: Bug
Components: net
Reporter: Steve Loughran
Assignee: Dr. Martin Menzel
Labels: ipv6, patch
Attachments: HADOOP-3619-v2.patch
reverseDNS tries to split a host address string by ., and so fails if :
is the separator, as it is in IPv6. When it tries to access the parts of the
address, a stack trace is seen.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11042) CryptoInputStream throwing wrong exception class on errors
[
https://issues.apache.org/jira/browse/HADOOP-11042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310890#comment-14310890
]
Steve Loughran commented on HADOOP-11042:
-
-sorry for not replying about this; I think I'd missed the patch in my inbox.
Looks good. I was a bit worried about the refactoring but it makes sense -and
the way the contract tests are designed, FS-specific subclasses will pick up
the changes without any changes needed to their code (I'll do a test run of the
s3 and openstack clients just to make sure there before it gets its +1.
# can you switch to using FSExceptionMessages constants for your exception
text? With the same text everywhere we can simplify documentation and maybe
even add wiki links in future.
# testRenameFileBeingAppended()` tries to rename a file that is being written
to. The patched version doesn't. (to be fair, the expected outcome of that
operation isn't defined AFAIK). It may be easiest to leave that as is and in
the subclass, just skip it (ie {{Assume.assumeTrue(false)}})
CryptoInputStream throwing wrong exception class on errors
--
Key: HADOOP-11042
URL: https://issues.apache.org/jira/browse/HADOOP-11042
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Steve Loughran
Assignee: Yi Liu
Attachments: HADOOP-11042.001.patch
Having had a quick look at the {{CryptoInputStream}} class, it's not in sync
with all the other filesystem's exception logic, as specified in
{{src/site/markdown/filesystem/fsdatainputstream.md}}
Operations MUST throw an {{IOException}} on out of bounds reads, ideally
{{EOFException}} :
# {{read(byte[] b, int off, int len)}}
# {{seek(long pos) }}
# {{seekToNewSource}}
The tests you want to extend to verify expected behaviour are in
{{AbstractContractOpenTest}} and {{AbstractContractSeekTest}}
also, the {{HasEnhancedByteBufferAccess}} implementations may want to think
about using {{checkStream()}} before acting on a potentially closed stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11042) CryptoInputStream throwing wrong exception class on errors
[
https://issues.apache.org/jira/browse/HADOOP-11042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310898#comment-14310898
]
Hadoop QA commented on HADOOP-11042:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665868/HADOOP-11042.001.patch
against trunk revision cfb829e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 17 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5630//console
This message is automatically generated.
CryptoInputStream throwing wrong exception class on errors
--
Key: HADOOP-11042
URL: https://issues.apache.org/jira/browse/HADOOP-11042
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Steve Loughran
Assignee: Yi Liu
Attachments: HADOOP-11042.001.patch
Having had a quick look at the {{CryptoInputStream}} class, it's not in sync
with all the other filesystem's exception logic, as specified in
{{src/site/markdown/filesystem/fsdatainputstream.md}}
Operations MUST throw an {{IOException}} on out of bounds reads, ideally
{{EOFException}} :
# {{read(byte[] b, int off, int len)}}
# {{seek(long pos) }}
# {{seekToNewSource}}
The tests you want to extend to verify expected behaviour are in
{{AbstractContractOpenTest}} and {{AbstractContractSeekTest}}
also, the {{HasEnhancedByteBufferAccess}} implementations may want to think
about using {{checkStream()}} before acting on a potentially closed stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311070#comment-14311070
]
Yi Liu commented on HADOOP-11541:
-
Kai, another thing for the committing.
For the log, I see you only write:
{quote}
HADOOP-11541. Raw XOR coder
{quote}
You should also write the contribution name, so it should by something like:
{quote}
HADOOP-11541. Raw XOR coder. (Contributed by Kai Zheng)
{quote}
Also you should change the corresponding {{CHANGES.txt}}, we should have a
separate {{CHANGES.txt}} file for the HDFS-EC branch and you should update it.
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311070#comment-14311070
]
Yi Liu edited comment on HADOOP-11541 at 2/8/15 3:15 AM:
-
Kai, another thing for the committing.
For the log:
{quote}
HADOOP-11541. Raw XOR coder
{quote}
You should also write the contribution name, so it should by something like:
{quote}
HADOOP-11541. Raw XOR coder. (Contributed by Kai Zheng)
{quote}
Also you should change the corresponding {{CHANGES.txt}}, we should have a
separate {{CHANGES.txt}} file for the HDFS-EC branch and you should update it.
was (Author: hitliuyi):
Kai, another thing for the committing.
For the log, I see you only write:
{quote}
HADOOP-11541. Raw XOR coder
{quote}
You should also write the contribution name, so it should by something like:
{quote}
HADOOP-11541. Raw XOR coder. (Contributed by Kai Zheng)
{quote}
Also you should change the corresponding {{CHANGES.txt}}, we should have a
separate {{CHANGES.txt}} file for the HDFS-EC branch and you should update it.
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11042) CryptoInputStream throwing wrong exception class on errors
[
https://issues.apache.org/jira/browse/HADOOP-11042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311092#comment-14311092
]
Yi Liu commented on HADOOP-11042:
-
Thanks Steve for taking a look, will rebase/update the patch.
CryptoInputStream throwing wrong exception class on errors
--
Key: HADOOP-11042
URL: https://issues.apache.org/jira/browse/HADOOP-11042
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Steve Loughran
Assignee: Yi Liu
Attachments: HADOOP-11042.001.patch
Having had a quick look at the {{CryptoInputStream}} class, it's not in sync
with all the other filesystem's exception logic, as specified in
{{src/site/markdown/filesystem/fsdatainputstream.md}}
Operations MUST throw an {{IOException}} on out of bounds reads, ideally
{{EOFException}} :
# {{read(byte[] b, int off, int len)}}
# {{seek(long pos) }}
# {{seekToNewSource}}
The tests you want to extend to verify expected behaviour are in
{{AbstractContractOpenTest}} and {{AbstractContractSeekTest}}
also, the {{HasEnhancedByteBufferAccess}} implementations may want to think
about using {{checkStream()}} before acting on a potentially closed stream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11541) Raw XOR coder
[
https://issues.apache.org/jira/browse/HADOOP-11541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14311068#comment-14311068
]
Yi Liu commented on HADOOP-11541:
-
{quote}
you need a +1 from someone else (not yourself).
OK. It clarifies the process for me, thanks. I thought it has been relaxed
somehow for the branch but not sure how I got the impression.
{quote}
let's stick to the review rule and keep the same quality as developing in
trunk, even we are in a branch.
Raw XOR coder
-
Key: HADOOP-11541
URL: https://issues.apache.org/jira/browse/HADOOP-11541
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Kai Zheng
Fix For: HDFS-EC
Attachments: HADOOP-11541-v1.patch, HADOOP-11541-v2.patch
This will implement XOR codes by porting the codes from HDFS-RAID. The coder
in the algorithm is needed by some high level codecs like LRC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
