[jira] [Updated] (HADOOP-9016) Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
[ https://issues.apache.org/jira/browse/HADOOP-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9016: --- Attachment: HADOOP-9016--b.patch Improved version of the patch, which, actually includes all the changes done in the file TestHadoopArchives.java by YARN-204. Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream - Key: HADOOP-9016 URL: https://issues.apache.org/jira/browse/HADOOP-9016 Project: Hadoop Common Issue Type: Improvement Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9016--b.patch, HADOOP-9016.patch unit-test coverage of classes org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream, org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream.HarFsInputStream is zero. Suggested to provide unit-tests covering these classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9016) Provide unit tests for class org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
[
https://issues.apache.org/jira/browse/HADOOP-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495246#comment-13495246
]
Hadoop QA commented on HADOOP-9016:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553110/HADOOP-9016--b.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-archives.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1730//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1730//console
This message is automatically generated.
Provide unit tests for class
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream
-
Key: HADOOP-9016
URL: https://issues.apache.org/jira/browse/HADOOP-9016
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
Attachments: HADOOP-9016--b.patch, HADOOP-9016.patch
unit-test coverage of classes
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream,
org.apache.hadoop.fs.HarFileSystem.HarFSDataInputStream.HarFsInputStream is
zero.
Suggested to provide unit-tests covering these classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Target Version/s: 2.0.3-alpha (was: 2.0.2-alpha) Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed I just committed this to branch-2 (using the original script). Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HADOOP-8860: -- Attachment: HADOOP-8860-23.patch HADOOP-8860-23.sh The changes don't apply to the 0.23 branch either, so I created equivalents for that branch. Bobby, please let me know if you'd like me to commit them there too. Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: HADOOP-8860-23.patch, HADOOP-8860-23.sh, HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8860) Split MapReduce and YARN sections in documentation navigation
[ https://issues.apache.org/jira/browse/HADOOP-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495278#comment-13495278 ] Hudson commented on HADOOP-8860: Integrated in Hadoop-trunk-Commit #2998 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2998/]) Add new files missed from HADOOP-8860 commit. (Revision 1408288) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1408288 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/resources * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/resources/css * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/resources/css/site.css * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/site.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/resources/css * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/resources/css/site.css * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/css * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/css/site.css * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/site.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/resources * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/resources/css * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/resources/css/site.css * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/site.xml Split MapReduce and YARN sections in documentation navigation - Key: HADOOP-8860 URL: https://issues.apache.org/jira/browse/HADOOP-8860 Project: Hadoop Common Issue Type: Task Components: documentation Affects Versions: 2.0.1-alpha Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: HADOOP-8860-23.patch, HADOOP-8860-23.sh, HADOOP-8860.patch, HADOOP-8860.patch, HADOOP-8860.sh, HADOOP-8860.sh This JIRA is to change the navigation on http://hadoop.apache.org/docs/r2.0.1-alpha/ to reflect the fact that MapReduce and YARN are separate modules/sub-projects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8977) multiple FsShell test failures on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495372#comment-13495372 ] Daryn Sharp commented on HADOOP-8977: - Here's a copy-n-paste from an offline discussion: The problem here is hadoop is designed around URIs and we are trying to support something (windows paths) that is NOT a URI. Earlier, I bent over backwards trying to temporarily handle windows paths but Suresh convinced me it was the wrong direction. The article I linked is about how to properly reference windows paths as URIs and says windows style \ paths are deprecated in IE which I think essentially means the file browser. The windows shell supports / paths so I'm grappling with why we should perpetuate deprecated windows paths as pseudo-URIs when real URIs appear to be fully supported in windows. I'd be a bit happier ( or less unhappy! :) )if \ support is more context specific to just windows local path names. As it stands, all URIs on windows are subject to \ to / conversion which prevents windows from accessing valid filenames in hdfs and other supported filesystems. I can understand/sympathize with the motivation to support c:\path, but I don't agree that hdfs:\\host\path, or hdfs:\/path/path2\path3 should be supported at all. This bizarre behavior creates compatibility issues where jobs accessing paths in that way are not cross-platform compatible.Ie. They work on hadoop for windows, but fail on every other OS. Once we let the cat of of the bag by adding more pseudo-support for non-URIs on windows, it's going to be that much harder to take it away. What if we did something a bit more selective: # [a-z]:\ considered a windows non-URI #* implicitly deemed to have a file scheme if not already declared #* all \ are converted to / - which means no quoting of metachars available, or we support ^ as the escape #* throw an exception if / already exists in the path # [a-z]:/ #* considered a standard URI #* implicitly deemed to have a file scheme if not already declared #* no \ conversion - quoting of metachars is supported # all other URI schemes and relative paths #* no change # add ctor Path(File) #* allow users to create Paths from non-URIs #* will eventually be the only supported way to access non-URI paths # eliminate treating : as an invalid path character to allow drive letters I'm curious what serious breakage we'll have if we just require standard URIs - ie. change little to nothing or implement the above proposal? multiple FsShell test failures on Windows - Key: HADOOP-8977 URL: https://issues.apache.org/jira/browse/HADOOP-8977 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: trunk-win Attachments: HADOOP-8977-branch-trunk-win.patch, HADOOP-8977-branch-trunk-win.patch, HADOOP-8977-branch-trunk-win.patch, HADOOP-8977.patch Multiple FsShell-related tests fail on Windows. Commands are returning non-zero exit status. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9024) Unit Test: TestKerberosAuthenticator fails when tested with ant-1.8.4
Amir Sanjar created HADOOP-9024:
---
Summary: Unit Test: TestKerberosAuthenticator fails when tested
with ant-1.8.4
Key: HADOOP-9024
URL: https://issues.apache.org/jira/browse/HADOOP-9024
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 1.0.4, 1.0.3
Environment: Linux Fedora 17_64
Reporter: Amir Sanjar
Priority: Minor
Fix For: 1.0.4, 1.0.3
Problem:
JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not JUnit4:
Solution:
Migrate the testcase to JUnit4
How:
Remove extends TestCase
SetUp and TearDown methods
@Override
protected void setUp() throws Exception { }
replaced by:
@Before
public void setUp() throws Exception { }
Same for tearDown():
@Override
protected void tearDown() throws Exception { }
replaced by
@After
public void tearDown() throws Exception { }
Imports
The imports has to be reorganized:
Remove import junit.framework.TestCase;
Add org.junit.*; or import org.junit.After; import org.junit.Before;
import org.junit.Test;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9024) Unit Test: TestKerberosAuthenticator fails when tested with ant-1.8.4
[
https://issues.apache.org/jira/browse/HADOOP-9024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495468#comment-13495468
]
Amir Sanjar commented on HADOOP-9024:
-
TestKerberosAuthenticator TestPseudoAuthenticator testcase classes extends
AuthenticatorTestCase class.
SO we first need to migrate TestPseudoAuthenticator class to JUnit4 and then
port classes TestKerberosAuthenticator TestPseudoAuthenticator
Unit Test: TestKerberosAuthenticator fails when tested with ant-1.8.4
--
Key: HADOOP-9024
URL: https://issues.apache.org/jira/browse/HADOOP-9024
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 1.0.3, 1.0.4
Environment: Linux Fedora 17_64
Reporter: Amir Sanjar
Priority: Minor
Fix For: 1.0.3, 1.0.4
Problem:
JUnit tag @Ignore is not recognized since the testcase is JUnit3 and not
JUnit4:
Solution:
Migrate the testcase to JUnit4
How:
Remove extends TestCase
SetUp and TearDown methods
@Override
protected void setUp() throws Exception { }
replaced by:
@Before
public void setUp() throws Exception { }
Same for tearDown():
@Override
protected void tearDown() throws Exception { }
replaced by
@After
public void tearDown() throws Exception { }
Imports
The imports has to be reorganized:
Remove import junit.framework.TestCase;
Add org.junit.*; or import org.junit.After; import org.junit.Before;
import org.junit.Test;
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9011) saveVersion.py does not include branch in version annotation
[ https://issues.apache.org/jira/browse/HADOOP-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495536#comment-13495536 ] Chris Nauroth commented on HADOOP-9011: --- Thank you, Raja. While working on this, I noticed that the name current_branch was inconsistent with the rest of the code. For all other substitutions into the annotation, the script variable name matches the annotation variable name (version, revision, user, etc.). To make the code more consistent, I renamed current_branch to branch. This didn't seem significant enough to warrant splitting it into a separate jira. saveVersion.py does not include branch in version annotation Key: HADOOP-9011 URL: https://issues.apache.org/jira/browse/HADOOP-9011 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Chris Nauroth Attachments: HADOOP-9011-branch-trunk-win.patch HADOOP-8924 created saveVersion.py on branch-trunk-win. Unlike saveVersion.sh on trunk, it did not include the branch attribute in the version annotation. This causes errors at runtime for anything that tries to read the annotation via VersionInfo. This also causes a unit test failure in TestYarnVersionInfo. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9017) fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version
[ https://issues.apache.org/jira/browse/HADOOP-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495560#comment-13495560 ] Matt Foley commented on HADOOP-9017: +1. Please commit to branch-1 and branch-1.1. Thanks, Giri! fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version -- Key: HADOOP-9017 URL: https://issues.apache.org/jira/browse/HADOOP-9017 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.0.4, 1.1.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: HADOOP-9017.patch hadoop-client-pom-template.xml and hadoop-client-pom-template.xml references to project.version variable, instead they should refer to @version token. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9005) Merge hadoop cmd line scripts from branch-1-win
[ https://issues.apache.org/jira/browse/HADOOP-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9005: Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 Thank you Ivan. I committed the patch to branch-trunk-win. Merge hadoop cmd line scripts from branch-1-win --- Key: HADOOP-9005 URL: https://issues.apache.org/jira/browse/HADOOP-9005 Project: Hadoop Common Issue Type: Bug Affects Versions: trunk-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: trunk-win Attachments: HADOOP-9005.branch-trunk-win.scripts.2.patch, HADOOP-9005.branch-trunk-win.scripts.3.patch, HADOOP-9005.branch-trunk-win.scripts.4.patch, HADOOP-9005.branch-trunk-win.scripts.5.patch, HADOOP-9005.branch-trunk-win.scripts.6.patch, HADOOP-9005.branch-trunk-win.scripts.7.patch, HADOOP-9005.branch-trunk-win.scripts.patch Tracking Jira for merging hadoop cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9005) Merge hadoop cmd line scripts from branch-1-win
[ https://issues.apache.org/jira/browse/HADOOP-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495569#comment-13495569 ] Suresh Srinivas commented on HADOOP-9005: - Missed other contributors to this patch. Thank you David Lao, Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic. Merge hadoop cmd line scripts from branch-1-win --- Key: HADOOP-9005 URL: https://issues.apache.org/jira/browse/HADOOP-9005 Project: Hadoop Common Issue Type: Bug Affects Versions: trunk-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: trunk-win Attachments: HADOOP-9005.branch-trunk-win.scripts.2.patch, HADOOP-9005.branch-trunk-win.scripts.3.patch, HADOOP-9005.branch-trunk-win.scripts.4.patch, HADOOP-9005.branch-trunk-win.scripts.5.patch, HADOOP-9005.branch-trunk-win.scripts.6.patch, HADOOP-9005.branch-trunk-win.scripts.7.patch, HADOOP-9005.branch-trunk-win.scripts.patch Tracking Jira for merging hadoop cmd line scripts from branch-1-win to trunk. Scripts also have to be updated to reflect their unix equivalents. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9019) KerberosAuthenticator.doSpnegoSequence(..) should create a HTTP principal with hostname everytime
[
https://issues.apache.org/jira/browse/HADOOP-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495596#comment-13495596
]
Aaron T. Myers commented on HADOOP-9019:
I agree with you, Allen, but I think the thing that Vinay is trying to do is
use principals that contain FQDNs, but put IP addresses in the config files for
the hosts. My understanding is that Vinay is proposing that we make Hadoop
perform the reverse DNS lookup you're referring to.
Vinay: please correct me if I misunderstand your issue or proposal.
KerberosAuthenticator.doSpnegoSequence(..) should create a HTTP principal
with hostname everytime
--
Key: HADOOP-9019
URL: https://issues.apache.org/jira/browse/HADOOP-9019
Project: Hadoop Common
Issue Type: Bug
Reporter: Vinay
in KerberosAuthenticator.doSpnegoSequence(..) following line of code will
just create a principal of the form HTTP/host,
{code}String servicePrincipal =
KerberosUtil.getServicePrincipal(HTTP,
KerberosAuthenticator.this.url.getHost());{code}
but uri.getHost() is not sure of always getting hostname. If uri contains
IP, then it just returns IP.
For SPNEGO authentication principal should always be created with hostname.
This code should be something like this, which will look /etc/hosts to get
hostname
{code}String hostname = InetAddress.getByName(
KerberosAuthenticator.this.url.getHost()).getHostName();
String servicePrincipal = KerberosUtil.getServicePrincipal(HTTP,
hostname);{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9021) Enforce configured SASL method on the server
[ https://issues.apache.org/jira/browse/HADOOP-9021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9021: Attachment: HADOOP-9021.patch More precise logging and exceptions as suggested. Enforce configured SASL method on the server Key: HADOOP-9021 URL: https://issues.apache.org/jira/browse/HADOOP-9021 Project: Hadoop Common Issue Type: Sub-task Components: ipc, security Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-9021.patch, HADOOP-9021.patch The RPC needs to restrict itself to only using the configured SASL method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9008) Building hadoop tarball fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9008:
--
Attachment: HADOOP-9008-branch-trunk-win.patch
Raja, I am attaching an updated patch that incorporates your advice to use
abspath and / where applicable to improve readability. I verified that this
works cross-platform by retesting all build variations.
Building hadoop tarball fails on Windows
Key: HADOOP-9008
URL: https://issues.apache.org/jira/browse/HADOOP-9008
Project: Hadoop Common
Issue Type: Bug
Affects Versions: trunk-win
Reporter: Ivan Mitic
Assignee: Chris Nauroth
Attachments: HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests
-Dtar}} fails on Windows.
Build system generates sh scripts that execute build tasks what does not work
on Windows without Cygwin. It might make sense to apply the same pattern as
in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9008) Building hadoop tarball fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495633#comment-13495633
]
Raja Aluri commented on HADOOP-9008:
+1. LGTM.
Chris,
Thanks for incorporating the feedback.
Building hadoop tarball fails on Windows
Key: HADOOP-9008
URL: https://issues.apache.org/jira/browse/HADOOP-9008
Project: Hadoop Common
Issue Type: Bug
Affects Versions: trunk-win
Reporter: Ivan Mitic
Assignee: Chris Nauroth
Attachments: HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests
-Dtar}} fails on Windows.
Build system generates sh scripts that execute build tasks what does not work
on Windows without Cygwin. It might make sense to apply the same pattern as
in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9021) Enforce configured SASL method on the server
[
https://issues.apache.org/jira/browse/HADOOP-9021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495638#comment-13495638
]
Hadoop QA commented on HADOOP-9021:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553177/HADOOP-9021.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1731//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1731//console
This message is automatically generated.
Enforce configured SASL method on the server
Key: HADOOP-9021
URL: https://issues.apache.org/jira/browse/HADOOP-9021
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc, security
Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HADOOP-9021.patch, HADOOP-9021.patch
The RPC needs to restrict itself to only using the configured SASL method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9008) Building hadoop tarball fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HADOOP-9008.
-
Resolution: Fixed
Fix Version/s: trunk-win
+1. I committed the patch to branch-trunk-win.
Thank you Chris. Thank you Raja for the review.
Building hadoop tarball fails on Windows
Key: HADOOP-9008
URL: https://issues.apache.org/jira/browse/HADOOP-9008
Project: Hadoop Common
Issue Type: Bug
Affects Versions: trunk-win
Reporter: Ivan Mitic
Assignee: Chris Nauroth
Fix For: trunk-win
Attachments: HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch,
HADOOP-9008-branch-trunk-win.patch, HADOOP-9008-branch-trunk-win.patch
Trying to build Hadoop trunk tarball via {{mvn package -Pdist -DskipTests
-Dtar}} fails on Windows.
Build system generates sh scripts that execute build tasks what does not work
on Windows without Cygwin. It might make sense to apply the same pattern as
in HADOOP-8924, and use python instead of sh.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned HADOOP-9022: --- Assignee: Jonathan Eagles Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8922) Provide alternate JSONP output for JMXJsonServlet to allow javascript in browser dashboard
[
https://issues.apache.org/jira/browse/HADOOP-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins reassigned HADOOP-8922:
---
Assignee: Damien Hardy
Provide alternate JSONP output for JMXJsonServlet to allow javascript in
browser dashboard
--
Key: HADOOP-8922
URL: https://issues.apache.org/jira/browse/HADOOP-8922
Project: Hadoop Common
Issue Type: Improvement
Components: metrics
Affects Versions: 2.0.0-alpha
Reporter: Damien Hardy
Assignee: Damien Hardy
Priority: Trivial
Labels: newbie, patch
Fix For: 3.0.0, 2.0.3-alpha
Attachments: HADOOP-8922-3.patch, HADOOP-8922-4-branch-2.patch,
test.html
JMXJsonServlet may provide a JSONP alternative to JSON to allow javascript in
browser GUI to make requests.
For security purpose about XSS, browser limit request on other
domain[¹|#ref1] so that metrics from cluster nodes cannot be used in a full
js interface.
An example of this kind of dashboard is the bigdesk[²|#ref2] plugin for
ElasticSearch.
In order to achieve that the servlet should detect a GET parameter
(callback=) and modify the response by surrounding the Json value with
( and ); [³|#ref3]
value is variable and should be provide by client as callback
parameter value.
{anchor:ref1}[1]
https://developer.mozilla.org/en-US/docs/Same_origin_policy_for_JavaScript
{anchor:ref2}[2] https://github.com/lukas-vlcek/bigdesk
{anchor:ref3}[3] http://en.wikipedia.org/wiki/JSONP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-9022: Attachment: HADOOP-9022.patch Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-9022: Status: Patch Available (was: Open) Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.4, 0.23.3, 0.23.1 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9017) fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version
[ https://issues.apache.org/jira/browse/HADOOP-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated HADOOP-9017: --- Fix Version/s: 1.2.0 1.1.1 fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version -- Key: HADOOP-9017 URL: https://issues.apache.org/jira/browse/HADOOP-9017 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.0.4, 1.1.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 1.1.1, 1.2.0 Attachments: HADOOP-9017.patch hadoop-client-pom-template.xml and hadoop-client-pom-template.xml references to project.version variable, instead they should refer to @version token. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495673#comment-13495673 ] Robert Joseph Evans commented on HADOOP-9022: - The patch looks fine to me. I am +1 pending Jenkins. Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9017) fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version
[ https://issues.apache.org/jira/browse/HADOOP-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan resolved HADOOP-9017. Resolution: Fixed Thanks Matt, committed to branch-1.1 and merged to branch-1 fix hadoop-client-pom-template.xml and hadoop-client-pom-template.xml for version -- Key: HADOOP-9017 URL: https://issues.apache.org/jira/browse/HADOOP-9017 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.0.4, 1.1.0 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Fix For: 1.1.1, 1.2.0 Attachments: HADOOP-9017.patch hadoop-client-pom-template.xml and hadoop-client-pom-template.xml references to project.version variable, instead they should refer to @version token. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9021) Enforce configured SASL method on the server
[ https://issues.apache.org/jira/browse/HADOOP-9021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495680#comment-13495680 ] Robert Joseph Evans commented on HADOOP-9021: - Looks good +1. I'll check it in. Enforce configured SASL method on the server Key: HADOOP-9021 URL: https://issues.apache.org/jira/browse/HADOOP-9021 Project: Hadoop Common Issue Type: Sub-task Components: ipc, security Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-9021.patch, HADOOP-9021.patch The RPC needs to restrict itself to only using the configured SASL method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9021) Enforce configured SASL method on the server
[ https://issues.apache.org/jira/browse/HADOOP-9021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-9021: Resolution: Fixed Fix Version/s: 2.0.3-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks Daryn, I put this in trunk, and branch-2 Enforce configured SASL method on the server Key: HADOOP-9021 URL: https://issues.apache.org/jira/browse/HADOOP-9021 Project: Hadoop Common Issue Type: Sub-task Components: ipc, security Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-9021.patch, HADOOP-9021.patch The RPC needs to restrict itself to only using the configured SASL method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[
https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495686#comment-13495686
]
Hadoop QA commented on HADOOP-9022:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553197/HADOOP-9022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-distcp:
org.apache.hadoop.tools.TestCopyListing
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//console
This message is automatically generated.
Hadoop distcp tool fails to copy file if -m 0 specified
---
Key: HADOOP-9022
URL: https://issues.apache.org/jira/browse/HADOOP-9022
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.23.1, 0.23.3, 0.23.4
Reporter: Haiyang Jiang
Assignee: Jonathan Eagles
Attachments: HADOOP-9022.patch
When trying to copy file using distcp on H23, if -m 0 is specified, distcp
will just spawn 0 mapper tasks and the file will not be copied.
But this used to work before H23, even when -m 0 specified, distcp will
always copy the files.
Checked the code of DistCp.java
Before the rewrite, it set the number maps at least to 1
job.setNumMapTasks(Math.max(numMaps, 1));
But in the newest code, it just takes the input from user:
job.getConfiguration().set(JobContext.NUM_MAPS,
String.valueOf(inputOptions.getMaxMaps()));
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9021) Enforce configured SASL method on the server
[ https://issues.apache.org/jira/browse/HADOOP-9021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495697#comment-13495697 ] Hudson commented on HADOOP-9021: Integrated in Hadoop-trunk-Commit #3004 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3004/]) HADOOP-9021. Enforce configured SASL method on the server (daryn via bobby) (Revision 1408473) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1408473 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestSaslRPC.java Enforce configured SASL method on the server Key: HADOOP-9021 URL: https://issues.apache.org/jira/browse/HADOOP-9021 Project: Hadoop Common Issue Type: Sub-task Components: ipc, security Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.0.3-alpha Attachments: HADOOP-9021.patch, HADOOP-9021.patch The RPC needs to restrict itself to only using the configured SASL method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
Robert Joseph Evans created HADOOP-9025: --- Summary: org.apache.hadoop.tools.TestCopyListing failing Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495710#comment-13495710 ] Robert Joseph Evans commented on HADOOP-9022: - The test failure seems to be happening with or without the patch. I filed HADOOP-9025 for it. Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated HADOOP-9022: Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha 3.0.0 Status: Resolved (was: Patch Available) Thanks Jon, I pulled this into trunk, branch-2, and branch-0.23 Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9022) Hadoop distcp tool fails to copy file if -m 0 specified
[ https://issues.apache.org/jira/browse/HADOOP-9022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495725#comment-13495725 ] Hudson commented on HADOOP-9022: Integrated in Hadoop-trunk-Commit #3005 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3005/]) HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified (Jonathan Eagles vai bobby) (Revision 1408488) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1408488 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java Hadoop distcp tool fails to copy file if -m 0 specified --- Key: HADOOP-9022 URL: https://issues.apache.org/jira/browse/HADOOP-9022 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.23.1, 0.23.3, 0.23.4 Reporter: Haiyang Jiang Assignee: Jonathan Eagles Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: HADOOP-9022.patch When trying to copy file using distcp on H23, if -m 0 is specified, distcp will just spawn 0 mapper tasks and the file will not be copied. But this used to work before H23, even when -m 0 specified, distcp will always copy the files. Checked the code of DistCp.java Before the rewrite, it set the number maps at least to 1 job.setNumMapTasks(Math.max(numMaps, 1)); But in the newest code, it just takes the input from user: job.getConfiguration().set(JobContext.NUM_MAPS, String.valueOf(inputOptions.getMaxMaps())); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9026) Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
Ivan Mitic created HADOOP-9026: -- Summary: Hadoop.cmd fails to initialize if user's %path% variable has parenthesis Key: HADOOP-9026 URL: https://issues.apache.org/jira/browse/HADOOP-9026 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Hadoop.cmd fails to initialize if user's %path% variable has parenthesis. This happens in updatepath module while tring the add Hadoop_bin_path to %path% Repro: 1. Create a folder C:\random() 2. Add this path to %path% 3. Start Hadoop command line Error: path_after_c:\rand(); unexpected at this time Reported by Ramya Nimmagadda. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9026) Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
[ https://issues.apache.org/jira/browse/HADOOP-9026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9026: --- Attachment: HADOOP-9026.branch-1-win.pathfix.patch Attaching a proposal for a fix. Hadoop.cmd fails to initialize if user's %path% variable has parenthesis Key: HADOOP-9026 URL: https://issues.apache.org/jira/browse/HADOOP-9026 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9026.branch-1-win.pathfix.patch Hadoop.cmd fails to initialize if user's %path% variable has parenthesis. This happens in updatepath module while tring the add Hadoop_bin_path to %path% Repro: 1. Create a folder C:\random() 2. Add this path to %path% 3. Start Hadoop command line Error: path_after_c:\rand(); unexpected at this time Reported by Ramya Nimmagadda. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9026) Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
[ https://issues.apache.org/jira/browse/HADOOP-9026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9026: --- Status: Patch Available (was: Open) Hadoop.cmd fails to initialize if user's %path% variable has parenthesis Key: HADOOP-9026 URL: https://issues.apache.org/jira/browse/HADOOP-9026 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9026.branch-1-win.pathfix.patch Hadoop.cmd fails to initialize if user's %path% variable has parenthesis. This happens in updatepath module while tring the add Hadoop_bin_path to %path% Repro: 1. Create a folder C:\random() 2. Add this path to %path% 3. Start Hadoop command line Error: path_after_c:\rand(); unexpected at this time Reported by Ramya Nimmagadda. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9026) Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
[
https://issues.apache.org/jira/browse/HADOOP-9026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495784#comment-13495784
]
Hadoop QA commented on HADOOP-9026:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553227/HADOOP-9026.branch-1-win.pathfix.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1733//console
This message is automatically generated.
Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
Key: HADOOP-9026
URL: https://issues.apache.org/jira/browse/HADOOP-9026
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: HADOOP-9026.branch-1-win.pathfix.patch
Hadoop.cmd fails to initialize if user's %path% variable has parenthesis.
This happens in updatepath module while tring the add Hadoop_bin_path to
%path%
Repro:
1. Create a folder C:\random()
2. Add this path to %path%
3. Start Hadoop command line
Error:
path_after_c:\rand(); unexpected at this time
Reported by Ramya Nimmagadda.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8999) SASL negotiation is flawed
[ https://issues.apache.org/jira/browse/HADOOP-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-8999: Attachment: HADOOP-8999.patch Fixing this issue correctly by always ending the SASL handshake with a success/failure introduces an rpc incompatibility. Purely for short-term compatibility, the rpc server will send a final success response to the client if and only if the SASL method is PLAIN. The client code is also changed to ensure that at least one rpc response is always read. The reason why the currently supported KERBEROS and DIGEST methods work is they exchange multiple challenge/responses. This guarantees an exception response or a switch-to-simple message will be read since they will tend to occur on the first exchange. The PLAIN method however involves a single exchange. The client sends an initial response and considers itself done - which means no rpc response will be read. The server handles the response and also considers itself done. However, the server may have responded with an exception or a switch to simple message. But the client won't read it until the next proxy call where it will generate a protobuf exception. Hence why this patch forces the server to send a success response for PLAIN, and to require the client to read at least one rpc so it can read the success or error message from the server. In the long term, the server should always send a final success response for authentication. Ideally there should also be an intermediate RPC state for SASL to disambiguate if negotiation is occurring or has completed. SASL negotiation is flawed -- Key: HADOOP-8999 URL: https://issues.apache.org/jira/browse/HADOOP-8999 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-8999.patch The RPC protocol used for SASL negotiation is flawed. The server's RPC response contains the next SASL challenge token, but a SASL server can return null (I'm done) or a N-many byte challenge. The server currently will not send a RPC success response to the client if the SASL server returns null, which causes the client to hang until it times out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8999) SASL negotiation is flawed
[ https://issues.apache.org/jira/browse/HADOOP-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-8999: Status: Patch Available (was: Open) SASL negotiation is flawed -- Key: HADOOP-8999 URL: https://issues.apache.org/jira/browse/HADOOP-8999 Project: Hadoop Common Issue Type: Sub-task Components: ipc Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HADOOP-8999.patch The RPC protocol used for SASL negotiation is flawed. The server's RPC response contains the next SASL challenge token, but a SASL server can return null (I'm done) or a N-many byte challenge. The server currently will not send a RPC success response to the client if the SASL server returns null, which causes the client to hang until it times out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8999) SASL negotiation is flawed
[
https://issues.apache.org/jira/browse/HADOOP-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495799#comment-13495799
]
Hadoop QA commented on HADOOP-8999:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553229/HADOOP-8999.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1734//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1734//console
This message is automatically generated.
SASL negotiation is flawed
--
Key: HADOOP-8999
URL: https://issues.apache.org/jira/browse/HADOOP-8999
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HADOOP-8999.patch
The RPC protocol used for SASL negotiation is flawed. The server's RPC
response contains the next SASL challenge token, but a SASL server can return
null (I'm done) or a N-many byte challenge. The server currently will not
send a RPC success response to the client if the SASL server returns null,
which causes the client to hang until it times out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8820: -- Attachment: HADOOP-8820.b1.002.patch Hi Junping, maybe you're busy with other stuff. Since to address Nicholas's comments only needs simple changes, I just make these changes on your patch and run unit tests for you. All the test cases have passed on my machine. Thanks! Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[
https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495829#comment-13495829
]
Jing Zhao commented on HADOOP-8820:
---
Besides, some other small issues:
1. {noformat}
+@Override
+boolean isRack() {
+ // it is node group
+ if (getChildren().isEmpty()) {
+return false;
+ }
+
+ Node firstChild = children.get(0);
+
+ if (firstChild instanceof InnerNode) {
+Node firstGrandChild = (((InnerNode) firstChild).children).get(0);
+if (firstGrandChild instanceof InnerNode) {
{noformat}
I think here maybe firstChild.children can be empty thus the get(0) may cause
an IndexOutofBoundsException. So maybe we need to check if children is empty
first (and if it is, we can return true)?
2. {noformat}
+ public String getNodeGroup(String loc) {
+try {
+ netlock.readLock().lock();
+ loc = InnerNode.normalize(loc);
+ Node locNode = getNode(loc);
+ if (locNode instanceof InnerNodeWithNodeGroup) {
+InnerNodeWithNodeGroup node = (InnerNodeWithNodeGroup) locNode;
+if (node.isNodeGroup()) {
+ return loc;
+} else if (node.isRack()) {
+ // not sure the node group for a rack
+ return null;
+} else {
+ // may be a leaf node
+ return getNodeGroup(node.getNetworkLocation());
+}
{noformat}
Since the locNode is already an instance of InnerNodeWithNodeGroup, I think
it cannot be a leaf node anymore, thus the code may never run into the else
part?
I can create a separate jira to fix if you think the two issues stand.
Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new
NetworkTopology (also known as NetworkTopologyWithNodeGroup)
---
Key: HADOOP-8820
URL: https://issues.apache.org/jira/browse/HADOOP-8820
Project: Hadoop Common
Issue Type: New Feature
Components: net
Affects Versions: 1.0.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch
This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes:
1. Make NetworkTopology class pluggable for extension.
2. Implement a 4-layer NetworkTopology class (named as
NetworkTopologyWithNodeGroup) to use in virtualized environment (or other
situation with additional layer between host and rack).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495830#comment-13495830 ] Junping Du commented on HADOOP-8820: Thanks Jing for the help here. However, I think the patch in HADOOP-8817 is the latest code and it fix a bug on placing replicas (the first repilca's host is not disallowed before). I did a lot of test (functional and performance) in a real deployed cluster on that patch, but my ant test still meet some trouble on my environment. Can you help to run ant test on that patch? Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495839#comment-13495839 ] Jing Zhao commented on HADOOP-8820: --- Sure. I will re-generate the patch based on HADOOP-8817 and run ant test for the new patch. Can you give more details about the changes you made in 8817? Thanks. Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HADOOP-8820: -- Attachment: HADOOP-8820.b1.003.patch New patch based on HADOOP-8817 and Nicholas's comments. Will run ant test for the new patch. Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495863#comment-13495863 ] Jing Zhao commented on HADOOP-8820: --- Hi Junping, could you please recheck the new patch and make sure all the corresponding changes in HADOOP-8817 have been included? Thanks. Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8817) Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-8817: --- Attachment: HADOOP-8817-v3.patch Update patch slightly due to Nicholas's comments in HADOOP-8820 Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1 Key: HADOOP-8817 URL: https://issues.apache.org/jira/browse/HADOOP-8817 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Labels: features Attachments: HADOOP-8817.patch, HADOOP-8817-v2.patch, HADOOP-8817-v3.patch HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes: 1. Add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470 2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601 3. Update balancer policy:HDFS-3495 4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19. This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8817) Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495869#comment-13495869 ] Junping Du commented on HADOOP-8817: I meet some issues in running ant test on local environment that even branch-1 without patch cannot pass all tests. I run related unit tests which is good and did lots of tests on real cluster for this patch (reliability and performance) which you can find in attachment in HADOOP-8848. Anyone can help to verify this patch to pass ant test? Backport Network Topology Extension for Virtualization (HADOOP-8468) to branch-1 Key: HADOOP-8817 URL: https://issues.apache.org/jira/browse/HADOOP-8817 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Labels: features Attachments: HADOOP-8817.patch, HADOOP-8817-v2.patch, HADOOP-8817-v3.patch HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes: 1. Add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470 2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601 3. Update balancer policy:HDFS-3495 4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19. This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495870#comment-13495870 ] Junping Du commented on HADOOP-8820: Hi Jing, can you post ant test result on HADOOP-8817? I just adjust the patch due to Nicholas' comments. Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8820) Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup)
[ https://issues.apache.org/jira/browse/HADOOP-8820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495886#comment-13495886 ] Junping Du commented on HADOOP-8820: Also, forget to note that, I think it could be convenient to have only one patch for NodeGroup layer with both HDFS and MAPREDUCE. So I suggest we don't commit this patch but commit HADOOP-8817 directly. Backport HADOOP-8469 and HADOOP-8470: add NodeGroup layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup) --- Key: HADOOP-8820 URL: https://issues.apache.org/jira/browse/HADOOP-8820 Project: Hadoop Common Issue Type: New Feature Components: net Affects Versions: 1.0.0 Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8820.b1.002.patch, HADOOP-8820.b1.003.patch, HADOOP-8820.patch This patch backport HADOOP-8469 and HADOOP-8470 to branch-1 and includes: 1. Make NetworkTopology class pluggable for extension. 2. Implement a 4-layer NetworkTopology class (named as NetworkTopologyWithNodeGroup) to use in virtualized environment (or other situation with additional layer between host and rack). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned HADOOP-9025: --- Assignee: Jonathan Eagles org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495916#comment-13495916 ] Jonathan Eagles commented on HADOOP-9025: - TestCopyListing#testDuplicates relied on the strange globbing behavior that existed prior to HADOOP-8906. Updated test to have the glob match two file with the same name. org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-9025: Attachment: HADOOP-9025.patch org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HADOOP-9025: Status: Patch Available (was: Open) org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[
https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495939#comment-13495939
]
Hadoop QA commented on HADOOP-9025:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553263/HADOOP-9025.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1735//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1735//console
This message is automatically generated.
org.apache.hadoop.tools.TestCopyListing failing
---
Key: HADOOP-9025
URL: https://issues.apache.org/jira/browse/HADOOP-9025
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Robert Joseph Evans
Assignee: Jonathan Eagles
Attachments: HADOOP-9025.patch
https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495971#comment-13495971 ] Jason Lowe commented on HADOOP-9025: +1, will commit shortly. org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated HADOOP-9025: --- Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Jonathan. I committed this to trunk, branch-2, and branch-0.23. org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Fix For: 2.0.3-alpha, 0.23.5 Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9025) org.apache.hadoop.tools.TestCopyListing failing
[ https://issues.apache.org/jira/browse/HADOOP-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13495999#comment-13495999 ] Hudson commented on HADOOP-9025: Integrated in Hadoop-trunk-Commit #3007 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3007/]) HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing. Contributed by Jonathan Eagles (Revision 1408604) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1408604 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestCopyListing.java org.apache.hadoop.tools.TestCopyListing failing --- Key: HADOOP-9025 URL: https://issues.apache.org/jira/browse/HADOOP-9025 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Robert Joseph Evans Assignee: Jonathan Eagles Fix For: 2.0.3-alpha, 0.23.5 Attachments: HADOOP-9025.patch https://builds.apache.org/job/PreCommit-HADOOP-Build/1732//testReport/org.apache.hadoop.tools/TestCopyListing/testDuplicates/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9027) Build fails on Windows without sh/sed/echo in the path
Ivan Mitic created HADOOP-9027: -- Summary: Build fails on Windows without sh/sed/echo in the path Key: HADOOP-9027 URL: https://issues.apache.org/jira/browse/HADOOP-9027 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Branch-1-win still has a dependency on a few unix tools in compile time. Tracking Jira to remove this dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9027) Build fails on Windows without sh/sed/echo in the path
[ https://issues.apache.org/jira/browse/HADOOP-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9027: --- Attachment: HADOOP-9027.branch-1-win.cleanbuild.patch Attaching the patch. Tested on Ubuntu and Windows. Build fails on Windows without sh/sed/echo in the path -- Key: HADOOP-9027 URL: https://issues.apache.org/jira/browse/HADOOP-9027 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9027.branch-1-win.cleanbuild.patch Branch-1-win still has a dependency on a few unix tools in compile time. Tracking Jira to remove this dependency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
