[jira] [Commented] (HADOOP-11518) Prefix-based per-RPC server configuration
[ https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302034#comment-14302034 ] Hadoop QA commented on HADOOP-11518: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695968/HADOOP-11518.v2.patch against trunk revision 3472e3b. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5557//console This message is automatically generated. Prefix-based per-RPC server configuration - Key: HADOOP-11518 URL: https://issues.apache.org/jira/browse/HADOOP-11518 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch If a process runs multiple RPC servers, there is no way to individually set their IPC parameters beyond what's supported by the RPC Builder. If a configuration is shared among many nodes, this is more problematic. In this jira, a prefix-based per-server configuration is proposed. When an RPC server is built, a prefix can be specified, which will then be prepended to keys when retrieving configured values. There will be an option to fall-back to using the key without the prefix when no value is found with the prefix, thus making this feature compatible with the current way of configuring servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10948) SwiftNativeFileSystem's directory is incompatible with Swift and Horizon
[ https://issues.apache.org/jira/browse/HADOOP-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302051#comment-14302051 ] David Dobbins commented on HADOOP-10948: Is there an avenue for maintaining backwards compatibility? It seems the only risk is for directories already created in swift with SwiftNativeFileSystem. Would it suffice to only create new directories using the trailing-slash model but to recognize either model for the purposes of reading existing directories? SwiftNativeFileSystem's directory is incompatible with Swift and Horizon Key: HADOOP-10948 URL: https://issues.apache.org/jira/browse/HADOOP-10948 Project: Hadoop Common Issue Type: Bug Components: fs/swift Affects Versions: 3.0.0 Reporter: Kazuki OIKAWA Assignee: Kazuki OIKAWA Attachments: HADOOP-10948-2.patch, HADOOP-10948.patch SwiftNativeFileSystem's directory representation is zero-byte file. But in Swift / Horizon, directory representation is a trailing-slash. This incompatibility has the following issues. * SwiftNativeFileSystem can't see pseudo-directory made by OpenStack Horizon * Swift/Horizon can't see pseudo-directory made by SwiftNativeFileSystem. But Swift/Horizon see a zero-byte file instead of that pseudo-directory. * SwiftNativeFileSystem can't see a file if there is no intermediate pseudo-directory object. * SwiftNativeFileSystem makes two objects when making a single directory (e.g. hadoop fs -mkdir swift://test.test/dir/ = dir and dir/ created) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11442) hadoop-azure: Create test jar
[ https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301821#comment-14301821 ] Hudson commented on HADOOP-11442: - SUCCESS: Integrated in Hadoop-trunk-Commit #6980 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6980/]) HADOOP-11442. hadoop-azure: Create test jar. Contributed by Shashank Khandelwal. (cnauroth: rev 1c09ca2ba4d97d827cc6e86ef6bd2930bc77b5ae) * hadoop-tools/hadoop-azure/pom.xml * hadoop-common-project/hadoop-common/CHANGES.txt hadoop-azure: Create test jar - Key: HADOOP-11442 URL: https://issues.apache.org/jira/browse/HADOOP-11442 Project: Hadoop Common Issue Type: Improvement Components: tools Environment: windows, azure Reporter: shashank Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, HADOOP-11442v2.patch pom of hadoop-azure project to needs to be modified to create a test jar as well. This test jar is required to run test cases of Windowsazuretablesink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11518) Prefix-based per-RPC server configuration
[ https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302068#comment-14302068 ] Kihwal Lee commented on HADOOP-11518: - The release audit is fixed in YARN-3113. Prefix-based per-RPC server configuration - Key: HADOOP-11518 URL: https://issues.apache.org/jira/browse/HADOOP-11518 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch If a process runs multiple RPC servers, there is no way to individually set their IPC parameters beyond what's supported by the RPC Builder. If a configuration is shared among many nodes, this is more problematic. In this jira, a prefix-based per-server configuration is proposed. When an RPC server is built, a prefix can be specified, which will then be prepended to keys when retrieving configured values. There will be an option to fall-back to using the key without the prefix when no value is found with the prefix, thus making this feature compatible with the current way of configuring servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow
[ https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302026#comment-14302026 ] Hadoop QA commented on HADOOP-11506: {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696002/HADOOP-11506.004.patch against trunk revision 5f9a0dd. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5559//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5559//console This message is automatically generated. Configuration.get() is unnecessarily slow - Key: HADOOP-11506 URL: https://issues.apache.org/jira/browse/HADOOP-11506 Project: Hadoop Common Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Gera Shegalov Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, HADOOP-11506.003.patch, HADOOP-11506.004.patch Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-10867) NPE in heartbeat when the configured topology script doesn't exist
[ https://issues.apache.org/jira/browse/HADOOP-10867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari reassigned HADOOP-10867: - Assignee: Mauro Murari (was: Ivan Mitic) NPE in heartbeat when the configured topology script doesn't exist -- Key: HADOOP-10867 URL: https://issues.apache.org/jira/browse/HADOOP-10867 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Reporter: Vinod Kumar Vavilapalli Assignee: Mauro Murari Labels: newbie -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup
[ https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301995#comment-14301995 ] Andrew Johnson commented on HADOOP-10181: - Great, thanks! GangliaContext does not work with multicast ganglia setup - Key: HADOOP-10181 URL: https://issues.apache.org/jira/browse/HADOOP-10181 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.6.0 Reporter: Andrew Otto Assignee: Andrew Johnson Priority: Minor Labels: ganglia, hadoop, metrics, multicast Fix For: 2.7.0 Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, HADOOP-10181.003.patch The GangliaContext class which is used to send Hadoop metrics to Ganglia uses a DatagramSocket to send these metrics. This works fine for Ganglia multicast setups that are all on the same VLAN. However, when working with multiple VLANs, a packet sent via DatagramSocket to a multicast address will end up with a TTL of 1. Multicast TTL indicates the number of network hops for which a particular multicast packet is valid. The packets sent by GangliaContext do not make it to ganglia aggregrators on the same multicast group, but in different VLANs. To fix, we'd need a configuration property that specifies that multicast is to be used, and another that allows setting of the multicast packet TTL. With these set, we could then use MulticastSocket setTimeToLive() instead of just plain ol' DatagramSocket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking
[ https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301769#comment-14301769 ] Ray Chiang commented on HADOOP-11104: - RE: Release audit Sorting icons.psd not part of the patch org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking - Key: HADOOP-11104 URL: https://issues.apache.org/jira/browse/HADOOP-11104 Project: Hadoop Common Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: HADOOP-11104.001.patch Passing a negative value to the interval field of MetricsRegistry#newQuantiles should throw a MetricsException with a clear error message. The current stack trace looks something like: java.lang.IllegalArgumentException: null at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420) at org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107) at org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200) Along similar lines, should the other methods like MetricsRegistry#newCounter() also have parameter checking for negative int/long values? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.
[ https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301861#comment-14301861 ] Steve Loughran commented on HADOOP-11525: - Mark it as included; I did lift your patch to the FS client to make it the first real use case. Now, one thing that could be good would to be to make some publishing of the semantics something that every FS does -but then, life gets complex fast. For the HADOOP-9361 contract tests I did make every FS publish their rules, [such as for localfs|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/resources/contract/localfs.xml] ... but we kept them in test/resources as they were intended to be only stuff that tests should care about when measuring how different the other implementations were from HDFS, using the flags to verify that their different behavior was at least as expected. I'd hate to have code checks everywhere that looks for filesystem quirks. Either code for HDFS or target the lowest common denominator of an object store with write-on-close and minimal consistency guarantees. And for that, we need to identify those critical code paths that assume HDFS but end up with the latter FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm. --- Key: HADOOP-11525 URL: https://issues.apache.org/jira/browse/HADOOP-11525 Project: Hadoop Common Issue Type: Improvement Components: tools Affects Versions: 2.6.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HADOOP-11525.000.patch When running {{hadoop fs -put}}, {{FsShell}} creates a {{._COPYING_.}} file on the target directory, and then renames it to target file when the write is done. However, for some targeted systems, such as S3, Azure and Swift, a partial failure write request (i.e., {{PUT}}) has not side effect, while the {{rename}} operation is expensive. {{FileSystem}} should expose some characteristics so that the operation such as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the right way to do. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11506) Configuration.get() is unnecessarily slow
[ https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HADOOP-11506: --- Attachment: HADOOP-11506.004.patch Minor fix: I had an unintended HashSet size specification: {code} diff --git a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java index 9380d68..c47e771 100644 --- a/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java +++ b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java @@ -963,7 +963,7 @@ private String substituteVars(String expr) { final int afterRightBrace = varBounds[SUB_END_IDX] + }.length(); final String refVar = eval.substring(dollar, afterRightBrace); if (evalSet == null) { -evalSet = new HashSetString(1); +evalSet = new HashSetString(); } if (!evalSet.add(refVar)) { return expr; // return original expression if there is a loop {code} Configuration.get() is unnecessarily slow - Key: HADOOP-11506 URL: https://issues.apache.org/jira/browse/HADOOP-11506 Project: Hadoop Common Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Gera Shegalov Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, HADOOP-11506.003.patch, HADOOP-11506.004.patch Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-5454) SortedMapWritable: readFields() will not clear values before deserialization
[ https://issues.apache.org/jira/browse/HADOOP-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari updated HADOOP-5454: - Assignee: (was: Mauro Murari) SortedMapWritable: readFields() will not clear values before deserialization Key: HADOOP-5454 URL: https://issues.apache.org/jira/browse/HADOOP-5454 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.19.1 Reporter: Stefan Podkowinski In case SortedMapWritable is used as value in a reducer, the user must explicitly call clear() on the map between iterating values. This is because SortedMapWritable will be reused once instantiated, but consecutive calls to readFields() will not reset the maps internal state, as e.g. done by MapWritable. Please add this.instance.clear(); on top of readFields(). You may also want to consider HADOOP-5028 for fixing another issue with this class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-11529: -- Status: Patch Available (was: Open) Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-3602) DataInputBuffer::getLength returns the end position, not the length
[ https://issues.apache.org/jira/browse/HADOOP-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari reassigned HADOOP-3602: Assignee: Mauro Murari DataInputBuffer::getLength returns the end position, not the length --- Key: HADOOP-3602 URL: https://issues.apache.org/jira/browse/HADOOP-3602 Project: Hadoop Common Issue Type: Bug Reporter: Chris Douglas Assignee: Mauro Murari In DataInputBuffer, a call to reset(byte[] buffer, offset, len) resets the data read by the buffer. However, when the offset is not zero, getLength returns the index marking the end of valid data in the buffer and not the length as specified in the last call to reset. The consequence is that consumers of DataInputBuffers may inaccurately gauge the length of the input given them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class
[ https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari reassigned HADOOP-10646: - Assignee: Mauro Murari (was: Alejandro Abdelnur) KeyProvider buildVersionName method should be moved to a utils class Key: HADOOP-10646 URL: https://issues.apache.org/jira/browse/HADOOP-10646 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur Assignee: Mauro Murari The buildVersionName() method should not be part of the KeyProvider public API because keyversions could be opaque (not built based on the keyname and key generation counter). KeyProvider implementations may choose to use buildVersionName() for reasons such as described in HADOOP-10611. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302556#comment-14302556 ] Hadoop QA commented on HADOOP-11529: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696049/HADOOP-11529.002.patch against trunk revision 8acc5e9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-archives. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5561//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5561//console This message is automatically generated. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11162) Unclosed InputStream in ApplicationClassLoader
[ https://issues.apache.org/jira/browse/HADOOP-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302561#comment-14302561 ] Mauro Murari commented on HADOOP-11162: --- It was closed, please see this commit https://github.com/apache/hadoop/commit/825923f7b9a8771ff5b3f08646e96304d1acc367 Unclosed InputStream in ApplicationClassLoader -- Key: HADOOP-11162 URL: https://issues.apache.org/jira/browse/HADOOP-11162 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Priority: Minor {code} static { InputStream is = null; {code} The above InputStream is not closed upon leaving the static block. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-11529: -- Attachment: HADOOP-11529.001.patch attaching patch. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HADOOP-11529: -- Attachment: HADOOP-11529.002.patch Thanks for the comment [~wheat9]! I updated the patch. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10646) KeyProvider buildVersionName method should be moved to a utils class
[ https://issues.apache.org/jira/browse/HADOOP-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari updated HADOOP-10646: -- Assignee: (was: Mauro Murari) KeyProvider buildVersionName method should be moved to a utils class Key: HADOOP-10646 URL: https://issues.apache.org/jira/browse/HADOOP-10646 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 3.0.0 Reporter: Alejandro Abdelnur The buildVersionName() method should not be part of the KeyProvider public API because keyversions could be opaque (not built based on the keyname and key generation counter). KeyProvider implementations may choose to use buildVersionName() for reasons such as described in HADOOP-10611. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-3602) DataInputBuffer::getLength returns the end position, not the length
[ https://issues.apache.org/jira/browse/HADOOP-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mauro Murari updated HADOOP-3602: - Assignee: (was: Mauro Murari) DataInputBuffer::getLength returns the end position, not the length --- Key: HADOOP-3602 URL: https://issues.apache.org/jira/browse/HADOOP-3602 Project: Hadoop Common Issue Type: Bug Reporter: Chris Douglas In DataInputBuffer, a call to reset(byte[] buffer, offset, len) resets the data read by the buffer. However, when the offset is not zero, getLength returns the index marking the end of valid data in the buffer and not the length as specified in the last call to reset. The consequence is that consumers of DataInputBuffers may inaccurately gauge the length of the input given them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302325#comment-14302325 ] Haohui Mai commented on HADOOP-11529: - LGTM. Minor nit: {code} -reader.close(); +if (reader != null) { + reader.close(); +} {code} You can use Java 7's try-with-resource statement here. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302493#comment-14302493 ] Haohui Mai commented on HADOOP-11529: - +1 pending jenkins. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch, HADOOP-11529.002.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking
[ https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302593#comment-14302593 ] Ray Chiang commented on HADOOP-11104: - It looks like the release audit problem is fixed in YARN-3113. org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking - Key: HADOOP-11104 URL: https://issues.apache.org/jira/browse/HADOOP-11104 Project: Hadoop Common Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: HADOOP-11104.001.patch Passing a negative value to the interval field of MetricsRegistry#newQuantiles should throw a MetricsException with a clear error message. The current stack trace looks something like: java.lang.IllegalArgumentException: null at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420) at org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107) at org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200) Along similar lines, should the other methods like MetricsRegistry#newCounter() also have parameter checking for negative int/long values? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11529) Fix findbugs warnings in hadoop-archives
[ https://issues.apache.org/jira/browse/HADOOP-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302321#comment-14302321 ] Hadoop QA commented on HADOOP-11529: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12696028/HADOOP-11529.001.patch against trunk revision 8acc5e9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-archives. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5560//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5560//console This message is automatically generated. Fix findbugs warnings in hadoop-archives Key: HADOOP-11529 URL: https://issues.apache.org/jira/browse/HADOOP-11529 Project: Hadoop Common Issue Type: Bug Components: tools Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Attachments: HADOOP-11529.001.patch {code} NPPossible null pointer dereference of reader in org.apache.hadoop.tools.HadoopArchives$HArchiveInputFormat.getSplits(JobConf, int) on exception path DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.close(): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.configure(JobConf): String.getBytes() DmFound reliance on default encoding in org.apache.hadoop.tools.HadoopArchives$HArchivesReducer.reduce(IntWritable, Iterator, OutputCollector, Reporter): String.getBytes() {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301260#comment-14301260 ] Hadoop QA commented on HADOOP-11520: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695622/HADOOP-11520.001.patch against trunk revision ffc75d6. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-aws. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build///testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build///artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build///console This message is automatically generated. Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11520: Status: Patch Available (was: Open) Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8640) DU thread transient failures propagate to callers
[ https://issues.apache.org/jira/browse/HADOOP-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301084#comment-14301084 ] Elad Itzhakian commented on HADOOP-8640: Getting this in 2.6.0 too, during TestDFSIO {quote} 2014-12-15 20:40:19,944 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver constructor. Cause is org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access `/data3/dfs/dn/current/BP-1172228382-36.0.0.112-1418564551057/current/rbw/blk_1073754273': No such file or directory at org.apache.hadoop.util.Shell.runCommand(Shell.java:511) at org.apache.hadoop.util.Shell.run(Shell.java:424) at org.apache.hadoop.fs.DU.run(DU.java:190) at org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:119) at java.lang.Thread.run(Thread.java:745) {quote} DU thread transient failures propagate to callers - Key: HADOOP-8640 URL: https://issues.apache.org/jira/browse/HADOOP-8640 Project: Hadoop Common Issue Type: Bug Components: fs, io Affects Versions: 2.0.0-alpha, 1.2.1 Reporter: Todd Lipcon When running some stress tests, I saw a failure where the DURefreshThread failed due to the filesystem changing underneath it: {code} org.apache.hadoop.util.Shell$ExitCodeException: du: cannot access `/data/4/dfs/dn/current/BP-1928785663-172.20.90.20-1343880685858/current/rbw/blk_4637779214690837894': No such file or directory {code} (the block was probably finalized while the du process was running, which caused it to fail) The next block write, then, called {{getUsed()}}, and the exception got propagated causing the write to fail. Since it was a pseudo-distributed cluster, the client was unable to pick a different node to write to and failed. The current behavior of propagating the exception to the next (and only the next) caller doesn't seem well-thought-out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
Kai Zheng created HADOOP-11535: -- Summary: TableMapping related tests failed due to 'correct' resolving for test hostname Key: HADOOP-11535 URL: https://issues.apache.org/jira/browse/HADOOP-11535 Project: Hadoop Common Issue Type: Test Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor When mvn test in my environment, it reported the following. {noformat} Failed tests: TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but was:/[default-rack] TestTableMapping.testTableCaching:79 expected:/[rack1] but was:/[default-rack] TestTableMapping.testResolve:56 expected:/[rack1] but was:/[default-rack] {noformat} It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as follows. {noformat} [drankye@zkdesk hadoop-common-project]$ ping a.b.c PING a.b.c (220.250.64.228) 56(84) bytes of data. {noformat} I understand it may happen in just my local environment, and document this just in case others also meet this. We may use even worse hostname than 'a.b.c' to avoid such situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11462) TestSocketIOWithTimeout needs change for PowerPC platform
[ https://issues.apache.org/jira/browse/HADOOP-11462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302802#comment-14302802 ] Ayappan commented on HADOOP-11462: -- Above build failure is not due to this patch since the failure happens in mapreduce related testcase whereas this patch is for TestSocketIOWithTimeout ( hadoop-common module ). Check out https://www.mail-archive.com/mapreduce-dev@hadoop.apache.org/msg12794.html TestSocketIOWithTimeout needs change for PowerPC platform - Key: HADOOP-11462 URL: https://issues.apache.org/jira/browse/HADOOP-11462 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.5.2 Environment: PowerPC Reporter: Ayappan Assignee: Ayappan Fix For: 2.7.0 Attachments: HADOOP-9627-v1.patch, HADOOP-9627-v2.patch TestSocketIOWithTimeout uses a block size of 4192 bytes to simulate a partial write. This seems to be a valid in x86 architecture where the default minimum blocksize is 4096. This testcase fails in PowerPC where the default minimum block size is 65536 bytes (64KB). So for PowerPC, using a blocksize little more than 64K , say 6(65536 + 19) holds good for this scenario. I attached a patch here where i made it very general by introducing NativeIO.POSIX.getCacheManipulator().getOperatingSystemPageSize() to get the page size. I tested my patch in both ppc64 and x86 linux machines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11043) Use Path Instead of String in DistCp Tests
[ https://issues.apache.org/jira/browse/HADOOP-11043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302800#comment-14302800 ] Tsuyoshi OZAWA commented on HADOOP-11043: - Cancelling a patch. Use Path Instead of String in DistCp Tests -- Key: HADOOP-11043 URL: https://issues.apache.org/jira/browse/HADOOP-11043 Project: Hadoop Common Issue Type: Improvement Components: tools/distcp Reporter: Gary Steelman Assignee: Gary Steelman Priority: Minor Attachments: HADOOP-11043.1.patch Many DistCp tests mix String and Path for files for no real reason. This patch aligns use to Path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11532) RAT checker complaining about PSD images
[ https://issues.apache.org/jira/browse/HADOOP-11532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved HADOOP-11532. - Resolution: Duplicate This is a duplicate of YARN-3113. RAT checker complaining about PSD images Key: HADOOP-11532 URL: https://issues.apache.org/jira/browse/HADOOP-11532 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Steve Loughran Jenkins is rejecting builds as {{Sorting icons.psd}} doesn't have an ASF header. {code} !? /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/webapps/static/dt-1.9.4/images/Sorting icons.psd Lines that start with ? in the release audit report indicate files that do not have an Apache license header. {code} It's a layered photoshop image that either needs to be excluded from RAT or cut from the source tree -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11293: Status: Open (was: Patch Available) Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301374#comment-14301374 ] Steve Loughran commented on HADOOP-11293: - lets bounce Jenkins and see what happens again...there's sometimes conflict with other tests on the same host Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11293: Status: Patch Available (was: Open) Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11520) Clean incomplete multi-part uploads in S3A tests
[ https://issues.apache.org/jira/browse/HADOOP-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301344#comment-14301344 ] Thomas Demoor commented on HADOOP-11520: unrelated error. photoshop documents lol? Clean incomplete multi-part uploads in S3A tests Key: HADOOP-11520 URL: https://issues.apache.org/jira/browse/HADOOP-11520 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.6.0 Reporter: Thomas Demoor Assignee: Thomas Demoor Priority: Minor Fix For: 2.7.0 Attachments: HADOOP-11520.001.patch As proposed in HADOOP-11488. This patch activates the purging functionality of s3a at the start of each test. This cleans up any in-progress multi-part uploads in the test bucket, preventing unknowing users from eternally paying Amazon for the space of the already uploaded parts of previous tests that failed during a multi-part upload. People who have run the s3a tests should run a single test (evidently after this patch is applied) against all their testbuckets (or manually abort multipart). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11293) Factor OSType out from Shell
[ https://issues.apache.org/jira/browse/HADOOP-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301412#comment-14301412 ] Yongjun Zhang commented on HADOOP-11293: Hi Steve, many thanks for your help! Factor OSType out from Shell Key: HADOOP-11293 URL: https://issues.apache.org/jira/browse/HADOOP-11293 Project: Hadoop Common Issue Type: Improvement Components: fs, util Reporter: Yongjun Zhang Assignee: Yongjun Zhang Attachments: HADOOP-11293.001.patch, HADOOP-11293.002.patch, HADOOP-11293.003.patch, HADOOP-11293.004.patch, HADOOP-11293.005.patch Currently the code that detects the OS type is located in Shell.java. Code that need to check OS type refers to Shell, even if no other stuff of Shell is needed. I am proposing to refactor OSType out to its own class, so to make the OSType easier to access and the dependency cleaner. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11535) TableMapping related tests failed due to 'correct' resolving for test hostname
[ https://issues.apache.org/jira/browse/HADOOP-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301436#comment-14301436 ] Steve Loughran commented on HADOOP-11535: - It's traditional to use {{something.example.org}} as the IETF mandates that these hostnames must never resolve. However if you are using Verizon fibre @ home, it may still resolve for you. There is an implicit expectation that a network doesn't work as defined. TableMapping related tests failed due to 'correct' resolving for test hostname -- Key: HADOOP-11535 URL: https://issues.apache.org/jira/browse/HADOOP-11535 Project: Hadoop Common Issue Type: Test Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor When mvn test in my environment, it reported the following. {noformat} Failed tests: TestTableMapping.testClearingCachedMappings:144 expected:/[rack1] but was:/[default-rack] TestTableMapping.testTableCaching:79 expected:/[rack1] but was:/[default-rack] TestTableMapping.testResolve:56 expected:/[rack1] but was:/[default-rack] {noformat} It's caused by the good resolving for the 'bad test' hostname 'a.b.c' as follows. {noformat} [drankye@zkdesk hadoop-common-project]$ ping a.b.c PING a.b.c (220.250.64.228) 56(84) bytes of data. {noformat} I understand it may happen in just my local environment, and document this just in case others also meet this. We may use even worse hostname than 'a.b.c' to avoid such situation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking
[ https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-11104: Attachment: HADOOP-11104.001.patch Initial version org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking - Key: HADOOP-11104 URL: https://issues.apache.org/jira/browse/HADOOP-11104 Project: Hadoop Common Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: HADOOP-11104.001.patch Passing a negative value to the interval field of MetricsRegistry#newQuantiles should throw a MetricsException with a clear error message. The current stack trace looks something like: java.lang.IllegalArgumentException: null at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420) at org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107) at org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200) Along similar lines, should the other methods like MetricsRegistry#newCounter() also have parameter checking for negative int/long values? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9565) Add a Blobstore interface to add to blobstore FileSystems
[ https://issues.apache.org/jira/browse/HADOOP-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301606#comment-14301606 ] Chris Nauroth commented on HADOOP-9565: --- This is nice work, Steve. Thanks for putting this together. Here are a few comments: # Have you considered attaching semantics directly to {{FileSystem}} and {{FileContext}} instead of introducing the {{ObjectStore}} abstract base class? This could give us a modest simplification of calling code, because it wouldn't have to downcast, either directly or implicitly by taking the extra hop through the static {{ObjectStore#hasStoreSemantics}}. # Shall we change {{StoreSemantics}} to an enum and then phrase the API in terms of {{EnumSetStoreSemantics}} for improved type safety? There is existing precedent for use of {{EnumSet}} in the {{FileSystem}} API, so I expect callers will find it familiar. # I saw JavaDoc typos on {{ATOMIC_DIR_DELETE}} and {{DISTINCT_FILES_AND_DIRECTORIES}}. Add a Blobstore interface to add to blobstore FileSystems - Key: HADOOP-9565 URL: https://issues.apache.org/jira/browse/HADOOP-9565 Project: Hadoop Common Issue Type: Improvement Components: fs, fs/s3, fs/swift Affects Versions: 2.6.0 Reporter: Steve Loughran Assignee: Steve Loughran Attachments: HADOOP-9565-001.patch, HADOOP-9565-002.patch, HADOOP-9565-003.patch We can make the fact that some {{FileSystem}} implementations are really blobstores, with different atomicity and consistency guarantees, by adding a {{Blobstore}} interface to add to them. This could also be a place to add a {{Copy(Path,Path)}} method, assuming that all blobstores implement at server-side copy operation as a substitute for rename. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow
[ https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301624#comment-14301624 ] Andrew Wang commented on HADOOP-11506: -- +1 thanks Gera Configuration.get() is unnecessarily slow - Key: HADOOP-11506 URL: https://issues.apache.org/jira/browse/HADOOP-11506 Project: Hadoop Common Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Gera Shegalov Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, HADOOP-11506.003.patch Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking
[ https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated HADOOP-11104: Status: Patch Available (was: Open) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking - Key: HADOOP-11104 URL: https://issues.apache.org/jira/browse/HADOOP-11104 Project: Hadoop Common Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: HADOOP-11104.001.patch Passing a negative value to the interval field of MetricsRegistry#newQuantiles should throw a MetricsException with a clear error message. The current stack trace looks something like: java.lang.IllegalArgumentException: null at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420) at org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107) at org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200) Along similar lines, should the other methods like MetricsRegistry#newCounter() also have parameter checking for negative int/long values? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.
[ https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301613#comment-14301613 ] Chris Nauroth commented on HADOOP-11525: Thanks, Steve. I had a hunch there was an existing jira. I'm going to back off on my suggestion of a new {{FileSystem}} method for transactional put. Even if we did that, we'd still need the characteristics/semantics querying approach for other reasons, and I don't see a benefit to expanding the public API footprint in multiple different ways. Eddy, I see that the HADOOP-9565 patch does include the same basic {{FsShell}} change that you want to achieve. Are you comfortable resolving this as duplicate and participating over there? FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm. --- Key: HADOOP-11525 URL: https://issues.apache.org/jira/browse/HADOOP-11525 Project: Hadoop Common Issue Type: Improvement Components: tools Affects Versions: 2.6.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HADOOP-11525.000.patch When running {{hadoop fs -put}}, {{FsShell}} creates a {{._COPYING_.}} file on the target directory, and then renames it to target file when the write is done. However, for some targeted systems, such as S3, Azure and Swift, a partial failure write request (i.e., {{PUT}}) has not side effect, while the {{rename}} operation is expensive. {{FileSystem}} should expose some characteristics so that the operation such as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the right way to do. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow
[ https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301611#comment-14301611 ] Gera Shegalov commented on HADOOP-11506: The release audit warning is tracked by YARN-3113. The patch is reviewable, [~andrew.wang]. Configuration.get() is unnecessarily slow - Key: HADOOP-11506 URL: https://issues.apache.org/jira/browse/HADOOP-11506 Project: Hadoop Common Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Gera Shegalov Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, HADOOP-11506.003.patch Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10101) Update guava dependency to the latest version
[ https://issues.apache.org/jira/browse/HADOOP-10101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301615#comment-14301615 ] Tsuyoshi OZAWA commented on HADOOP-10101: - Thanks Steve for your review. I'll check the javac warnings. Update guava dependency to the latest version - Key: HADOOP-10101 URL: https://issues.apache.org/jira/browse/HADOOP-10101 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.2.0, 2.6.0 Reporter: Rakesh R Assignee: Vinayakumar B Attachments: HADOOP-10101-002.patch, HADOOP-10101-004.patch, HADOOP-10101-005.patch, HADOOP-10101-006.patch, HADOOP-10101-007.patch, HADOOP-10101-008.patch, HADOOP-10101.patch, HADOOP-10101.patch The existing guava version is 11.0.2 which is quite old. This -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11518) Prefix-based per-RPC server configuration
[ https://issues.apache.org/jira/browse/HADOOP-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-11518: Attachment: HADOOP-11518.v2.patch The new patch modifies {{Configuration}} class directly for the prefix support, instead of adding {{PrefixedConfiguration}}. Prefix-based per-RPC server configuration - Key: HADOOP-11518 URL: https://issues.apache.org/jira/browse/HADOOP-11518 Project: Hadoop Common Issue Type: Improvement Components: ipc Reporter: Kihwal Lee Assignee: Kihwal Lee Attachments: HADOOP-11518.patch, HADOOP-11518.v2.patch If a process runs multiple RPC servers, there is no way to individually set their IPC parameters beyond what's supported by the RPC Builder. If a configuration is shared among many nodes, this is more problematic. In this jira, a prefix-based per-server configuration is proposed. When an RPC server is built, a prefix can be specified, which will then be prepended to keys when retrieving configured values. There will be an option to fall-back to using the key without the prefix when no value is found with the prefix, thus making this feature compatible with the current way of configuring servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread
[ https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301633#comment-14301633 ] Hudson commented on HADOOP-11494: - FAILURE: Integrated in Hadoop-trunk-Commit #6977 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6977/]) HADOOP-11494. Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread. Contributed by Ted Yu. (benoy: rev 3472e3bd6c50558870b86c9ccfea5072385fa991) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SaslRpcClient.java Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread -- Key: HADOOP-11494 URL: https://issues.apache.org/jira/browse/HADOOP-11494 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch In SaslRpcClient, starting at line 576: {code} public int read(byte[] buf, int off, int len) throws IOException { synchronized(unwrappedRpcBuffer) { // fill the buffer with the next RPC message if (unwrappedRpcBuffer.remaining() == 0) { readNextRpcPacket(); } {code} readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, making the lock on previous ByteBuffer not useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11525) FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm.
[ https://issues.apache.org/jira/browse/HADOOP-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HADOOP-11525: --- Resolution: Duplicate Status: Resolved (was: Patch Available) [~stev...@iseran.com] Thanks for working on HADOOP-9565. It looks like a better solution than this one. I will close this one instead. [~cnauroth] Sure. I'd love to resolve this as duplicated. FileSystem should expose some performance characteristics for caller (e.g., FsShell) to choose the right algorithm. --- Key: HADOOP-11525 URL: https://issues.apache.org/jira/browse/HADOOP-11525 Project: Hadoop Common Issue Type: Improvement Components: tools Affects Versions: 2.6.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HADOOP-11525.000.patch When running {{hadoop fs -put}}, {{FsShell}} creates a {{._COPYING_.}} file on the target directory, and then renames it to target file when the write is done. However, for some targeted systems, such as S3, Azure and Swift, a partial failure write request (i.e., {{PUT}}) has not side effect, while the {{rename}} operation is expensive. {{FileSystem}} should expose some characteristics so that the operation such as {{CommandWithDestination#copyStreamToTarget()}} can detect and choose the right way to do. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread
[ https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-11494: -- Fix Version/s: 2.7.0 Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread -- Key: HADOOP-11494 URL: https://issues.apache.org/jira/browse/HADOOP-11494 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.7.0 Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch In SaslRpcClient, starting at line 576: {code} public int read(byte[] buf, int off, int len) throws IOException { synchronized(unwrappedRpcBuffer) { // fill the buffer with the next RPC message if (unwrappedRpcBuffer.remaining() == 0) { readNextRpcPacket(); } {code} readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, making the lock on previous ByteBuffer not useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread
[ https://issues.apache.org/jira/browse/HADOOP-11494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HADOOP-11494: -- Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk and branch-2. Thank you Ted. Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread -- Key: HADOOP-11494 URL: https://issues.apache.org/jira/browse/HADOOP-11494 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Assignee: Ted Yu Priority: Minor Fix For: 2.7.0 Attachments: hadoop-11494-001.patch, hadoop-11494-002.patch In SaslRpcClient, starting at line 576: {code} public int read(byte[] buf, int off, int len) throws IOException { synchronized(unwrappedRpcBuffer) { // fill the buffer with the next RPC message if (unwrappedRpcBuffer.remaining() == 0) { readNextRpcPacket(); } {code} readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, making the lock on previous ByteBuffer not useful. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11442) hadoop-azure: Create test jar
[ https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301768#comment-14301768 ] Hadoop QA commented on HADOOP-11442: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695981/HADOOP-11442v2.patch against trunk revision f33c99b. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-tools/hadoop-azure. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5558//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5558//console This message is automatically generated. hadoop-azure: Create test jar - Key: HADOOP-11442 URL: https://issues.apache.org/jira/browse/HADOOP-11442 Project: Hadoop Common Issue Type: Improvement Components: tools Environment: windows, azure Reporter: shashank Assignee: Chris Nauroth Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, HADOOP-11442v2.patch pom of hadoop-azure project to needs to be modified to create a test jar as well. This test jar is required to run test cases of Windowsazuretablesink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11442) hadoop-azure: Create test jar
[ https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-11442: --- Resolution: Fixed Fix Version/s: 2.7.0 Target Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Shashank, thank you for the contribution. hadoop-azure: Create test jar - Key: HADOOP-11442 URL: https://issues.apache.org/jira/browse/HADOOP-11442 Project: Hadoop Common Issue Type: Improvement Components: tools Environment: windows, azure Reporter: shashank Assignee: Chris Nauroth Fix For: 2.7.0 Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, HADOOP-11442v2.patch pom of hadoop-azure project to needs to be modified to create a test jar as well. This test jar is required to run test cases of Windowsazuretablesink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11506) Configuration.get() is unnecessarily slow
[ https://issues.apache.org/jira/browse/HADOOP-11506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301691#comment-14301691 ] Gera Shegalov commented on HADOOP-11506: Thank you, Andrew. I'll wait for a couple of days to see if there are more comments before committing. Configuration.get() is unnecessarily slow - Key: HADOOP-11506 URL: https://issues.apache.org/jira/browse/HADOOP-11506 Project: Hadoop Common Issue Type: Bug Reporter: Dmitriy V. Ryaboy Assignee: Gera Shegalov Attachments: HADOOP-11506.001.patch, HADOOP-11506.002.patch, HADOOP-11506.003.patch Profiling several large Hadoop jobs, we discovered that a surprising amount of time was spent inside Configuration.get, more specifically, in regex matching caused by the substituteVars call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11442) hadoop-azure: Create test jar
[ https://issues.apache.org/jira/browse/HADOOP-11442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-11442: --- Attachment: HADOOP-11442v2.patch [~shkhande], thank you for the update. Here is the same patch, but converted to Linux line endings. I expect Jenkins will be able to run with this. +1 pending Jenkins. hadoop-azure: Create test jar - Key: HADOOP-11442 URL: https://issues.apache.org/jira/browse/HADOOP-11442 Project: Hadoop Common Issue Type: Improvement Components: tools Environment: windows, azure Reporter: shashank Assignee: Chris Nauroth Attachments: HADOOP-11442.patch, HADOOP-11442v1.patch, HADOOP-11442v2.patch pom of hadoop-azure project to needs to be modified to create a test jar as well. This test jar is required to run test cases of Windowsazuretablesink -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11104) org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking
[ https://issues.apache.org/jira/browse/HADOOP-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301718#comment-14301718 ] Hadoop QA commented on HADOOP-11104: {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12695961/HADOOP-11104.001.patch against trunk revision ffc75d6. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:red}-1 release audit{color}. The applied patch generated 1 release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//artifact/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5556//console This message is automatically generated. org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical parameter checking - Key: HADOOP-11104 URL: https://issues.apache.org/jira/browse/HADOOP-11104 Project: Hadoop Common Issue Type: Bug Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: newbie Attachments: HADOOP-11104.001.patch Passing a negative value to the interval field of MetricsRegistry#newQuantiles should throw a MetricsException with a clear error message. The current stack trace looks something like: java.lang.IllegalArgumentException: null at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:420) at org.apache.hadoop.metrics2.lib.MutableQuantiles.init(MutableQuantiles.java:107) at org.apache.hadoop.metrics2.lib.MetricsRegistry.newQuantiles(MetricsRegistry.java:200) Along similar lines, should the other methods like MetricsRegistry#newCounter() also have parameter checking for negative int/long values? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup
[ https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301739#comment-14301739 ] Hudson commented on HADOOP-10181: - SUCCESS: Integrated in Hadoop-trunk-Commit #6979 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6979/]) HADOOP-10181. GangliaContext does not work with multicast ganglia setup. Contributed by Andrew Johnson. (cnauroth: rev 8004a002307940176cc188657c68e85171a5b5a8) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/sink/ganglia/AbstractGangliaSink.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/sink/ganglia/TestGangliaSink.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics/ganglia/TestGangliaContext.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/ganglia/package.html GangliaContext does not work with multicast ganglia setup - Key: HADOOP-10181 URL: https://issues.apache.org/jira/browse/HADOOP-10181 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.6.0 Reporter: Andrew Otto Assignee: Andrew Johnson Priority: Minor Labels: ganglia, hadoop, metrics, multicast Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, HADOOP-10181.003.patch The GangliaContext class which is used to send Hadoop metrics to Ganglia uses a DatagramSocket to send these metrics. This works fine for Ganglia multicast setups that are all on the same VLAN. However, when working with multiple VLANs, a packet sent via DatagramSocket to a multicast address will end up with a TTL of 1. Multicast TTL indicates the number of network hops for which a particular multicast packet is valid. The packets sent by GangliaContext do not make it to ganglia aggregrators on the same multicast group, but in different VLANs. To fix, we'd need a configuration property that specifies that multicast is to be used, and another that allows setting of the multicast packet TTL. With these set, we could then use MulticastSocket setTimeToLive() instead of just plain ol' DatagramSocket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup
[ https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-10181: --- Resolution: Fixed Fix Version/s: 2.7.0 Release Note: Hadoop metrics sent to Ganglia over multicast now support optional configuration of socket TTL. The default TTL is 1, which preserves the behavior of prior Hadoop versions. Clusters that span multiple subnets/VLANs will likely want to increase this. Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Andrew, thank you for contributing this patch. GangliaContext does not work with multicast ganglia setup - Key: HADOOP-10181 URL: https://issues.apache.org/jira/browse/HADOOP-10181 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: 2.6.0 Reporter: Andrew Otto Assignee: Andrew Johnson Priority: Minor Labels: ganglia, hadoop, metrics, multicast Fix For: 2.7.0 Attachments: HADOOP-10181.001.patch, HADOOP-10181.002.patch, HADOOP-10181.003.patch The GangliaContext class which is used to send Hadoop metrics to Ganglia uses a DatagramSocket to send these metrics. This works fine for Ganglia multicast setups that are all on the same VLAN. However, when working with multiple VLANs, a packet sent via DatagramSocket to a multicast address will end up with a TTL of 1. Multicast TTL indicates the number of network hops for which a particular multicast packet is valid. The packets sent by GangliaContext do not make it to ganglia aggregrators on the same multicast group, but in different VLANs. To fix, we'd need a configuration property that specifies that multicast is to be used, and another that allows setting of the multicast packet TTL. With these set, we could then use MulticastSocket setTimeToLive() instead of just plain ol' DatagramSocket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301032#comment-14301032 ] Vinayakumar B commented on HADOOP-11534: +1 for the latest changes. Will commit soon Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Improvement Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-11534: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to HDFS-EC branch Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HADOOP-11534: --- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-11264 Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11534) Minor improvements for raw erasure coders
[ https://issues.apache.org/jira/browse/HADOOP-11534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301042#comment-14301042 ] Kai Zheng commented on HADOOP-11534: Thanks [~vinayrpet] for the commit. Minor improvements for raw erasure coders - Key: HADOOP-11534 URL: https://issues.apache.org/jira/browse/HADOOP-11534 Project: Hadoop Common Issue Type: Sub-task Affects Versions: HDFS-EC Reporter: Kai Zheng Assignee: Kai Zheng Priority: Minor Fix For: HDFS-EC Attachments: HADOOP-11534-v1.patch, HADOOP-11534-v2.patch For the raw erasure coder API codes introduced by HADOOP-11514, there're some minor improvements that were noticed and can be done separately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)