[jira] [Resolved] (HADOOP-6171) package task in build.xml should copy source with preservelastmodified
[ https://issues.apache.org/jira/browse/HADOOP-6171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HADOOP-6171. Resolution: Later package task in build.xml should copy source with preservelastmodified Key: HADOOP-6171 URL: https://issues.apache.org/jira/browse/HADOOP-6171 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Todd Lipcon Assignee: Ted Yu Labels: newbie Attachments: HADOOP-6171.patch, HDFS-6171.patch, MAPRED-6171.patch In the package task, it copies the source to dist.dir without using preservelastmodified=true. This can cause issues with the autotools configure process for the C++ builds. Namely, it will sometimes think the configure script is out of date with respect to its source files and try to re-bootstrap, which relies on particular versions of autotools on the building computer. This isn't something that should be required for those wanting to build from a distribution tarball. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HADOOP-6695) Job Tracker should be able to refresh classpath without restarting
Job Tracker should be able to refresh classpath without restarting -- Key: HADOOP-6695 URL: https://issues.apache.org/jira/browse/HADOOP-6695 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.20.1 Reporter: Ted Yu When I tried to run HBase export in cluster mode, I copied zookeeper-3.2.1.jar to hadoop lib directory. But initial attempts failed with java.lang.ClassNotFoundException because job tracker wasn't restarted. It is desirable for job tracker to pick up new classpath without restarting which is sometimes hard to do in production environment. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6729) serializer.JavaSerialization should be added to io.serializations by default
serializer.JavaSerialization should be added to io.serializations by default Key: HADOOP-6729 URL: https://issues.apache.org/jira/browse/HADOOP-6729 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.20.2 Reporter: Ted Yu org.apache.hadoop.io.serializer.JavaSerialization isn't included in io.serializations by default. When a class which implements the Serializable interface is used, user would see the following without serializer.JavaSerialization: java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:759) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:487) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:575) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.Child.main(Child.java:170) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6749) Add mapred.tasktracker.tasks.maximum to limit the total number of map and reduce tasks
Add mapred.tasktracker.tasks.maximum to limit the total number of map and reduce tasks -- Key: HADOOP-6749 URL: https://issues.apache.org/jira/browse/HADOOP-6749 Project: Hadoop Common Issue Type: New Feature Components: conf Affects Versions: 0.20.2 Reporter: Ted Yu We have mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum now. It is desirable to have mapred.tasktracker.tasks.maximum which limits the total number of map and reduce tasks. Meaning its value may be lower than (mapred.tasktracker.map.tasks.maximum + mapred.tasktracker.reduce.tasks.maximum) This would be useful in a cluster whose nodes are multi-core. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6774) Namenode is not able to recover from disk full condition
Namenode is not able to recover from disk full condition Key: HADOOP-6774 URL: https://issues.apache.org/jira/browse/HADOOP-6774 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.20.2 Environment: Linux sjc9-flash-grid00.ciq.com 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux Reporter: Ted Yu We ran an internal flow which resulted in: Exception in thread main java.lang.RuntimeException: initialization of flow executor failed After that we freed disk space on the Namenode server, but restarting Namenode failed. Here is from Namenode log: 2010-05-19 17:15:15,514 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: sjc1-qa-certiq1.sjc1.ciq.com/10.201.8.247:9000 2010-05-19 17:15:15,516 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null 2010-05-19 17:15:15,518 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hadoop,hadoop 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup 2010-05-19 17:15:15,579 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true 2010-05-19 17:15:15,588 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext 2010-05-19 17:15:15,590 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2010-05-19 17:15:15,637 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1874 2010-05-19 17:15:16,202 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 2 2010-05-19 17:15:16,204 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 259450 loaded in 0 seconds. 2010-05-19 17:15:16,599 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NumberFormatException: For input string: at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) at java.lang.Long.parseLong(Long.java:431) at java.lang.Long.parseLong(Long.java:468) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.readLong(FSEditLog.java:1273) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:656) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:999) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:88) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:312) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.init(FSNamesystem.java:293) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:224) at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:306) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1004) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1013) 2010-05-19 17:15:16,599 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6804) allow FileSystem.copyFromLocalFile() to execute under specified username
allow FileSystem.copyFromLocalFile() to execute under specified username Key: HADOOP-6804 URL: https://issues.apache.org/jira/browse/HADOOP-6804 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 0.20.2 Reporter: Ted Yu Priority: Minor When the user calling FileSystem.copyFromLocalFile() doesn't have permission to write to certain hdfs path: Thread [main] (Suspended (exception AccessControlException)) DFSClient.mkdirs(String, FsPermission) line: 905 DistributedFileSystem.mkdirs(Path, FsPermission) line: 262 DistributedFileSystem(FileSystem).mkdirs(Path) line: 1162 FileUtil.copy(FileSystem, Path, FileSystem, Path, boolean, boolean, Configuration) line: 194 DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, boolean, Path, Path) line: 1231 DistributedFileSystem(FileSystem).copyFromLocalFile(boolean, Path, Path) line: 1207 DistributedFileSystem(FileSystem).copyFromLocalFile(Path, Path) line: 1179 GridM2mInstallation.copyInputFiles(FlowConfigurations$FlowConf) line: 380 Passwordless ssh has been setup for current user, tyu, on localhost and user hadoop on hdfs FileSystem.copyFromLocalFile() should be able to execute using a different username than effective user on localhost so that AccessControlException can be avoided. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6816) enhance FsShell.dus() to include sum of totalSize
enhance FsShell.dus() to include sum of totalSize - Key: HADOOP-6816 URL: https://issues.apache.org/jira/browse/HADOOP-6816 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 0.20.2 Reporter: Ted Yu Fix For: 0.20.3 FsShell.dus() prints out totalSize for each file found. It would be desirable to print sum of totalSize's at the end. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] [Created] (HADOOP-7319) Coarse-grained dynamic configuration changes
Coarse-grained dynamic configuration changes Key: HADOOP-7319 URL: https://issues.apache.org/jira/browse/HADOOP-7319 Project: Hadoop Common Issue Type: Improvement Components: conf Reporter: Ted Yu HADOOP-7001 introduced mechanism for performing dynamic configuration changes where reconfigureProperty()/reconfigurePropertyImpl() only notifies single property change. Normally, components which use ReconfigurableBase would involve several related properties whose update should be done atomically. This JIRA provides coarse-grained dynamic configuration changes with the following benefits: 1. consistency updating related properties dynamically 2. reduction of lock contention when multiple properties are changed in proximity -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-10137) TestRMNMSecretKeys#testNMUpdation fails with NullPointerException
Ted Yu created HADOOP-10137:
---
Summary: TestRMNMSecretKeys#testNMUpdation fails with
NullPointerException
Key: HADOOP-10137
URL: https://issues.apache.org/jira/browse/HADOOP-10137
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Here is the stack trace:
{code}
testNMUpdation(org.apache.hadoop.yarn.server.TestRMNMSecretKeys) Time elapsed:
2.704 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:820)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:471)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:844)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.transitionToActive(RMHAProtocolService.java:187)
at
org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService.serviceStart(RMHAProtocolService.java:101)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:871)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.validateRMNMKeyExchange(TestRMNMSecretKeys.java:69)
at
org.apache.hadoop.yarn.server.TestRMNMSecretKeys.testNMUpdation(TestRMNMSecretKeys.java:49)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HADOOP-10163) Enhance jenkinsPrecommitAdmin.py to pass attachment Id for last tested patch
Ted Yu created HADOOP-10163: --- Summary: Enhance jenkinsPrecommitAdmin.py to pass attachment Id for last tested patch Key: HADOOP-10163 URL: https://issues.apache.org/jira/browse/HADOOP-10163 Project: Hadoop Common Issue Type: Improvement Reporter: Ted Yu In HBASE-10044, attempt was made to filter attachments according to known file extensions. However, that change alone wouldn't work because when non-patch is attached, QA bot doesn't provide attachment Id for last tested patch. This results in the modified test-patch.sh to seek backward and launch duplicate test run for last tested patch. If attachment Id for last tested patch is provided, test-patch.sh can decide whether there is need to run test. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HADOOP-10165) TestMetricsSystemImpl#testMultiThreadedPublish occasionally fails
Ted Yu created HADOOP-10165:
---
Summary: TestMetricsSystemImpl#testMultiThreadedPublish
occasionally fails
Key: HADOOP-10165
URL: https://issues.apache.org/jira/browse/HADOOP-10165
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
From
https://builds.apache.org/job/Hadoop-Common-trunk/982/testReport/junit/org.apache.hadoop.metrics2.impl/TestMetricsSystemImpl/testMultiThreadedPublish/
:
{code}
Error Message
Passed
Passed
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
Stacktrace
java.lang.AssertionError: Passed
Passed
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Metric not collected!
Passed
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:233)
Standard Output
2013-12-15 09:14:49,144 INFO impl.MetricsConfig
(MetricsConfig.java:loadFirst(111)) - loaded properties from
hadoop-metrics2-test.properties
2013-12-15 09:14:49,146 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:startTimer(341)) - Scheduled snapshot period at 80
second(s).
2013-12-15 09:14:49,146 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:start(183)) - Test metrics system started
2013-12-15 09:14:49,147 INFO impl.MetricsSinkAdapter
(MetricsSinkAdapter.java:start(190)) - Sink Collector started
2013-12-15 09:14:49,147 INFO impl.MetricsSystemImpl
(MetricsSystemImpl.java:registerSink(275)) - Registered sink Collector
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Resolved] (HADOOP-10163) Attachment Id for last tested patch should be passed to test-patch.sh
[ https://issues.apache.org/jira/browse/HADOOP-10163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HADOOP-10163. - Resolution: Later Attachment Id for last tested patch should be passed to test-patch.sh - Key: HADOOP-10163 URL: https://issues.apache.org/jira/browse/HADOOP-10163 Project: Hadoop Common Issue Type: Improvement Reporter: Ted Yu In HBASE-10044, attempt was made to filter attachments according to known file extensions. However, that change alone wouldn't work because when non-patch is attached, QA bot doesn't provide attachment Id for last tested patch. This results in the modified test-patch.sh to seek backward and launch duplicate test run for last tested patch. If attachment Id for last tested patch is provided, test-patch.sh can decide whether there is need to run test. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HADOOP-10202) OK_JAVADOC_WARNINGS is out of date, leading to negative javadoc warning count
Ted Yu created HADOOP-10202:
---
Summary: OK_JAVADOC_WARNINGS is out of date, leading to negative
javadoc warning count
Key: HADOOP-10202
URL: https://issues.apache.org/jira/browse/HADOOP-10202
Project: Hadoop Common
Issue Type: Task
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/PreCommit-HDFS-Build/5813//testReport/ :
{code}
-1 javadoc. The javadoc tool appears to have generated -14 warning messages.
{code}
OK_JAVADOC_WARNINGS should be updated.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HADOOP-10223) MiniKdc#main() should close the FileReader it creates
Ted Yu created HADOOP-10223: --- Summary: MiniKdc#main() should close the FileReader it creates Key: HADOOP-10223 URL: https://issues.apache.org/jira/browse/HADOOP-10223 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Priority: Minor Attachments: hadoop-10223.txt FileReader is used to read MiniKDC properties. This FileReader should be closed after reading. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HADOOP-10316) HadoopArchives#HArchiveInputFormat#getSplits() should check reader against null before calling close()
Ted Yu created HADOOP-10316:
---
Summary: HadoopArchives#HArchiveInputFormat#getSplits() should
check reader against null before calling close()
Key: HADOOP-10316
URL: https://issues.apache.org/jira/browse/HADOOP-10316
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Around line 267:
{code}
finally {
reader.close();
}
{code}
reader should be checked against null
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HADOOP-10318) Incorrect reference to nodeFile in RumenToSLSConverter error message
Ted Yu created HADOOP-10318:
---
Summary: Incorrect reference to nodeFile in RumenToSLSConverter
error message
Key: HADOOP-10318
URL: https://issues.apache.org/jira/browse/HADOOP-10318
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
if (! nodeFile.getParentFile().exists()
! nodeFile.getParentFile().mkdirs()) {
System.err.println(ERROR: Cannot create output directory in path:
+ jsonFile.getParentFile().getAbsoluteFile());
{code}
jsonFile on the last line should be nodeFile
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HADOOP-10331) SwiftRestClient#buildException() references wrong length
Ted Yu created HADOOP-10331:
---
Summary: SwiftRestClient#buildException() references wrong length
Key: HADOOP-10331
URL: https://issues.apache.org/jira/browse/HADOOP-10331
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Header availableContentRange = method.getResponseHeader(
HEADER_CONTENT_RANGE);
if (requestContentLen!=null) {
errorText.append( available
).append(availableContentRange.getValue());
}
{code}
availableContentRange should be checked instead.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HADOOP-10404) Some accesses to DomainSocketWatcher#closed are not protected by lock
Ted Yu created HADOOP-10404:
---
Summary: Some accesses to DomainSocketWatcher#closed are not
protected by lock
Key: HADOOP-10404
URL: https://issues.apache.org/jira/browse/HADOOP-10404
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
* Lock which protects toAdd, toRemove, and closed.
*/
private final ReentrantLock lock = new ReentrantLock();
{code}
There're two places, NotificationHandler.handle() and kick(), where access to
closed is without holding lock.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10501) Server#getHandlers() accesses handlers without synchronization
Ted Yu created HADOOP-10501:
---
Summary: Server#getHandlers() accesses handlers without
synchronization
Key: HADOOP-10501
URL: https://issues.apache.org/jira/browse/HADOOP-10501
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Iterable? extends Thread getHandlers() {
return Arrays.asList(handlers);
}
{code}
All the other methods accessing handlers are synchronized methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10541) InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not closed
Ted Yu created HADOOP-10541:
---
Summary: InputStream in MiniKdc#initKDCServer for minikdc.ldiff is
not closed
Key: HADOOP-10541
URL: https://issues.apache.org/jira/browse/HADOOP-10541
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
The same InputStream variable is used for minikdc.ldiff and minikdc-krb5.conf :
{code}
InputStream is = cl.getResourceAsStream(minikdc.ldiff);
...
is = cl.getResourceAsStream(minikdc-krb5.conf);
{code}
Before the second assignment, is should be closed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10542) Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
Ted Yu created HADOOP-10542:
---
Summary: Potential null pointer dereference in
Jets3tFileSystemStore#retrieveBlock()
Key: HADOOP-10542
URL: https://issues.apache.org/jira/browse/HADOOP-10542
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
in = get(blockToKey(block), byteRangeStart);
out = new BufferedOutputStream(new FileOutputStream(fileBlock));
byte[] buf = new byte[bufferSize];
int numRead;
while ((numRead = in.read(buf)) = 0) {
{code}
get() may return null.
The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10615) FileInputStream in JenkinsHash#main() is never closed
Ted Yu created HADOOP-10615:
---
Summary: FileInputStream in JenkinsHash#main() is never closed
Key: HADOOP-10615
URL: https://issues.apache.org/jira/browse/HADOOP-10615
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
FileInputStream in = new FileInputStream(args[0]);
{code}
The above FileInputStream is not closed upon exit of main.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10642) Provide option to limit heap memory consumed by dynamic metrics2 metrics
Ted Yu created HADOOP-10642:
---
Summary: Provide option to limit heap memory consumed by dynamic
metrics2 metrics
Key: HADOOP-10642
URL: https://issues.apache.org/jira/browse/HADOOP-10642
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
User sunweiei provided the following jmap output in HBase 0.96 deployment:
{code}
num #instances #bytes class name
--
1: 14917882 3396492464 [C
2: 1996994 2118021808 [B
3: 43341650 1733666000 java.util.LinkedHashMap$Entry
4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
5: 14446577 924580928
org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
{code}
Heap consumption by Interns$CacheWith2Keys$2 (and indirectly by [C) could be
due to calls to Interns.info() in DynamicMetricsRegistry which was cloned off
metrics2/lib/MetricsRegistry.java.
This scenario would arise when large number of regions are tracked through
metrics2 dynamically.
Interns class doesn't provide API to remove entries in its internal Map.
One solution is to provide an option that allows skipping calls to
Interns.info() in metrics2/lib/MetricsRegistry.java
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10660) GraphiteSink should implement Closeable
Ted Yu created HADOOP-10660: --- Summary: GraphiteSink should implement Closeable Key: HADOOP-10660 URL: https://issues.apache.org/jira/browse/HADOOP-10660 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu GraphiteSink wraps OutputStreamWriter around socket's output stream. Currently the socket is never closed. GraphiteSink should implement Closeable such that MetricsSystem can close the socket when it is stopped. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10668) TestZKFailoverControllerStress#testExpireBackAndForth occasionally fails
Ted Yu created HADOOP-10668:
---
Summary: TestZKFailoverControllerStress#testExpireBackAndForth
occasionally fails
Key: HADOOP-10668
URL: https://issues.apache.org/jira/browse/HADOOP-10668
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From
https://builds.apache.org/job/PreCommit-HADOOP-Build/4018//testReport/org.apache.hadoop.ha/TestZKFailoverControllerStress/testExpireBackAndForth/
:
{code}
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
at org.apache.zookeeper.server.DataTree.getData(DataTree.java:648)
at org.apache.zookeeper.server.ZKDatabase.getData(ZKDatabase.java:371)
at
org.apache.hadoop.ha.MiniZKFCCluster.expireActiveLockHolder(MiniZKFCCluster.java:199)
at
org.apache.hadoop.ha.MiniZKFCCluster.expireAndVerifyFailover(MiniZKFCCluster.java:234)
at
org.apache.hadoop.ha.TestZKFailoverControllerStress.testExpireBackAndForth(TestZKFailoverControllerStress.java:84)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10690) Lack of synchronization on access to InputStream in NativeAzureFileSystem#NativeAzureFsInputStream#close()
Ted Yu created HADOOP-10690:
---
Summary: Lack of synchronization on access to InputStream in
NativeAzureFileSystem#NativeAzureFsInputStream#close()
Key: HADOOP-10690
URL: https://issues.apache.org/jira/browse/HADOOP-10690
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void close() throws IOException {
in.close();
}
{code}
The close() method should be protected by synchronized keyword.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10715) Make GraphiteSink#setWriter() private
Ted Yu created HADOOP-10715: --- Summary: Make GraphiteSink#setWriter() private Key: HADOOP-10715 URL: https://issues.apache.org/jira/browse/HADOOP-10715 Project: Hadoop Common Issue Type: Task Reporter: Ted Yu Priority: Minor During review of HADOOP-10660, Ravi brought up the notion of making GraphiteSink#setWriter() private. This JIRA is to address Ravi's comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10732) Update without holding write lock in JavaKeyStoreProvider#innerSetCredential()
Ted Yu created HADOOP-10732: --- Summary: Update without holding write lock in JavaKeyStoreProvider#innerSetCredential() Key: HADOOP-10732 URL: https://issues.apache.org/jira/browse/HADOOP-10732 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu Priority: Minor In hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/JavaKeyStoreProvider.java, innerSetCredential() doesn't wrap update with writeLock.lock() / writeLock.unlock(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10751) HBase Master needs data locality
Ted Yu created HADOOP-10751:
---
Summary: HBase Master needs data locality
Key: HADOOP-10751
URL: https://issues.apache.org/jira/browse/HADOOP-10751
Project: Hadoop Common
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Currently:
{code}
// Master doesn't need data locality
ROLES.add(new ProviderRole(HBaseKeys.ROLE_MASTER,
KEY_MASTER,PlacementPolicy.NO_DATA_LOCALITY));
{code}
But in RoleHistory#findNodeForNewInstance():
{code}
if (role.getNoDataLocality()) {
return null;
}
{code}
This implies that HBase master instances might be scheduled on one host,
obviating the goal for HA.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-10763) TestZKFailoverControllerStress#testExpireBackAndForth fails occasionally in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-10763.
-
Resolution: Duplicate
Dup of HADOOP-10668
TestZKFailoverControllerStress#testExpireBackAndForth fails occasionally in
trunk
-
Key: HADOOP-10763
URL: https://issues.apache.org/jira/browse/HADOOP-10763
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
See https://builds.apache.org/job/Hadoop-Common-trunk/1153/console
I was able to reproduce on Mac:
{code}
Running org.apache.hadoop.ha.TestZKFailoverControllerStress
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 87.907 sec
FAILURE! - in org.apache.hadoop.ha.TestZKFailoverControllerStress
testExpireBackAndForth(org.apache.hadoop.ha.TestZKFailoverControllerStress)
Time elapsed: 25.038 sec ERROR!
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
at org.apache.zookeeper.server.DataTree.getData(DataTree.java:648)
at org.apache.zookeeper.server.ZKDatabase.getData(ZKDatabase.java:371)
at
org.apache.hadoop.ha.MiniZKFCCluster.expireActiveLockHolder(MiniZKFCCluster.java:199)
at
org.apache.hadoop.ha.MiniZKFCCluster.expireAndVerifyFailover(MiniZKFCCluster.java:234)
at
org.apache.hadoop.ha.TestZKFailoverControllerStress.testExpireBackAndForth(TestZKFailoverControllerStress.java:79)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10920) site plugin couldn't parse index.apt.vm
Ted Yu created HADOOP-10920:
---
Summary: site plugin couldn't parse index.apt.vm
Key: HADOOP-10920
URL: https://issues.apache.org/jira/browse/HADOOP-10920
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
From log of https://builds.apache.org/job/Hadoop-Common-trunk/1193 :
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (docs) on project
hadoop-kms: Error during page generation: Error parsing
'https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm':
line [126] expected SECTION2, found SECTION3 - [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (docs) on project
hadoop-kms: Error during page generation
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:76)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:116)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:361)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:213)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error during page
generation
at org.apache.maven.plugins.site.SiteMojo.execute(SiteMojo.java:143)
at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:133)
at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
... 19 more
Caused by: org.apache.maven.doxia.siterenderer.RendererException: Error parsing
'https://builds.apache.org/job/Hadoop-Common-trunk/ws/trunk/hadoop-common-project/hadoop-kms/src/site/apt/index.apt.vm':
line [126] expected SECTION2, found SECTION3
at
org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.renderDocument(DefaultSiteRenderer.java:414)
at
org.apache.maven.doxia.siterenderer.DoxiaDocumentRenderer.renderDocument(DoxiaDocumentRenderer.java:53)
at
org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.renderModule(DefaultSiteRenderer.java:319)
at
org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.render(DefaultSiteRenderer.java:135)
at
org.apache.maven.plugins.site.SiteMojo.renderLocale(SiteMojo.java:175)
at org.apache.maven.plugins.site.SiteMojo.execute(SiteMojo.java:138)
... 21 more
Caused by: org.apache.maven.doxia.module.apt.AptParseException: expected
SECTION2, found SECTION3
at org.apache.maven.doxia.module.apt.AptParser.parse(AptParser.java:235)
at org.apache.maven.doxia.DefaultDoxia.parse(DefaultDoxia.java:65)
at
org.apache.maven.doxia.siterenderer.DefaultSiteRenderer.renderDocument(DefaultSiteRenderer.java:406)
... 26 more
Caused by: org.apache.maven.doxia.module.apt.AptParseException: expected
SECTION2, found SECTION3
at
org.apache.maven.doxia.module.apt.AptParser.expectedBlock(AptParser.java:1404)
at
org.apache.maven.doxia.module.apt.AptParser.traverseSection(AptParser.java:787)
at
org.apache.maven.doxia.module.apt.AptParser.traverseSection(AptParser.java:823)
at
org.apache.maven.doxia.module.apt.AptParser.traverseBody(AptParser.java:765)
at org.apache.maven.doxia.module.apt.AptParser.parse(AptParser.java:230)
... 28 more
[ERROR]
{code}
--
This message was sent by
[jira] [Created] (HADOOP-10980) TestActiveStandbyElector fails occasionally in trunk
Ted Yu created HADOOP-10980:
---
Summary: TestActiveStandbyElector fails occasionally in trunk
Key: HADOOP-10980
URL: https://issues.apache.org/jira/browse/HADOOP-10980
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Common-trunk/1211/consoleFull :
{code}
Running org.apache.hadoop.ha.TestActiveStandbyElector
Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.7 sec
FAILURE! - in org.apache.hadoop.ha.TestActiveStandbyElector
testWithoutZKServer(org.apache.hadoop.ha.TestActiveStandbyElector) Time
elapsed: 0.051 sec FAILURE!
java.lang.AssertionError: Did not throw zookeeper connection loss exceptions!
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.ha.TestActiveStandbyElector.testWithoutZKServer(TestActiveStandbyElector.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-11014) Potential resource leak in JavaKeyStoreProvider due to unclosed stream
Ted Yu created HADOOP-11014:
---
Summary: Potential resource leak in JavaKeyStoreProvider due to
unclosed stream
Key: HADOOP-11014
URL: https://issues.apache.org/jira/browse/HADOOP-11014
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
From
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/JavaKeyStoreProvider.java
:
{code}
private void writeToNew(Path newPath) throws IOException {
FSDataOutputStream out =
FileSystem.create(fs, newPath, permissions);
try {
keyStore.store(out, password);
} catch (KeyStoreException e) {
throw new IOException(Can't store keystore + this, e);
} catch (NoSuchAlgorithmException e) {
throw new IOException(
No such algorithm storing keystore + this, e);
} catch (CertificateException e) {
throw new IOException(
Certificate exception storing keystore + this, e);
}
out.close();
{code}
IOException is not among the catch blocks.
According to
http://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#store(java.io.OutputStream,%20char[]),
IOException may be thrown from the store() call. In that case, out would be
left unclosed.
In loadFromPath():
{code}
keyStore.load(fs.open(p), password);
{code}
The InputStream should be closed upon return from load()
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-11162) Unclosed InputStream in ApplicationClassLoader
Ted Yu created HADOOP-11162:
---
Summary: Unclosed InputStream in ApplicationClassLoader
Key: HADOOP-11162
URL: https://issues.apache.org/jira/browse/HADOOP-11162
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
static {
InputStream is = null;
{code}
The above InputStream is not closed upon leaving the static block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11165) TestUTF8 fails when run against java 8
Ted Yu created HADOOP-11165:
---
Summary: TestUTF8 fails when run against java 8
Key: HADOOP-11165
URL: https://issues.apache.org/jira/browse/HADOOP-11165
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Using jdk1.8.0_20 , I got:
{code}
testGetBytes(org.apache.hadoop.io.TestUTF8) Time elapsed: 0.007 sec
FAILURE!
junit.framework.ComparisonFailure:
expected:쑼ь⣄鬘㟻햫紖燺[?炀⑰풸낓⨵ἲꬌホ쭷㛕曬䟊⁍䴥䳠領蟭뱻宭竕昚鍳튇ꊕ혶齲쏈㠮胨䩦隼䍻킿喝벁ࢼ듿饭玳Մ剌䒤?䳛슟녚沖᯳?訨
牙⍖?䎠旘薑春觀葝礫⁑ﻱ⣽゚굿뒦ݦ︀偆?]古絥萟浐 but
was:쑼ь⣄鬘㟻햫紖燺[�炀⑰풸낓⨵ἲꬌホ쭷㛕曬䟊⁍䴥䳠領蟭뱻宭竕昚鍳튇ꊕ혶齲쏈㠮胨䩦隼䍻킿喝벁ࢼ듿饭玳Մ剌䒤�䳛슟녚᯳�訨牙⍖�䎠旘薑春觀葝礫⁑ﻱ⣽゚굿뒦ݦ︀偆�]古絥萟浐
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at junit.framework.TestCase.assertEquals(TestCase.java:269)
at org.apache.hadoop.io.TestUTF8.testGetBytes(TestUTF8.java:58)
testIO(org.apache.hadoop.io.TestUTF8) Time elapsed: 0.002 sec FAILURE!
junit.framework.ComparisonFailure: expected:...ᨍ⁖粩⧬车﹂脖朷䝄懒댵突疼資⍣眠畠忁[?]䪐ゑ鬍鍅遻ꈸ釡
but was:...ᨍ⁖粩⧬车﹂脖朷䝄懒댵突疼資⍣眠畠忁[�]䪐ゑ鬍鍅遻ꈸ釡
at junit.framework.Assert.assertEquals(Assert.java:100)
at junit.framework.Assert.assertEquals(Assert.java:107)
at junit.framework.TestCase.assertEquals(TestCase.java:269)
at org.apache.hadoop.io.TestUTF8.testIO(TestUTF8.java:86)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11190) Potentially stale value is used in SelfRenewingLease ctor
Ted Yu created HADOOP-11190:
---
Summary: Potentially stale value is used in SelfRenewingLease ctor
Key: HADOOP-11190
URL: https://issues.apache.org/jira/browse/HADOOP-11190
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is w.r.t. threadNumber, shown in the code around line 102:
{code}
renewer.setName(AzureLeaseRenewer- + threadNumber++);
{code}
Since there is no synchronization involved, potentially stale value may be read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11191) NativeAzureFileSystem#close() should be synchronized
Ted Yu created HADOOP-11191:
---
Summary: NativeAzureFileSystem#close() should be synchronized
Key: HADOOP-11191
URL: https://issues.apache.org/jira/browse/HADOOP-11191
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void close() throws IOException {
in.close();
closed = true;
}
{code}
The other methods, such as seek(), are synchronized.
close() should be as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11198) Typo in javadoc for FileSystem#listStatus()
Ted Yu created HADOOP-11198:
---
Summary: Typo in javadoc for FileSystem#listStatus()
Key: HADOOP-11198
URL: https://issues.apache.org/jira/browse/HADOOP-11198
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
* @return the statuses of the files/directories in the given patch
{code}
'patch' should be path
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11229) JobStoryProducer is not closed upon return from Gridmix#setupDistCacheEmulation()
Ted Yu created HADOOP-11229:
---
Summary: JobStoryProducer is not closed upon return from
Gridmix#setupDistCacheEmulation()
Key: HADOOP-11229
URL: https://issues.apache.org/jira/browse/HADOOP-11229
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is related code:
{code}
JobStoryProducer jsp = createJobStoryProducer(traceIn, conf);
exitCode = distCacheEmulator.setupGenerateDistCacheData(jsp);
{code}
jsp should be closed upon return from setupDistCacheEmulation().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11275) TestSSLFactory fails on Java 8
Ted Yu created HADOOP-11275:
---
Summary: TestSSLFactory fails on Java 8
Key: HADOOP-11275
URL: https://issues.apache.org/jira/browse/HADOOP-11275
Project: Hadoop Common
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
Below are a few of the exceptions I got running this test against Java 8:
{code}
Running org.apache.hadoop.security.ssl.TestSSLFactory
Tests run: 15, Failures: 0, Errors: 14, Skipped: 0, Time elapsed: 1.724 sec
FAILURE! - in org.apache.hadoop.security.ssl.TestSSLFactory
testNoClientCertsInitialization(org.apache.hadoop.security.ssl.TestSSLFactory)
Time elapsed: 0.177 sec ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:96)
at
org.apache.hadoop.security.ssl.KeyStoreTestUtil.setupSSLConfig(KeyStoreTestUtil.java:268)
at
org.apache.hadoop.security.ssl.TestSSLFactory.createConfiguration(TestSSLFactory.java:64)
at
org.apache.hadoop.security.ssl.TestSSLFactory.testNoClientCertsInitialization(TestSSLFactory.java:337)
testServerKeyPasswordDefaultsToPassword(org.apache.hadoop.security.ssl.TestSSLFactory)
Time elapsed: 0.189 sec ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:96)
at
org.apache.hadoop.security.ssl.TestSSLFactory.checkSSLFactoryInitWithPasswords(TestSSLFactory.java:283)
at
org.apache.hadoop.security.ssl.TestSSLFactory.checkSSLFactoryInitWithPasswords(TestSSLFactory.java:248)
at
org.apache.hadoop.security.ssl.TestSSLFactory.testServerKeyPasswordDefaultsToPassword(TestSSLFactory.java:205)
testServerCredProviderPasswords(org.apache.hadoop.security.ssl.TestSSLFactory)
Time elapsed: 0.462 sec ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:96)
at
org.apache.hadoop.security.ssl.TestSSLFactory.checkSSLFactoryInitWithPasswords(TestSSLFactory.java:283)
at
org.apache.hadoop.security.ssl.TestSSLFactory.testServerCredProviderPasswords(TestSSLFactory.java:224)
testClientDifferentPasswordAndKeyPassword(org.apache.hadoop.security.ssl.TestSSLFactory)
Time elapsed: 0.059 sec ERROR!
java.security.cert.CertificateException: Subject class type invalid.
at sun.security.x509.X509CertInfo.setSubject(X509CertInfo.java:888)
at sun.security.x509.X509CertInfo.set(X509CertInfo.java:415)
at
org.apache.hadoop.security.ssl.KeyStoreTestUtil.generateCertificate(KeyStoreTestUtil.java:96)
at
org.apache.hadoop.security.ssl.TestSSLFactory.checkSSLFactoryInitWithPasswords(TestSSLFactory.java:283)
at
org.apache.hadoop.security.ssl.TestSSLFactory.checkSSLFactoryInitWithPasswords(TestSSLFactory.java:248)
at
org.apache.hadoop.security.ssl.TestSSLFactory.testClientDifferentPasswordAndKeyPassword(TestSSLFactory.java:211)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11283) Potentially unclosed SequenceFile.Writer in DistCpV1#setup()
Ted Yu created HADOOP-11283:
---
Summary: Potentially unclosed SequenceFile.Writer in
DistCpV1#setup()
Key: HADOOP-11283
URL: https://issues.apache.org/jira/browse/HADOOP-11283
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
SequenceFile.Writer src_writer = SequenceFile.createWriter(jobfs, jobConf,
srcfilelist, LongWritable.class, FilePair.class,
SequenceFile.CompressionType.NONE);
Path dstfilelist = new Path(jobDirectory, _distcp_dst_files);
SequenceFile.Writer dst_writer = SequenceFile.createWriter(jobfs, jobConf,
dstfilelist, Text.class, Text.class,
SequenceFile.CompressionType.NONE);
{code}
If creation of dst_writer throws exception, src_writer would be left unclosed
since there is no finally clause doing that for the above code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11414) Close of Reader should be enclosed in finally block in FileBasedIPList#readLines()
Ted Yu created HADOOP-11414:
---
Summary: Close of Reader should be enclosed in finally block in
FileBasedIPList#readLines()
Key: HADOOP-11414
URL: https://issues.apache.org/jira/browse/HADOOP-11414
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
Reader fileReader = new InputStreamReader(
new FileInputStream(file), Charsets.UTF_8);
BufferedReader bufferedReader = new BufferedReader(fileReader);
ListString lines = new ArrayListString();
String line = null;
while ((line = bufferedReader.readLine()) != null) {
lines.add(line);
}
bufferedReader.close();
{code}
Since bufferedReader.readLine() may throw IOE, so the close of bufferedReader
should be enclosed within finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11446) S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
Ted Yu created HADOOP-11446:
---
Summary: S3AOutputStream should use shared thread pool to avoid
OutOfMemoryError
Key: HADOOP-11446
URL: https://issues.apache.org/jira/browse/HADOOP-11446
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Here is part of the output including the OOME when hbase snapshot is exported
to s3a (nofile ulimit was increased to 102400):
{code}
2014-12-19 13:15:03,895 INFO [main] s3a.S3AFileSystem: OutputStream for key
'FastQueryPOC/2014-12-11/EVENT1-IDX-snapshot/.hbase-snapshot/.tmp/EVENT1_IDX_snapshot_2012_12_11/
650a5678810fbdaa91809668d11ccf09/.regioninfo' closed. Now beginning upload
2014-12-19 13:15:03,895 INFO [main] s3a.S3AFileSystem: Minimum upload part
size: 16777216 threshold2147483647
Exception in thread main java.lang.OutOfMemoryError: unable to create new
native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:713)
at
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at
com.amazonaws.services.s3.transfer.internal.UploadMonitor.init(UploadMonitor.java:129)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:449)
at
com.amazonaws.services.s3.transfer.TransferManager.upload(TransferManager.java:382)
at
org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:127)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:54)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:791)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:882)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:886)
{code}
In S3AOutputStream#close():
{code}
TransferManager transfers = new TransferManager(client);
{code}
This results in each TransferManager creating its own thread pool, leading to
the OOME.
One solution is to pass shared thread pool to TransferManager.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11191) NativeAzureFileSystem#close() should be synchronized
[
https://issues.apache.org/jira/browse/HADOOP-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-11191.
-
Resolution: Later
NativeAzureFileSystem#close() should be synchronized
Key: HADOOP-11191
URL: https://issues.apache.org/jira/browse/HADOOP-11191
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
public void close() throws IOException {
in.close();
closed = true;
}
{code}
The other methods, such as seek(), are synchronized.
close() should be as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11480) Typo in hadoop-aws/index.md uses wrong scheme for test.fs.s3.name
Ted Yu created HADOOP-11480:
---
Summary: Typo in hadoop-aws/index.md uses wrong scheme for
test.fs.s3.name
Key: HADOOP-11480
URL: https://issues.apache.org/jira/browse/HADOOP-11480
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
Around line 270:
{code}
property
nametest.fs.s3.name/name
values3a://test-aws-s3//value
/property
{code}
The scheme should be s3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11494) Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread
Ted Yu created HADOOP-11494:
---
Summary: Lock acquisition on WrappedInputStream#unwrappedRpcBuffer
may race with another thread
Key: HADOOP-11494
URL: https://issues.apache.org/jira/browse/HADOOP-11494
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
In SaslRpcClient, starting at line 576:
{code}
public int read(byte[] buf, int off, int len) throws IOException {
synchronized(unwrappedRpcBuffer) {
// fill the buffer with the next RPC message
if (unwrappedRpcBuffer.remaining() == 0) {
readNextRpcPacket();
}
{code}
readNextRpcPacket() may assign another ByteBuffer to unwrappedRpcBuffer, making
the lock on previous ByteBuffer not useful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11499) Check of executorThreadsStarted in ValueQueue#submitRefillTask() evades lock acquisition
Ted Yu created HADOOP-11499:
---
Summary: Check of executorThreadsStarted in
ValueQueue#submitRefillTask() evades lock acquisition
Key: HADOOP-11499
URL: https://issues.apache.org/jira/browse/HADOOP-11499
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
{code}
if (!executorThreadsStarted) {
synchronized (this) {
// To ensure all requests are first queued, make coreThreads =
// maxThreads
// and pre-start all the Core Threads.
executor.prestartAllCoreThreads();
executorThreadsStarted = true;
}
}
{code}
It is possible that two threads executing the above code both see
executorThreadsStarted as being false, leading to
executor.prestartAllCoreThreads() called twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11500) InputStream is left unclosed in ApplicationClassLoader
Ted Yu created HADOOP-11500:
---
Summary: InputStream is left unclosed in ApplicationClassLoader
Key: HADOOP-11500
URL: https://issues.apache.org/jira/browse/HADOOP-11500
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Priority: Minor
{code}
InputStream is = null;
try {
is = ApplicationClassLoader.class.getClassLoader().
getResourceAsStream(PROPERTIES_FILE);
{code}
The InputStream is not closed in the static block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-10980) TestActiveStandbyElector fails occasionally in trunk
[
https://issues.apache.org/jira/browse/HADOOP-10980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-10980.
-
Resolution: Cannot Reproduce
Hadoop-Common-trunk has been green for a while.
TestActiveStandbyElector fails occasionally in trunk
Key: HADOOP-10980
URL: https://issues.apache.org/jira/browse/HADOOP-10980
Project: Hadoop Common
Issue Type: Test
Affects Versions: 3.0.0
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Common-trunk/1211/consoleFull :
{code}
Running org.apache.hadoop.ha.TestActiveStandbyElector
Tests run: 23, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.7 sec
FAILURE! - in org.apache.hadoop.ha.TestActiveStandbyElector
testWithoutZKServer(org.apache.hadoop.ha.TestActiveStandbyElector) Time
elapsed: 0.051 sec FAILURE!
java.lang.AssertionError: Did not throw zookeeper connection loss exceptions!
at org.junit.Assert.fail(Assert.java:88)
at
org.apache.hadoop.ha.TestActiveStandbyElector.testWithoutZKServer(TestActiveStandbyElector.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11190) Potentially stale value is used in SelfRenewingLease ctor
[
https://issues.apache.org/jira/browse/HADOOP-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-11190.
-
Resolution: Later
Potentially stale value is used in SelfRenewingLease ctor
-
Key: HADOOP-11190
URL: https://issues.apache.org/jira/browse/HADOOP-11190
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Here is w.r.t. threadNumber, shown in the code around line 102:
{code}
renewer.setName(AzureLeaseRenewer- + threadNumber++);
{code}
Since there is no synchronization involved, potentially stale value may be
read.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11475) Utilize try-with-resource to close StopWatch
Ted Yu created HADOOP-11475:
---
Summary: Utilize try-with-resource to close StopWatch
Key: HADOOP-11475
URL: https://issues.apache.org/jira/browse/HADOOP-11475
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor
Currently the stop() method of StopWatch is called without using finally clause.
This can result in resource leak if there is IOE thrown.
Here is one example from Journal#journal():
{code}
StopWatch sw = new StopWatch();
sw.start();
curSegment.flush(shouldFsync);
sw.stop();
{code}
If curSegment.flush() throws IOE, sw would be left unclosed.
Propose using try-with-resource structure to close the StopWatch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12805) Annotate CanUnbuffer with LimitedPrivate
Ted Yu created HADOOP-12805:
---
Summary: Annotate CanUnbuffer with LimitedPrivate
Key: HADOOP-12805
URL: https://issues.apache.org/jira/browse/HADOOP-12805
Project: Hadoop Common
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
See comments toward the tail of HBASE-9393.
The change in HBASE-9393 adds dependency on CanUnbuffer interface which is
currently marked @InterfaceAudience.Private
To facilitate downstream projects such as HBase in using this interface,
CanUnbuffer interface should be annotated LimitedPrivate({"HBase"}).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12724) Let BufferedFSInputStream implement CanUnbuffer
Ted Yu created HADOOP-12724:
---
Summary: Let BufferedFSInputStream implement CanUnbuffer
Key: HADOOP-12724
URL: https://issues.apache.org/jira/browse/HADOOP-12724
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
When trying to determine reason for test failure over in HBASE-9393, I saw the
following exception:
{code}
testSeekTo[4](org.apache.hadoop.hbase.io.hfile.TestSeekTo) Time elapsed: 0.033
sec <<< ERROR!
java.lang.UnsupportedOperationException: this stream does not support
unbuffering.
at
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:229)
at
org.apache.hadoop.fs.FSDataInputStream.unbuffer(FSDataInputStream.java:227)
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:518)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:562)
at
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekToInternals(TestSeekTo.java:307)
at
org.apache.hadoop.hbase.io.hfile.TestSeekTo.testSeekTo(TestSeekTo.java:298)
{code}
Here is the cause:
{code}
java.lang.ClassCastException: org.apache.hadoop.fs.BufferedFSInputStream cannot
be cast to org.apache.hadoop.fs.CanUnbuffer
{code}
See the comments starting with
https://issues.apache.org/jira/browse/HBASE-9393?focusedCommentId=15105939=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15105939
for background on the HBase patch.
This issue is to make BufferedFSInputStream implement CanUnbuffer.
Thanks to [~cmccabe] for discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-13135) Encounter response code 500 when accessing /metrics endpoint
Ted Yu created HADOOP-13135:
---
Summary: Encounter response code 500 when accessing /metrics
endpoint
Key: HADOOP-13135
URL: https://issues.apache.org/jira/browse/HADOOP-13135
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Ted Yu
When accessing /metrics endpoint on hbase master through hadoop 2.7.1, I got:
{code}
HTTP ERROR 500
Problem accessing /metrics. Reason:
INTERNAL_SERVER_ERROR
Caused by:
java.lang.NullPointerException
at
org.apache.hadoop.http.HttpServer2.isInstrumentationAccessAllowed(HttpServer2.java:1029)
at
org.apache.hadoop.metrics.MetricsServlet.doGet(MetricsServlet.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:113)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
{code}
[~ajisakaa] suggested that code 500 should be 404 (NOT FOUND).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13496) Include file lengths in Mismatch in length error for distcp
Ted Yu created HADOOP-13496:
---
Summary: Include file lengths in Mismatch in length error for
distcp
Key: HADOOP-13496
URL: https://issues.apache.org/jira/browse/HADOOP-13496
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
Priority: Minor
Currently RetriableFileCopyCommand doesn't show the perceived lengths in
Mismatch in length error:
{code}
2016-08-12 10:23:14,231 ERROR [LocalJobRunner Map Task Executor #0]
util.RetriableCommand(89): Failure in Retriable command: Copying
hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-
c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182
to hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/10.22.9.
171%2C53952%2C1471022508087.regiongroup-1.1471022510182
java.io.IOException: Mismatch in length of
source:hdfs://localhost:53941/user/tyu/test-data/dc7c674a-c463-4798-8260-c5d1e3440a4b/WALs/10.22.9.171,53952,1471022508087/10.22.9.171%2C53952%2C1471022508087.regiongroup-1.1471022510182
and
target:hdfs://localhost:53941/backupUT/backup_1471022580616/WALs/.distcp.tmp.attempt_local344329843_0006_m_00_0
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareFileLengths(RetriableFileCopyCommand.java:193)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:126)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:281)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:253)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
{code}
It would be helpful to include what's the expected length and what's the real
length.
Thanks to [~yzhangal] for offline discussion.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13489) DistCp may incorrectly return success status when the underlying Job failed
Ted Yu created HADOOP-13489:
---
Summary: DistCp may incorrectly return success status when the
underlying Job failed
Key: HADOOP-13489
URL: https://issues.apache.org/jira/browse/HADOOP-13489
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
I was troubleshooting HBASE-14450 where at the end of BackupdistCp#execute(),
distcp job was marked unsuccessful (BackupdistCp is a wrapper of DistCp).
Yet in IncrementalTableBackupProcedure#incrementalCopy(), the return value from
copyService.copy() was 0.
Here is related code from DistCp:
{code}
try {
execute();
} catch (InvalidInputException e) {
LOG.error("Invalid input: ", e);
return DistCpConstants.INVALID_ARGUMENT;
} catch (DuplicateFileException e) {
LOG.error("Duplicate files in input path: ", e);
return DistCpConstants.DUPLICATE_INPUT;
} catch (AclsNotSupportedException e) {
LOG.error("ACLs not supported on at least one file system: ", e);
return DistCpConstants.ACLS_NOT_SUPPORTED;
} catch (XAttrsNotSupportedException e) {
LOG.error("XAttrs not supported on at least one file system: ", e);
return DistCpConstants.XATTRS_NOT_SUPPORTED;
} catch (Exception e) {
LOG.error("Exception encountered ", e);
return DistCpConstants.UNKNOWN_ERROR;
}
return DistCpConstants.SUCCESS;
{code}
We don't check whether the Job returned by execute() was successful.
Even if the Job fails, DistCpConstants.SUCCESS is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13394) Swift should have proper HttpClient dependencies
[
https://issues.apache.org/jira/browse/HADOOP-13394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-13394.
-
Resolution: Duplicate
Close as dup of HADOOP-11614
> Swift should have proper HttpClient dependencies
>
>
> Key: HADOOP-13394
> URL: https://issues.apache.org/jira/browse/HADOOP-13394
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>
> In hadoop-tools/hadoop-openstack/pom.xml :
> {code}
>
> commons-httpclient
> commons-httpclient
> compile
>
> {code}
> The dependency should be migrated to httpclient of org.apache.httpcomponents
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] (HADOOP-14043) Shade netty dependency
Ted Yu created HADOOP-14043: --- Summary: Shade netty dependency Key: HADOOP-14043 URL: https://issues.apache.org/jira/browse/HADOOP-14043 Project: Hadoop Common Issue Type: Sub-task Reporter: Ted Yu During review of HADOOP-13866, [~andrew.wang] mentioned considering shading netty before putting the fix into branch-2. This would give users better experience when upgrading hadoop. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14076) Allow Configuration to be persisted given path to file
Ted Yu created HADOOP-14076:
---
Summary: Allow Configuration to be persisted given path to file
Key: HADOOP-14076
URL: https://issues.apache.org/jira/browse/HADOOP-14076
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
Currently Configuration has the following methods for persistence:
{code}
public void writeXml(OutputStream out) throws IOException {
public void writeXml(Writer out) throws IOException {
{code}
Adding API for persisting to file given path would be useful:
{code}
public void writeXml(String path) throws IOException {
{code}
Background: I recently worked on exporting Configuration to a file using JNI.
Without the proposed API, I resorted to some trick such as the following:
http://www.kfu.com/~nsayer/Java/jni-filedesc.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13489) DistCp may incorrectly return success status when the underlying Job failed
[
https://issues.apache.org/jira/browse/HADOOP-13489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-13489.
-
Resolution: Won't Fix
After adjusting hbase code, the problem is gone.
> DistCp may incorrectly return success status when the underlying Job failed
> ---
>
> Key: HADOOP-13489
> URL: https://issues.apache.org/jira/browse/HADOOP-13489
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: distcp
> Attachments: HADOOP-13489.v1.patch, HADOOP-13489.v2.patch,
> HADOOP-13489.v3.patch, MapReduceBackupCopyService.java,
> testIncrementalBackup-8-12-rethrow.txt, testIncrementalBackup-8-12.txt,
> TestIncrementalBackup-output.txt
>
>
> I was troubleshooting HBASE-14450 where at the end of BackupdistCp#execute(),
> distcp job was marked unsuccessful (BackupdistCp is a wrapper of DistCp).
> Yet in IncrementalTableBackupProcedure#incrementalCopy(), the return value
> from copyService.copy() was 0.
> Here is related code from DistCp:
> {code}
> try {
> execute();
> } catch (InvalidInputException e) {
> LOG.error("Invalid input: ", e);
> return DistCpConstants.INVALID_ARGUMENT;
> } catch (DuplicateFileException e) {
> LOG.error("Duplicate files in input path: ", e);
> return DistCpConstants.DUPLICATE_INPUT;
> } catch (AclsNotSupportedException e) {
> LOG.error("ACLs not supported on at least one file system: ", e);
> return DistCpConstants.ACLS_NOT_SUPPORTED;
> } catch (XAttrsNotSupportedException e) {
> LOG.error("XAttrs not supported on at least one file system: ", e);
> return DistCpConstants.XATTRS_NOT_SUPPORTED;
> } catch (Exception e) {
> LOG.error("Exception encountered ", e);
> return DistCpConstants.UNKNOWN_ERROR;
> }
> return DistCpConstants.SUCCESS;
> {code}
> We don't check whether the Job returned by execute() was successful - we rely
> on all failure cases going through the last catch clause but there may be
> special case.
> Even if the Job fails, DistCpConstants.SUCCESS is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12854) Move to netty 4.1.x release
[ https://issues.apache.org/jira/browse/HADOOP-12854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu resolved HADOOP-12854. - Resolution: Duplicate Dup of HADOOP-13866 > Move to netty 4.1.x release > --- > > Key: HADOOP-12854 > URL: https://issues.apache.org/jira/browse/HADOOP-12854 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > Netty is getting close to having a final release of a 4.1 netty-all artifact; > HDFS currently pulls in 4.1.0.Beta5 > Once a 4.1 release is out, switch to it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
[ https://issues.apache.org/jira/browse/HADOOP-13866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HADOOP-13866: - > Upgrade netty-all to 4.1.1.Final > > > Key: HADOOP-13866 > URL: https://issues.apache.org/jira/browse/HADOOP-13866 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: HADOOP-13866.v1.patch, HADOOP-13866.v2.patch > > > netty-all 4.1.1.Final is stable release which we should upgrade to. > See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13866) Upgrade netty-all to 4.1.1.Final
Ted Yu created HADOOP-13866: --- Summary: Upgrade netty-all to 4.1.1.Final Key: HADOOP-13866 URL: https://issues.apache.org/jira/browse/HADOOP-13866 Project: Hadoop Common Issue Type: Improvement Reporter: Ted Yu netty-all 4.1.1.Final is stable release which we should upgrade to. See bottom of HADOOP-12927 for related discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14076) Allow Configuration to be persisted given path to file
[
https://issues.apache.org/jira/browse/HADOOP-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-14076.
-
Resolution: Later
This can be done client side.
> Allow Configuration to be persisted given path to file
> --
>
> Key: HADOOP-14076
> URL: https://issues.apache.org/jira/browse/HADOOP-14076
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Currently Configuration has the following methods for persistence:
> {code}
> public void writeXml(OutputStream out) throws IOException {
> public void writeXml(Writer out) throws IOException {
> {code}
> Adding API for persisting to file given path would be useful:
> {code}
> public void writeXml(String path) throws IOException {
> {code}
> Background: I recently worked on exporting Configuration to a file using JNI.
> Without the proposed API, I resorted to some trick such as the following:
> http://www.kfu.com/~nsayer/Java/jni-filedesc.html
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14895) Consider exposing SimpleCopyListing#computeSourceRootPath() for downstream project
Ted Yu created HADOOP-14895: --- Summary: Consider exposing SimpleCopyListing#computeSourceRootPath() for downstream project Key: HADOOP-14895 URL: https://issues.apache.org/jira/browse/HADOOP-14895 Project: Hadoop Common Issue Type: Improvement Reporter: Ted Yu Over in HBASE-18843, [~vrodionov] needs to override SimpleCopyListing#computeSourceRootPath() . Since the method is private, some duplicated code appears in hbase. We should consider exposing SimpleCopyListing#computeSourceRootPath() so that its behavior can be overridden. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14930) Upgrade Jetty to 9.4 version
Ted Yu created HADOOP-14930:
---
Summary: Upgrade Jetty to 9.4 version
Key: HADOOP-14930
URL: https://issues.apache.org/jira/browse/HADOOP-14930
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
Currently 9.3.19.v20170502 is used.
In hbase 2.0+, 9.4.6.v20170531 is used.
When starting mini dfs cluster in hbase unit tests, we get the following:
{code}
java.lang.NoSuchMethodError:
org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
at
org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:548)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:529)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:119)
at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:415)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:157)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:887)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:723)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:949)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:928)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1277)
at
org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1046)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:921)
{code}
This issue is to upgrade Jetty to 9.4 version
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-10202) OK_JAVADOC_WARNINGS is out of date, leading to negative javadoc warning count
[
https://issues.apache.org/jira/browse/HADOOP-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-10202.
-
Resolution: Cannot Reproduce
> OK_JAVADOC_WARNINGS is out of date, leading to negative javadoc warning count
> -
>
> Key: HADOOP-10202
> URL: https://issues.apache.org/jira/browse/HADOOP-10202
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Minor
>
> From https://builds.apache.org/job/PreCommit-HDFS-Build/5813//testReport/ :
> {code}
> -1 javadoc. The javadoc tool appears to have generated -14 warning messages.
> {code}
> OK_JAVADOC_WARNINGS should be updated.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-10642) Provide option to limit heap memory consumed by dynamic metrics2 metrics
[
https://issues.apache.org/jira/browse/HADOOP-10642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-10642.
-
Resolution: Later
> Provide option to limit heap memory consumed by dynamic metrics2 metrics
>
>
> Key: HADOOP-10642
> URL: https://issues.apache.org/jira/browse/HADOOP-10642
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Ted Yu
>
> User sunweiei provided the following jmap output in HBase 0.96 deployment:
> {code}
> num #instances #bytes class name
> --
>1: 14917882 3396492464 [C
>2: 1996994 2118021808 [B
>3: 43341650 1733666000 java.util.LinkedHashMap$Entry
>4: 14453983 1156550896 [Ljava.util.HashMap$Entry;
>5: 14446577 924580928
> org.apache.hadoop.metrics2.lib.Interns$CacheWith2Keys$2
> {code}
> Heap consumption by Interns$CacheWith2Keys$2 (and indirectly by [C) could be
> due to calls to Interns.info() in DynamicMetricsRegistry which was cloned off
> metrics2/lib/MetricsRegistry.java.
> This scenario would arise when large number of regions are tracked through
> metrics2 dynamically.
> Interns class doesn't provide API to remove entries in its internal Map.
> One solution is to provide an option that allows skipping calls to
> Interns.info() in metrics2/lib/MetricsRegistry.java
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14942) DistCp#cleanup() should check whether jobFS is null
Ted Yu created HADOOP-14942:
---
Summary: DistCp#cleanup() should check whether jobFS is null
Key: HADOOP-14942
URL: https://issues.apache.org/jira/browse/HADOOP-14942
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Over in HBASE-18975, we observed the following:
{code}
2017-10-10 17:22:53,211 DEBUG [main] mapreduce.MapReduceBackupCopyJob(313):
Doing COPY_TYPE_DISTCP
2017-10-10 17:22:53,272 DEBUG [main] mapreduce.MapReduceBackupCopyJob(322):
DistCp options: [hdfs://localhost:55247/backupUT/.tmp/backup_1507681285309,
hdfs://localhost:55247/ backupUT]
2017-10-10 17:22:53,283 ERROR [main] tools.DistCp(167): Exception encountered
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.execute(MapReduceBackupCopyJob.java:234)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:153)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob.copy(MapReduceBackupCopyJob.java:331)
at
org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:286)
...
Caused by: java.lang.NullPointerException
at org.apache.hadoop.tools.DistCp.cleanup(DistCp.java:460)
... 45 more
{code}
NullPointerException came from second line below:
{code}
if (metaFolder == null) return;
jobFS.delete(metaFolder, true);
{code}
in which case jobFS was null.
A check against null should be added.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15051) FSDataOutputStream returned by LocalFileSystem#createNonRecursive doesn
Ted Yu created HADOOP-15051: --- Summary: FSDataOutputStream returned by LocalFileSystem#createNonRecursive doesn Key: HADOOP-15051 URL: https://issues.apache.org/jira/browse/HADOOP-15051 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0-beta1 Reporter: Ted Yu -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-15290) Imprecise assertion in FileStatus w.r.t. symlink
[
https://issues.apache.org/jira/browse/HADOOP-15290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HADOOP-15290.
-
Resolution: Duplicate
Dup of HADOOP-15289
> Imprecise assertion in FileStatus w.r.t. symlink
>
>
> Key: HADOOP-15290
> URL: https://issues.apache.org/jira/browse/HADOOP-15290
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
>
> In HBASE-20123, I logged the following stack trace:
> {code}
> 2018-03-03 14:46:10,858 ERROR [Time-limited test]
> mapreduce.MapReduceBackupCopyJob$BackupDistCp(237): java.io.IOException: Path
> hdfs://localhost:40578/backupUT/.tmp/backup_1520088356047 is not a symbolic
> link
> java.io.IOException: Path
> hdfs://localhost:40578/backupUT/.tmp/backup_1520088356047 is not a symbolic
> link
> at org.apache.hadoop.fs.FileStatus.getSymlink(FileStatus.java:338)
> at org.apache.hadoop.fs.FileStatus.readFields(FileStatus.java:461)
> at
> org.apache.hadoop.tools.CopyListingFileStatus.readFields(CopyListingFileStatus.java:155)
> at
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2308)
> at
> org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:163)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:91)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:84)
> at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:382)
> at
> org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.createInputFileListing(MapReduceBackupCopyJob.java:297)
> {code}
> [~ste...@apache.org] pointed out that the assertion in FileStatus.java is not
> accurate:
> {code}
> assert (isDirectory() && getSymlink() == null) || !isDirectory();
> {code}
> {quote}
> It's assuming that getSymlink() returns null if there is no symlink, but
> instead it raises and exception.
> {quote}
> Steve proposed the following replacement:
> {code}
> assert (!(isDirectory() && isSymlink())
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15290) Imprecise assertion in FileStatus w.r.t. symlink
Ted Yu created HADOOP-15290:
---
Summary: Imprecise assertion in FileStatus w.r.t. symlink
Key: HADOOP-15290
URL: https://issues.apache.org/jira/browse/HADOOP-15290
Project: Hadoop Common
Issue Type: Bug
Reporter: Ted Yu
In HBASE-20123, I logged the following stack trace:
{code}
2018-03-03 14:46:10,858 ERROR [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(237): java.io.IOException: Path
hdfs://localhost:40578/backupUT/.tmp/backup_1520088356047 is not a symbolic link
java.io.IOException: Path
hdfs://localhost:40578/backupUT/.tmp/backup_1520088356047 is not a symbolic link
at org.apache.hadoop.fs.FileStatus.getSymlink(FileStatus.java:338)
at org.apache.hadoop.fs.FileStatus.readFields(FileStatus.java:461)
at
org.apache.hadoop.tools.CopyListingFileStatus.readFields(CopyListingFileStatus.java:155)
at
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2308)
at
org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:163)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:91)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:84)
at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:382)
at
org.apache.hadoop.hbase.backup.mapreduce.MapReduceBackupCopyJob$BackupDistCp.createInputFileListing(MapReduceBackupCopyJob.java:297)
{code}
[~ste...@apache.org] pointed out that the assertion in FileStatus.java is not
accurate:
{code}
assert (isDirectory() && getSymlink() == null) || !isDirectory();
{code}
{quote}
It's assuming that getSymlink() returns null if there is no symlink, but
instead it raises and exception.
{quote}
Steve proposed the following replacement:
{code}
assert (!(isDirectory() && isSymlink())
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15850) Allow CopyCommitter to skip concatenating source files specified by DistCpConstants.CONF_LABEL_LISTING_FILE_PATH
Ted Yu created HADOOP-15850:
---
Summary: Allow CopyCommitter to skip concatenating source files
specified by DistCpConstants.CONF_LABEL_LISTING_FILE_PATH
Key: HADOOP-15850
URL: https://issues.apache.org/jira/browse/HADOOP-15850
Project: Hadoop Common
Issue Type: Task
Reporter: Ted Yu
I was investigating test failure of TestIncrementalBackupWithBulkLoad from
hbase against hadoop 3.1.1
hbase MapReduceBackupCopyJob$BackupDistCp would create listing file:
{code}
LOG.debug("creating input listing " + listing + " , totalRecords=" +
totalRecords);
cfg.set(DistCpConstants.CONF_LABEL_LISTING_FILE_PATH, listing);
cfg.setLong(DistCpConstants.CONF_LABEL_TOTAL_NUMBER_OF_RECORDS,
totalRecords);
{code}
For the test case, two bulk loaded hfiles are in the listing:
{code}
2018-10-13 14:09:24,123 DEBUG [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp :
hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
2018-10-13 14:09:24,125 DEBUG [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(195): BackupDistCp :
hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
2018-10-13 14:09:24,125 DEBUG [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(197): BackupDistCp execute for 2
files of 10242
{code}
Later on, CopyCommitter#concatFileChunks would throw the following exception:
{code}
2018-10-13 14:09:25,351 WARN [Thread-936] mapred.LocalJobRunner$Job(590):
job_local1795473782_0004
java.io.IOException: Inconsistent sequence file: current chunk file
org.apache.hadoop.tools.CopyListingFileStatus@bb8826ee{hdfs://localhost:42796/user/hbase/test-data/
160aeab5-6bca-9f87-465e-2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/a7599081e835440eb7bf0dd3ef4fd7a5_SeqId_205_
length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
org.apache.hadoop.tools.CopyListingFileStatus@243d544d{hdfs://localhost:42796/user/hbase/test-data/160aeab5-6bca-9f87-465e-
2517a0c43119/data/default/test-1539439707496/96b5a3613d52f4df1ba87a1cef20684c/f/394e6d39a9b94b148b9089c4fb967aad_SeqId_205_
length = 5142 aclEntries = null, xAttrs = null}
at
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
at
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
{code}
The above warning shouldn't happen - the two bulk loaded hfiles are independent.
>From the contents of the two CopyListingFileStatus instances, we can see that
>their isSplit() return false. Otherwise the following from toString should be
>logged:
{code}
if (isSplit()) {
sb.append(", chunkOffset = ").append(this.getChunkOffset());
sb.append(", chunkLength = ").append(this.getChunkLength());
}
{code}
>From hbase side, we can specify one bulk loaded hfile per job but that defeats
>the purpose of using DistCp.
There should be a way for DistCp to specify the skipping of source file
concatenation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15876) Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
Ted Yu created HADOOP-15876:
---
Summary: Use keySet().removeAll() to remove multiple keys from Map
in AzureBlobFileSystemStore
Key: HADOOP-15876
URL: https://issues.apache.org/jira/browse/HADOOP-15876
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
Looking at
hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java
, {{removeDefaultAcl}} in particular:
{code}
for (Map.Entry defaultAclEntry :
defaultAclEntries.entrySet()) {
aclEntries.remove(defaultAclEntry.getKey());
}
{code}
The above operation can be written this way:
{code}
aclEntries.keySet().removeAll(defaultAclEntries.keySet());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15910) Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS is wrong
Ted Yu created HADOOP-15910: --- Summary: Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS is wrong Key: HADOOP-15910 URL: https://issues.apache.org/jira/browse/HADOOP-15910 Project: Hadoop Common Issue Type: Bug Reporter: Ted Yu In LdapAuthenticationHandler, the javadoc for ENABLE_START_TLS has the same contents for BASE_DN -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15831) Include modificationTime in the toString method of CopyListingFileStatus
Ted Yu created HADOOP-15831:
---
Summary: Include modificationTime in the toString method of
CopyListingFileStatus
Key: HADOOP-15831
URL: https://issues.apache.org/jira/browse/HADOOP-15831
Project: Hadoop Common
Issue Type: Improvement
Reporter: Ted Yu
I was looking at a DistCp error observed in hbase backup test:
{code}
2018-10-08 18:12:03,067 WARN [Thread-933] mapred.LocalJobRunner$Job(590):
job_local1175594345_0004
java.io.IOException: Inconsistent sequence file: current chunk file
org.apache.hadoop.tools.CopyListingFileStatus@7ac56817{hdfs://localhost:41712/user/hbase/test-data/
c0f6352c-cf39-bbd1-7d10-57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/f565f49046b04eecbf8d129eac7a7b88_SeqId_205_
length = 5100 aclEntries = null, xAttrs = null} doesnt match prior entry
org.apache.hadoop.tools.CopyListingFileStatus@7aa4deb2{hdfs://localhost:41712/user/hbase/test-data/c0f6352c-cf39-bbd1-7d10-
57a9c01e7ce9/data/default/test-1539022262249/be1bf5445faddb63e45726410a07917a/f/41b6cb64bae64cbcac47d1fd9aae59f4_SeqId_205_
length = 5142 aclEntries = null, xAttrs = null}
at
org.apache.hadoop.tools.mapred.CopyCommitter.concatFileChunks(CopyCommitter.java:276)
at
org.apache.hadoop.tools.mapred.CopyCommitter.commitJob(CopyCommitter.java:100)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:567)
2018-10-08 18:12:03,150 INFO [Time-limited test]
mapreduce.MapReduceBackupCopyJob$BackupDistCp(226): Progress: 100.0% subTask:
1.0 mapProgress: 1.0
{code}
I noticed that modificationTime was not included in the toString of
CopyListingFileStatus.
I propose including modificationTime so that it is easier to tell when the
respective files last change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
