[jira] [Created] (HADOOP-16461) Regression: FileSystem cache lock parses XML within the lock
Gopal V created HADOOP-16461:
Summary: Regression: FileSystem cache lock parses XML within the
lock
Key: HADOOP-16461
URL: https://issues.apache.org/jira/browse/HADOOP-16461
Project: Hadoop Common
Issue Type: Bug
Components: filecache
Reporter: Gopal V
{code}
fs = createFileSystem(uri, conf);
synchronized (this) { // refetch the lock again
FileSystem oldfs = map.get(key);
if (oldfs != null) { // a file system is created while lock is releasing
fs.close(); // close the new file system
return oldfs; // return the old file system
}
// now insert the new file system into the map
if (map.isEmpty()
&& !ShutdownHookManager.get().isShutdownInProgress()) {
ShutdownHookManager.get().addShutdownHook(clientFinalizer,
SHUTDOWN_HOOK_PRIORITY);
}
fs.key = key;
map.put(key, fs);
if (conf.getBoolean(
FS_AUTOMATIC_CLOSE_KEY, FS_AUTOMATIC_CLOSE_DEFAULT)) {
toAutoClose.add(key);
}
return fs;
}
{code}
The lock now has a ShutdownHook creation, which ends up doing
{code}
HookEntry(Runnable hook, int priority) {
this(hook, priority,
getShutdownTimeout(new Configuration()),
TIME_UNIT_DEFAULT);
}
{code}
which ends up doing a "new Configuration()" within the locked section.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16341) ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
Gopal V created HADOOP-16341: Summary: ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 Key: HADOOP-16341 URL: https://issues.apache.org/jira/browse/HADOOP-16341 Project: Hadoop Common Issue Type: Bug Components: common Affects Versions: 3.1.2 Reporter: Gopal V Attachments: shutdown-hook-removal.png !shutdown-hook-removal.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14054) NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls in it
[
https://issues.apache.org/jira/browse/HADOOP-14054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V resolved HADOOP-14054.
--
Resolution: Won't Fix
Closing as WONTFIX - will file issues on Azure github
> NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls
> in it
> -
>
> Key: HADOOP-14054
> URL: https://issues.apache.org/jira/browse/HADOOP-14054
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Gopal V
>
> See the lock around the multiple buffered input streams with the read1
> java.io.BufferedInputStream.read() calls are all synchronized blocks.
> {code}
> IO-Elevator-Thread-9" #288 daemon prio=5 os_prio=0 tid=0x7fbf9c0aa000
> nid=0xa07d runnable [0x7faf3b4b4000]
>java.lang.Thread.State: RUNNABLE
> sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:140)
> at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:127)
> at
> java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:584)
> at java.security.MessageDigest.update(MessageDigest.java:325)
> at
> com.microsoft.azure.storage.core.Utility.writeToOutputStream(Utility.java:1329)
> at
> com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1410)
> at
> com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1310)
> at
> com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146)
> at
> com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1499)
> at
> com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255)
> - eliminated <0x7fbdd5475b68> (a
> com.microsoft.azure.storage.blob.BlobInputStream)
> at
> com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448)
> - locked <0x7fbdd5475b68> (a
> com.microsoft.azure.storage.blob.BlobInputStream)
> at
> com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x7fbdd5475b38> (a java.io.BufferedInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:820)
> - locked <0x7fbdd5475ac0> (a
> org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <0x7fbdd5475a90> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)
> at
> org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
> at
> org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14054) NativeAzureFileSystem has multiple synchronized BufferedStream:read() calls in it
Gopal V created HADOOP-14054:
Summary: NativeAzureFileSystem has multiple synchronized
BufferedStream:read() calls in it
Key: HADOOP-14054
URL: https://issues.apache.org/jira/browse/HADOOP-14054
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Reporter: Gopal V
See the lock around the multiple buffered input streams with the read1
java.io.BufferedInputStream.read() calls are all synchronized blocks.
{code}
IO-Elevator-Thread-9" #288 daemon prio=5 os_prio=0 tid=0x7fbf9c0aa000
nid=0xa07d runnable [0x7faf3b4b4000]
java.lang.Thread.State: RUNNABLE
sun.security.provider.DigestBase.implCompressMultiBlock(DigestBase.java:140)
at sun.security.provider.DigestBase.engineUpdate(DigestBase.java:127)
at
java.security.MessageDigest$Delegate.engineUpdate(MessageDigest.java:584)
at java.security.MessageDigest.update(MessageDigest.java:325)
at
com.microsoft.azure.storage.core.Utility.writeToOutputStream(Utility.java:1329)
at
com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1410)
at
com.microsoft.azure.storage.blob.CloudBlob$9.postProcessResponse(CloudBlob.java:1310)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:146)
at
com.microsoft.azure.storage.blob.CloudBlob.downloadRangeInternal(CloudBlob.java:1499)
at
com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:255)
- eliminated <0x7fbdd5475b68> (a
com.microsoft.azure.storage.blob.BlobInputStream)
at
com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448)
- locked <0x7fbdd5475b68> (a
com.microsoft.azure.storage.blob.BlobInputStream)
at
com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x7fbdd5475b38> (a java.io.BufferedInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream.read(NativeAzureFileSystem.java:820)
- locked <0x7fbdd5475ac0> (a
org.apache.hadoop.fs.azure.NativeAzureFileSystem$NativeAzureFsInputStream)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x7fbdd5475a90> (a
org.apache.hadoop.fs.BufferedFSInputStream)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.fillBuffer(UncompressedSplitLineReader.java:62)
at
org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at
org.apache.hadoop.mapreduce.lib.input.UncompressedSplitLineReader.readLine(UncompressedSplitLineReader.java:94)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-12956) Inevitable Log4j2 migration via slf4j
Gopal V created HADOOP-12956:
Summary: Inevitable Log4j2 migration via slf4j
Key: HADOOP-12956
URL: https://issues.apache.org/jira/browse/HADOOP-12956
Project: Hadoop Common
Issue Type: Improvement
Reporter: Gopal V
{{5 August 2015 --The Apache Logging Services™ Project Management Committee
(PMC) has announced that the Log4j™ 1.x logging framework has reached its end
of life (EOL) and is no longer officially supported.}}
https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces
A whole framework log4j2 upgrade has to be synchronized, partly for improved
performance brought about by log4j2.
https://logging.apache.org/log4j/2.x/manual/async.html#Performance
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11770) [Umbrella] locate static synchronized blocks in hadoop-common
Gopal V created HADOOP-11770: Summary: [Umbrella] locate static synchronized blocks in hadoop-common Key: HADOOP-11770 URL: https://issues.apache.org/jira/browse/HADOOP-11770 Project: Hadoop Common Issue Type: Improvement Components: conf, io, ipc Affects Versions: 2.7.0 Reporter: Gopal V There are several static synchronized blocks in the hadoop common functionality that hurts any multi-threaded processing system which uses the common APIs. Identify the static synchronized blocks and locate them for potential fixes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11772) RPC Invoker relies on static ClientCache which has synchronized(this) blocks
Gopal V created HADOOP-11772:
Summary: RPC Invoker relies on static ClientCache which has
synchronized(this) blocks
Key: HADOOP-11772
URL: https://issues.apache.org/jira/browse/HADOOP-11772
Project: Hadoop Common
Issue Type: Sub-task
Components: ipc
Reporter: Gopal V
Attachments: sync-client-bt.png, sync-client-threads.png
{code}
private static ClientCache CLIENTS=new ClientCache();
...
this.client = CLIENTS.getClient(conf, factory);
{code}
Meanwhile in ClientCache
{code}
public synchronized Client getClient(Configuration conf,
SocketFactory factory, Class? extends Writable valueClass) {
...
Client client = clients.get(factory);
if (client == null) {
client = new Client(valueClass, conf, factory);
clients.put(factory, client);
} else {
client.incCount();
}
{code}
All invokers end up calling these methods, resulting in IPC clients choking up.
!sync-client-threads.png!
!sync-client-bt.png!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-11239) [JDK8] azurenative tests fail builds on JDK8
[ https://issues.apache.org/jira/browse/HADOOP-11239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HADOOP-11239. -- Resolution: Not a Problem [JDK8] azurenative tests fail builds on JDK8 Key: HADOOP-11239 URL: https://issues.apache.org/jira/browse/HADOOP-11239 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Gopal V Assignee: Gopal V Priority: Trivial java.util.Base64 conflicts with com.microsoft.windowsazure.storage.core.Base64 in Azure unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11239) [JDK8] azurenative tests fail builds on JDK8
Gopal V created HADOOP-11239: Summary: [JDK8] azurenative tests fail builds on JDK8 Key: HADOOP-11239 URL: https://issues.apache.org/jira/browse/HADOOP-11239 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Gopal V Assignee: Gopal V Priority: Trivial java.util.Base64 conflicts with com.microsoft.windowsazure.storage.core.Base64 in Azure unit tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11226) ipc.Client has to use setTrafficClass() with IPTOS_LOWDELAY
Gopal V created HADOOP-11226: Summary: ipc.Client has to use setTrafficClass() with IPTOS_LOWDELAY Key: HADOOP-11226 URL: https://issues.apache.org/jira/browse/HADOOP-11226 Project: Hadoop Common Issue Type: Bug Reporter: Gopal V -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11223) Offer a read-only conf alternative to new Configuration()
Gopal V created HADOOP-11223:
Summary: Offer a read-only conf alternative to new Configuration()
Key: HADOOP-11223
URL: https://issues.apache.org/jira/browse/HADOOP-11223
Project: Hadoop Common
Issue Type: Bug
Reporter: Gopal V
new Configuration() is called from several static blocks across Hadoop.
This is incredibly inefficient, since each one of those involves primarily XML
parsing at a point where the JIT won't be triggered interpreter mode is
essentially forced on the JVM.
The alternate solution would be to offer a {{Configuration::getDefault()}}
alternative which disallows any modifications.
At the very least, such a method would need to be called from
# org.apache.hadoop.io.nativeio.NativeIO::clinit()
# org.apache.hadoop.security.SecurityUtil::clinit()
# org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider::clinit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-11029) LocalFS Statistics performs thread local call per byte written
Gopal V created HADOOP-11029:
Summary: LocalFS Statistics performs thread local call per byte
written
Key: HADOOP-11029
URL: https://issues.apache.org/jira/browse/HADOOP-11029
Project: Hadoop Common
Issue Type: Bug
Reporter: Gopal V
Attachments: local-fs-locking.png
This code is there in the hot-path of IFile writer via RawLocalFileSystem.
!local-fs-locking.png!
From a preliminary glance, the lock prefix calls are coming from a
threadlocal.get() within FileSystem.Statistics
{code}
/**
* Get or create the thread-local data associated with the current thread.
*/
private StatisticsData getThreadData() {
StatisticsData data = threadData.get();
if (data == null) {
data = new StatisticsData(
new WeakReferenceThread(Thread.currentThread()));
threadData.set(data);
synchronized(this) {
if (allData == null) {
allData = new LinkedListStatisticsData();
}
allData.add(data);
}
}
return data;
}
/**
* Increment the bytes read in the statistics
* @param newBytes the additional bytes read
*/
public void incrementBytesRead(long newBytes) {
getThreadData().bytesRead += newBytes;
}
{code}
This is incredibly inefficient when used from FSDataOutputStream
{code}
public void write(int b) throws IOException {
out.write(b);
position++;
if (statistics != null) {
statistics.incrementBytesWritten(1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HADOOP-10681) Compressor inner methods are all synchronized - within a tight loop
Gopal V created HADOOP-10681: Summary: Compressor inner methods are all synchronized - within a tight loop Key: HADOOP-10681 URL: https://issues.apache.org/jira/browse/HADOOP-10681 Project: Hadoop Common Issue Type: Bug Components: performance Affects Versions: 2.4.0, 2.2.0, 2.5.0 Reporter: Gopal V Attachments: compress-cmpxchg-small.png, perf-top-spill-merge.png The current implementation of SnappyCompressor spends more time within the java loop of copying from the user buffer into the direct buffer allocated to the compressor impl, than the time it takes to compress the buffers. !perf-top-spill-merge.png! The bottleneck was found to be java monitor code inside SnappyCompressor. The methods are neatly inlined by the JIT into the parent caller (BlockCompressorStream::write), which unfortunately does not flatten out the synchronized blocks. !compress-cmpxchg-small.png! The loop does a write of small byte[] buffers (each IFile key+value). I counted approximately 6 monitor enter/exit blocks per k-v pair written. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10513) ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
Gopal V created HADOOP-10513:
Summary: ReflectionUtils::CONSTRUCTOR_CACHE leaks class loaders
Key: HADOOP-10513
URL: https://issues.apache.org/jira/browse/HADOOP-10513
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 2.4.0
Reporter: Gopal V
Any code which uses custom class-loaders needs to be aware of the
ReflectionUtils::CONSTRUCTOR_CACHE
{code}
/**
* Cache of constructors for each class. Pins the classes so they
* can't be garbage collected until ReflectionUtils can be collected.
*/
private static final MapClass?, Constructor? CONSTRUCTOR_CACHE =
new ConcurrentHashMapClass?, Constructor?();
{code}
This is not a problem when using only 1 AppClassLoader.
But in cases where the application uses multiple classloaders (to isolate
UDFs), this holds onto class references and their associated class loaders by
ref ( that leaks all the statics).
The clear method for this cache is unfortunately marked only for testing.
{code}
// methods to support testing
static void clearCache() {
CONSTRUCTOR_CACHE.clear();
}
{code}
The cache shows up as the only reference for the class loaders.
!class-loader-leak.png!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-8921) ant build.xml in branch-1 ignores -Dcompile.native
[ https://issues.apache.org/jira/browse/HADOOP-8921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V resolved HADOOP-8921. - Resolution: Unresolved ant build.xml in branch-1 ignores -Dcompile.native -- Key: HADOOP-8921 URL: https://issues.apache.org/jira/browse/HADOOP-8921 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.2.0 Environment: Mac OS X 10.7.4 Reporter: Gopal V Assignee: Gopal V Priority: Trivial Labels: ant, autoconf, patch Attachments: HADOOP-8921.4.patch, HADOOP-8921.5.patch, HADOOP-8921.7.patch, hadoop-8921.6.patch ant -Dcompile.native=false still runs autoconf and libtoolize According to ant 1.8 manual, any target if conditions are checked only after the dependencies are run through. The current if condition in code fails to prevent the autoconf/libtool components from running. Fixing it by moving the if condition up into the compile-native target and changing it to a param substitution instead of being evaluated as a condition. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10047) Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API
Gopal V created HADOOP-10047: Summary: Allow Compressor/Decompressor APIs to expose a Direct ByteBuffer API Key: HADOOP-10047 URL: https://issues.apache.org/jira/browse/HADOOP-10047 Project: Hadoop Common Issue Type: New Feature Components: io Reporter: Gopal V Assignee: Gopal V With the Zero-Copy reads in HDFS (HDFS-5260), it becomes important to perform all I/O operations without copying data into byte[] buffers or other buffers which wrap over them. This is a proposal for adding new DirectCompressor and DirectDecompressor interfaces to the io.compress, to indicate codecs which want to surface the direct buffer layer upwards. The implementation may or may not copy the buffers passed in, but should work with direct heap/mmap buffers and cannot assume .array() availability. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HADOOP-9612) HADOOP client IPC layer should differentiate between interrupted threads and interrupts occurring during operations
Gopal V created HADOOP-9612:
---
Summary: HADOOP client IPC layer should differentiate between
interrupted threads and interrupts occurring during operations
Key: HADOOP-9612
URL: https://issues.apache.org/jira/browse/HADOOP-9612
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.5-beta
Environment: Ubuntu / x86_64
Reporter: Gopal V
Assignee: Gopal V
Priority: Trivial
Hadoop IPC throws misleading wrapped exceptions when operations are performed
on threads which have already been interrupted.
When a thread is already interrupted, the client IPC throws the following
exception
{code}
Exception in thread main java.io.IOException:
...
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1270)
at org.apache.hadoop.ipc.Client.call(Client.java:1218)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy9.getBlockLocations(Unknown Source)
{code}
This is completely indistinguishable between a real interruption happening to
the thread during the client call and very misleading.
Before client calls, it would be prudent to check for the
currentThread().isInterrupted() flag and throw a different exception to
indicate the thread was already interrupted before the operation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9570) Configuration.addResource() should only parse the new resource
Gopal V created HADOOP-9570:
---
Summary: Configuration.addResource() should only parse the new
resource
Key: HADOOP-9570
URL: https://issues.apache.org/jira/browse/HADOOP-9570
Project: Hadoop Common
Issue Type: Bug
Components: conf
Environment: Ubuntu LXC
Reporter: Gopal V
Assignee: Gopal V
Priority: Minor
Hadoop configuration parsing re-parses all configuration files when a new
resource is added to the configuration object.
This is wasteful and runs through every deprecation check unnecessarily.
{code}
JobConf clone = new JobConf(job);
clone.addResource(...);
{code}
will reparse every file including core-site, hdfs-site etc.
Resource addition can be taken care of cleanly, if the addResourceObject() call
applies the new resource, followed by applying the overlay, without re-parsing
any of the older files already loaded into the hashtable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9496) Bad merge of HADOOP-9450 on branch-2 breaks all bin/hadoop calls that need HADOOP_CLASSPATH
Gopal V created HADOOP-9496: --- Summary: Bad merge of HADOOP-9450 on branch-2 breaks all bin/hadoop calls that need HADOOP_CLASSPATH Key: HADOOP-9496 URL: https://issues.apache.org/jira/browse/HADOOP-9496 Project: Hadoop Common Issue Type: Bug Components: bin Affects Versions: 2.0.5-beta Reporter: Gopal V Assignee: Harsh J Priority: Critical Merge of HADOOP-9450 to branch-2 is broken for hadoop-config.sh on trunk http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh?r1=1453486r2=1469214pathrev=1469214 vs on branch-2 http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh?r1=1390222r2=1469215 This is breaking all hadoop client code which needs HADOOP_CLASSPATH to be set correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8971) Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926)
Gopal V created HADOOP-8971: --- Summary: Backport: hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data (HADOOP-8926) Key: HADOOP-8971 URL: https://issues.apache.org/jira/browse/HADOOP-8971 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 1.2.0 Reporter: Gopal V Assignee: Gopal V Priority: Minor Backport (HADOOP-8926) cache-aware improvements made to PureJavaCrc32 to branch-1. With patch, observed significant improvement in crc throughput at larger chunk sizes on branch-1 java.version = 1.6.0_37 java.runtime.name = Java(TM) SE Runtime Environment java.runtime.version = 1.6.0_37-b06 java.vm.version = 20.12-b01 java.vm.vendor = Sun Microsystems Inc. java.vm.name = Java HotSpot(TM) 64-Bit Server VM java.vm.specification.version = 1.0 java.specification.version = 1.6 os.arch = amd64 os.name = Linux os.version = 3.5.0-17-generic Performance Table (The unit is MB/sec) || Num Bytes ||CRC32 || PureJavaCrc32 || PureJavaCrc32+patch || |1 |16.548 |163.916 |122.289 | |2 |30.046 |213.591 |246.657 | |4 |59.853 |301.521 |322.862 | |8 | 107.435 |491.757 |512.487 | | 16 | 179.278 |509.099 |709.656 | | 32 | 265.496 |651.438 |908.673 | | 64 | 350.994 |766.566 | 1000.775 | |128 | 410.514 |795.719 | 1073.625 | |256 | 434.938 |828.010 | 1188.074 | |512 | 451.997 |883.219 | 1239.929 | | 1024 | 443.369 |902.318 | 1250.906 | | 2048 | 427.923 |864.701 | 1228.599 | | 4096 | 430.814 |863.258 | 1210.633 | | 8192 | 437.079 |883.766 | 1242.756 | |16384 | 433.349 |851.653 | 1211.161 | |32768 | 433.188 |906.142 | 1171.551 | |65536 | 437.420 |890.266 | 1089.157 | | 131072 | 431.189 |869.041 | 1180.171 | | 262144 | 430.636 |910.133 | 1212.344 | | 524288 | 429.365 |889.867 | 1197.842 | |1048576 | 433.012 |887.909 | 1153.719 | |2097152 | 432.520 |861.701 | 1179.221 | |4194304 | 439.791 |899.085 | 1178.736 | |8388608 | 442.422 |856.197 | 1211.953 | | 16777216 | 419.015 |866.560 | 1152.946 | -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8926) hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data
Gopal V created HADOOP-8926:
---
Summary: hadoop.util.PureJavaCrc32 cache hit-ratio is low for
static data
Key: HADOOP-8926
URL: https://issues.apache.org/jira/browse/HADOOP-8926
Project: Hadoop Common
Issue Type: Improvement
Components: util
Affects Versions: 2.0.3-alpha
Environment: Ubuntu 10.10 i386
Reporter: Gopal V
Priority: Trivial
While running microbenchmarks for HDFS write codepath, a significant part of
the CPU fraction was consumed by the DataChecksum.update().
The attached patch converts the static arrays in CRC32 into a single linear
array for a performance boost in the inner loop.
milli-seconds for 1Gig (16400 loop over a 64kb chunk)
|| platform || original || cache-aware || improvement ||
| x86 | 3894 | 2304 | 40.83 |
| x86_64 | 2131 | 1826 | 14 |
The performance improvement on x86 is rather larger than the 64bit case, due to
the extra register/stack pressure caused by the static arrays.
A closer analysis of the PureJavaCrc32 JIT code shows the following assembly
fragment
{code}
0x40f1e345: mov$0x184,%ecx
0x40f1e34a: mov0x4415b560(%ecx),%ecx ;*getstatic T8_5
; - PureJavaCrc32::update@95 (line 61)
; {oop('PureJavaCrc32')}
0x40f1e350: mov%ecx,0x2c(%esp)
{code}
Basically, the static variables T8_0 through to T8_7 are being spilled to the
stack because of register pressure. The x86_64 case has a lower likelihood of
such pessimistic JIT code due to the increased number of registers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8921) ant build.xml in branch-1 ignores -Dcompile.native
Gopal V created HADOOP-8921: --- Summary: ant build.xml in branch-1 ignores -Dcompile.native Key: HADOOP-8921 URL: https://issues.apache.org/jira/browse/HADOOP-8921 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 1.2.0 Environment: Mac OS X 10.7.4 Reporter: Gopal V Priority: Trivial ant -Dcompile.native=false still runs autoconf and libtoolize According to ant 1.8 manual, any target if conditions are checked only after the dependencies are run through. The current if condition in code fails to prevent the autoconf/libtool components from running. Fixing it by moving the if condition up into the compile-native target and changing it to a param substitution instead of being evaluated as a condition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
