[
https://issues.apache.org/jira/browse/ACCUMULO-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816944#comment-13816944
]
Josh Elser commented on ACCUMULO-1833:
--------------------------------------
I noticed that I'm seeing the ShellServerTest run after the
MultiTableBatchWriterTest. The finalizer appears to be trying to call close on
a MTBW who has a lock on the TabletLocator class (static synchronized method on
the class) and there are a bunch of other tests which are locked trying to get
this lock. I wonder if I just need to be more aggressive with closing resources
in my test.
{noformat}
"Finalizer" daemon prio=5 tid=0x00007fedbc051800 nid=0x3903 in Object.wait()
[0x000000011a021000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007aeb30878> (a
org.apache.accumulo.fate.zookeeper.ZooCache)
at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:146)
- locked <0x00000007aeb30878> (a
org.apache.accumulo.fate.zookeeper.ZooCache)
at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:233)
- locked <0x00000007aeb30878> (a
org.apache.accumulo.fate.zookeeper.ZooCache)
at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:188)
- locked <0x00000007aeb30878> (a
org.apache.accumulo.fate.zookeeper.ZooCache)
at
org.apache.accumulo.core.client.ZooKeeperInstance.getInstanceID(ZooKeeperInstance.java:151)
at
org.apache.accumulo.core.client.impl.TabletLocator.getInstance(TabletLocator.java:96)
- locked <0x00000007aecd5928> (a java.lang.Class for
org.apache.accumulo.core.client.impl.TabletLocator)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.getLocator(TabletServerBatchWriter.java:636)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.binMutations(TabletServerBatchWriter.java:648)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.addMutations(TabletServerBatchWriter.java:696)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.startProcessing(TabletServerBatchWriter.java:232)
- locked <0x00000007aeb31408> (a
org.apache.accumulo.core.client.impl.TabletServerBatchWriter)
at
org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:345)
- locked <0x00000007aeb31408> (a
org.apache.accumulo.core.client.impl.TabletServerBatchWriter)
at
org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.close(MultiTableBatchWriterImpl.java:126)
at
org.apache.accumulo.core.client.impl.MultiTableBatchWriterImpl.finalize(MultiTableBatchWriterImpl.java:138)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:101)
at java.lang.ref.Finalizer.access$100(Finalizer.java:32)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:190)
Locked ownable synchronizers:
- None
{noformat}
> MultiTableBatchWriterImpl.getBatchWriter() is not performant for multiple
> threads
> ---------------------------------------------------------------------------------
>
> Key: ACCUMULO-1833
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1833
> Project: Accumulo
> Issue Type: Improvement
> Affects Versions: 1.5.0, 1.6.0
> Reporter: Chris McCubbin
> Attachments: ACCUMULO-1833-test.patch, ZooKeeperThreadUtilization.png
>
>
> This issue comes from profiling our application. We have a
> MultiTableBatchWriter created by normal means. I am attempting to write to it
> with multiple threads by doing things like the following:
> {code}
> batchWriter.getBatchWriter(table).addMutations(mutations);
> {code}
> In my test with 4 threads writing to one table, this call is quite
> inefficient and results in a large performance degradation over a single
> BatchWriter.
> I believe the culprit is the fact that the call is synchronized. Also there
> is the possibility that the zookeeper call to Tables.getTableState on every
> call is negatively affecting performance:
> {code}
> @Override
> public synchronized BatchWriter getBatchWriter(String tableName) throws
> AccumuloException, AccumuloSecurityException, TableNotFoundException {
> ArgumentChecker.notNull(tableName);
> String tableId = Tables.getNameToIdMap(instance).get(tableName);
> if (tableId == null)
> throw new TableNotFoundException(tableId, tableName, null);
>
> if (Tables.getTableState(instance, tableId) == TableState.OFFLINE)
> throw new TableOfflineException(instance, tableId);
>
> BatchWriter tbw = tableWriters.get(tableId);
> if (tbw == null) {
> tbw = new TableBatchWriter(tableId);
> tableWriters.put(tableId, tbw);
> }
> return tbw;
> }
> {code}
> I recommend moving the synchronized block to happen only if the batchwriter
> is not present, and also only checking if the table is online at that time:
> {code}
> @Override
> public BatchWriter getBatchWriter(String tableName) throws
> AccumuloException, AccumuloSecurityException, TableNotFoundException {
> ArgumentChecker.notNull(tableName);
> String tableId = Tables.getNameToIdMap(instance).get(tableName);
> if (tableId == null)
> throw new TableNotFoundException(tableId, tableName, null);
> BatchWriter tbw = tableWriters.get(tableId);
> if (tbw == null) {
> if (Tables.getTableState(instance, tableId) == TableState.OFFLINE)
> throw new TableOfflineException(instance, tableId);
> tbw = new TableBatchWriter(tableId);
> synchronized(tableWriters){
> //only create a new table writer if we haven't been beaten to it.
> if (tableWriters.get(tableId) == null)
> tableWriters.put(tableId, tbw);
> }
> }
> return tbw;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
