[jira] [Commented] (ACCUMULO-4469) ConcurrentModificationException while running MultiTable.xml node in Random Walk
[
https://issues.apache.org/jira/browse/ACCUMULO-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515418#comment-15515418
]
Dima Spivak commented on ACCUMULO-4469:
---
Hm... looked into the {{State}} class and realized it's just wrapping a
{{HashMap}}, so not sure why it would be getting updated underneath me. :(
> ConcurrentModificationException while running MultiTable.xml node in Random
> Walk
> -
>
> Key: ACCUMULO-4469
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4469
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.7.2
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> After the resolution of ACCUMULO-4467, I got back to playing with Random Walk
> and had a failure caused by a {{ConcurrentModificationException}}:
> {code}
> 23 01:03:04,316 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node MultiTable.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at
> org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:157)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.accumulo.test.randomwalk.multitable.MultiTableFixture.tearDown(MultiTableFixture.java:64)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:365)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> {code}
> [This section of
> code|https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/randomwalk/multitable/MultiTableFixture.java#L61-L71]
> seems to be at fault. In particular, it looks like we're getting the table
> list, but then instead of doing a deep copy to a new {{ArrayList}}
> from which we choose tables to delete, we're looping through and deleting
> tables while referring to the changing list, which has the effect of
> modifying it and making Java unhappy. Am I missing something more complex or
> can I fix this one myself by just doing the aforementioned deep copy of the
> table list? Or is a better way to use the {{TableOperations.list()}} method
> and iterate through the {{SortedSet}} it provides?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (ACCUMULO-4469) ConcurrentModificationException while running MultiTable.xml node in Random Walk
[
https://issues.apache.org/jira/browse/ACCUMULO-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak reassigned ACCUMULO-4469:
-
Assignee: Dima Spivak
> ConcurrentModificationException while running MultiTable.xml node in Random
> Walk
> -
>
> Key: ACCUMULO-4469
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4469
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.7.2
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> After the resolution of ACCUMULO-4467, I got back to playing with Random Walk
> and had a failure caused by a {{ConcurrentModificationException}}:
> {code}
> 23 01:03:04,316 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node MultiTable.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at
> org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:157)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
> at java.util.ArrayList$Itr.next(ArrayList.java:831)
> at
> org.apache.accumulo.test.randomwalk.multitable.MultiTableFixture.tearDown(MultiTableFixture.java:64)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:365)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> {code}
> [This section of
> code|https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/randomwalk/multitable/MultiTableFixture.java#L61-L71]
> seems to be at fault. In particular, it looks like we're getting the table
> list, but then instead of doing a deep copy to a new {{ArrayList}}
> from which we choose tables to delete, we're looping through and deleting
> tables while referring to the changing list, which has the effect of
> modifying it and making Java unhappy. Am I missing something more complex or
> can I fix this one myself by just doing the aforementioned deep copy of the
> table list? Or is a better way to use the {{TableOperations.list()}} method
> and iterate through the {{SortedSet}} it provides?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (ACCUMULO-4469) ConcurrentModificationException while running MultiTable.xml node in Random Walk
[
https://issues.apache.org/jira/browse/ACCUMULO-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dima Spivak updated ACCUMULO-4469:
--
Description:
After the resolution of ACCUMULO-4467, I got back to playing with Random Walk
and had a failure caused by a {{ConcurrentModificationException}}:
{code}
23 01:03:04,316 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node MultiTable.xml
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
at
org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$2.run(Main.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.accumulo.test.randomwalk.multitable.MultiTableFixture.tearDown(MultiTableFixture.java:64)
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:365)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
{code}
[This section of
code|https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/randomwalk/multitable/MultiTableFixture.java#L61-L71]
seems to be at fault. In particular, it looks like we're getting the table
list, but then instead of doing a deep copy to a new {{ArrayList}} from
which we choose tables to delete, we're looping through and deleting tables
while referring to the changing list, which has the effect of modifying it and
making Java unhappy. Am I missing something more complex or can I fix this one
myself by just doing the aforementioned deep copy of the table list? Or is a
better way to use the {{TableOperations.list()}} method and iterate through the
{{SortedSet}} it provides?
was:
After the resolution of ACCUMULO-4467, I got back to playing with Random Walk
and had a failure caused by a {{ConcurrentModificationException}}:
{code}
23 01:03:04,316 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node MultiTable.xml
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
at
org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$2.run(Main.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.accumulo.test.randomwalk.multitable.MultiTableFixture.tearDown(MultiTableFixture.java:64)
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:365)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
{code}
[This section of
[jira] [Created] (ACCUMULO-4469) ConcurrentModificationException while running MultiTable.xml node in Random Walk
Dima Spivak created ACCUMULO-4469:
-
Summary: ConcurrentModificationException while running
MultiTable.xml node in Random Walk
Key: ACCUMULO-4469
URL: https://issues.apache.org/jira/browse/ACCUMULO-4469
Project: Accumulo
Issue Type: Bug
Components: test
Affects Versions: 1.7.2
Reporter: Dima Spivak
After the resolution of ACCUMULO-4467, I got back to playing with Random Walk
and had a failure caused by a {{ConcurrentModificationException}}:
{code}
23 01:03:04,316 [randomwalk.Framework] ERROR: Error during random walk
java.lang.Exception: Error running node MultiTable.xml
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
at
org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$2.run(Main.java:157)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.accumulo.test.randomwalk.multitable.MultiTableFixture.tearDown(MultiTableFixture.java:64)
at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:365)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
{code}
[This section of
code|https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/randomwalk/multitable/MultiTableFixture.java#L61-L71]
seems to be at fault. In particular, it looks like we're getting the table
list, but then instead of doing a deep copy to a new {{ArrayList}} from
which we choose tables to delete, we're looping through and deleting tables
while referring to the changing list, which has the effect of modifying it and
making Java unhappy. Am I missing something more complex or can I fix this one
myself by just doing the aforementioned deep copy of the table list? Or is a
better way to go to use the {{TableOperations.list()}} method and iterate
through the {{SortedSet}} it provides? Looks like this pattern is used
in a few other
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Accumulo-Pull-Requests - Build # 449 - Aborted
The Apache Jenkins build system has built Accumulo-Pull-Requests (build #449) Status: Aborted Check console output at https://builds.apache.org/job/Accumulo-Pull-Requests/449/ to view the results.
Accumulo-1.8 - Build # 107 - Aborted
The Apache Jenkins build system has built Accumulo-1.8 (build #107) Status: Aborted Check console output at https://builds.apache.org/job/Accumulo-1.8/107/ to view the results.
[jira] [Commented] (ACCUMULO-1280) Add close method to iterators
[ https://issues.apache.org/jira/browse/ACCUMULO-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514035#comment-15514035 ] Christopher Tubbs commented on ACCUMULO-1280: - With the 2.0 branch, we can actually make them all AutoCloseable now, and provide a default close() method, because we're now on Java 8. > Add close method to iterators > - > > Key: ACCUMULO-1280 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1280 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Michael Miller > Fix For: 2.0.0 > > > It would be useful if Accumulo iterators had a close method. Accumulo would > call this when its finished using the iterator stack. > How would this work w/ isolation? > Is it ok to break the iterator API? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (ACCUMULO-4467) Random Walk broken because of unmet dependency on commons-math
[
https://issues.apache.org/jira/browse/ACCUMULO-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated ACCUMULO-4467:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Random Walk broken because of unmet dependency on commons-math
> --
>
> Key: ACCUMULO-4467
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4467
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.6.6, 1.7.2
>Reporter: Dima Spivak
>Assignee: Sean Busbey
> Fix For: 1.7.3, 1.8.1, 2.0.0
>
> Attachments: ACCUMULO-4467-1.6.v1.patch, ACCUMULO-4467-1.7.v1.patch,
> ACCUMULO-4467-1.8.v1.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When trying to run the Random Walk with {{LongEach.xml}} module, I hit a
> failure once we reach the {{Shard.xml}} step:
> {code}
> 16 19:52:05,146 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Shard.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:157)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Error running node shard.BulkInsert
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> Caused by: java.lang.Exception: Failed to run map/red verify
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.sort(BulkInsert.java:186)
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.visit(BulkInsert.java:132)
> ... 9 more
> {code}
> Digging into YARN to see why the MR job became unhappy, I see the following:
> {code}
> Error: java.lang.ClassNotFoundException:
> org.apache.commons.math.stat.descriptive.SummaryStatistics at
> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
> java.security.AccessController.doPrivileged(Native Method) at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
> org.apache.accumulo.core.file.rfile.RFile$Writer.(RFile.java:310) at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:127)
> at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:106)
> at
> org.apache.accumulo.core.file.DispatchingFileFactory.openWriter(DispatchingFileFactory.java:78)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:172)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:152)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150) at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
>
[jira] [Commented] (ACCUMULO-4467) Random Walk broken because of unmet dependency on commons-math
[
https://issues.apache.org/jira/browse/ACCUMULO-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513960#comment-15513960
]
Christopher Tubbs commented on ACCUMULO-4467:
-
Yeah, the last commit on that branch was included in the 1.6.6 tag, and merged
forward. If we *had* to, we could branch again from the tag, to apply fixes...
but I don't think we should do that unless there's a *really* serious issue to
address. So happy to have immediately dropped Java 6 and LaTeX from my
build/test environments... :)
> Random Walk broken because of unmet dependency on commons-math
> --
>
> Key: ACCUMULO-4467
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4467
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.6.6, 1.7.2
>Reporter: Dima Spivak
>Assignee: Sean Busbey
> Fix For: 1.7.3, 1.8.1, 2.0.0
>
> Attachments: ACCUMULO-4467-1.6.v1.patch, ACCUMULO-4467-1.7.v1.patch,
> ACCUMULO-4467-1.8.v1.patch
>
>
> When trying to run the Random Walk with {{LongEach.xml}} module, I hit a
> failure once we reach the {{Shard.xml}} step:
> {code}
> 16 19:52:05,146 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Shard.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:157)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Error running node shard.BulkInsert
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> Caused by: java.lang.Exception: Failed to run map/red verify
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.sort(BulkInsert.java:186)
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.visit(BulkInsert.java:132)
> ... 9 more
> {code}
> Digging into YARN to see why the MR job became unhappy, I see the following:
> {code}
> Error: java.lang.ClassNotFoundException:
> org.apache.commons.math.stat.descriptive.SummaryStatistics at
> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
> java.security.AccessController.doPrivileged(Native Method) at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
> org.apache.accumulo.core.file.rfile.RFile$Writer.(RFile.java:310) at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:127)
> at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:106)
> at
> org.apache.accumulo.core.file.DispatchingFileFactory.openWriter(DispatchingFileFactory.java:78)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:172)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:152)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150) at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at
>
[jira] [Commented] (ACCUMULO-4467) Random Walk broken because of unmet dependency on commons-math
[
https://issues.apache.org/jira/browse/ACCUMULO-4467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513943#comment-15513943
]
Sean Busbey commented on ACCUMULO-4467:
---
{quote}
-0 for 1.6 (let it die)
{quote}
[~ctubbsii] has now cleaned up the 1.6 branch, so I'll skip 1.6. :)
> Random Walk broken because of unmet dependency on commons-math
> --
>
> Key: ACCUMULO-4467
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4467
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.6.6, 1.7.2
>Reporter: Dima Spivak
>Assignee: Sean Busbey
> Fix For: 1.7.3, 1.8.1, 2.0.0
>
> Attachments: ACCUMULO-4467-1.6.v1.patch, ACCUMULO-4467-1.7.v1.patch,
> ACCUMULO-4467-1.8.v1.patch
>
>
> When trying to run the Random Walk with {{LongEach.xml}} module, I hit a
> failure once we reach the {{Shard.xml}} step:
> {code}
> 16 19:52:05,146 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Shard.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:157)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Error running node shard.BulkInsert
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> Caused by: java.lang.Exception: Failed to run map/red verify
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.sort(BulkInsert.java:186)
> at
> org.apache.accumulo.test.randomwalk.shard.BulkInsert.visit(BulkInsert.java:132)
> ... 9 more
> {code}
> Digging into YARN to see why the MR job became unhappy, I see the following:
> {code}
> Error: java.lang.ClassNotFoundException:
> org.apache.commons.math.stat.descriptive.SummaryStatistics at
> java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
> java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
> java.security.AccessController.doPrivileged(Native Method) at
> java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
> org.apache.accumulo.core.file.rfile.RFile$Writer.(RFile.java:310) at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:127)
> at
> org.apache.accumulo.core.file.rfile.RFileOperations.openWriter(RFileOperations.java:106)
> at
> org.apache.accumulo.core.file.DispatchingFileFactory.openWriter(DispatchingFileFactory.java:78)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:172)
> at
> org.apache.accumulo.core.client.mapreduce.AccumuloFileOutputFormat$1.write(AccumuloFileOutputFormat.java:152)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
> at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150) at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
[jira] [Commented] (ACCUMULO-4468) accumulo.core.data.Key.equals(Key, PartialKey) improvement
[ https://issues.apache.org/jira/browse/ACCUMULO-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513616#comment-15513616 ] Keith Turner commented on ACCUMULO-4468: The network code uses the [Key.compress()|https://github.com/apache/accumulo/blob/1a663143c4d81367f7703bceb8b24374d59c154e/core/src/main/java/org/apache/accumulo/core/data/Key.java#L1085] and [Key.decompress()|https://github.com/apache/accumulo/blob/1a663143c4d81367f7703bceb8b24374d59c154e/core/src/main/java/org/apache/accumulo/core/data/Key.java#L1137] functions, maybe the test could use that to simulate what rfile does. Its based on thrift stuff though, so it would clunky to use. > accumulo.core.data.Key.equals(Key, PartialKey) improvement > -- > > Key: ACCUMULO-4468 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4468 > Project: Accumulo > Issue Type: Improvement > Components: core >Affects Versions: 1.8.0 >Reporter: Will Murnane >Priority: Trivial > Labels: newbie, performance > Attachments: benchmark.tar.gz, key_comparison.patch > > > In the Key.equals(Key, PartialKey) overload, the current method compares > starting at the beginning of the key, and works its way toward the end. This > functions correctly, of course, but one of the typical uses of this method is > to compare adjacent rows to break them into larger chunks. For example, > accumulo.core.iterators.Combiner repeatedly calls this method with subsequent > pairs of keys. > I have a patch which reverses the comparison order. That is, if the method is > called with ROW_COLFAM_COLQUAL_COLVIS, it will compare visibility, cq, cf, > and finally row. This (marginally) improves the speed of comparisons in the > relatively common case where only the last part is changing, with less > complex code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-4468) accumulo.core.data.Key.equals(Key, PartialKey) improvement
[ https://issues.apache.org/jira/browse/ACCUMULO-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513602#comment-15513602 ] Keith Turner commented on ACCUMULO-4468: bq. That said, maybe there would be benefits to storing all the pieces of the Key in a single byte array, and maintaining indices into it to track the individual parts, rather than several smaller arrays... Key used to be like that (one large byte array with pointers). I changed it when adding support for relative compression in rfile. The reasoning behind the change was so that when rfile deserializes a key and a field is the same as the last key, it can just point to the previous byte array. This makes equality comparisons on rows or columns that are the same really fast (because the byte array is the same and equality checks that). The code that serializes keys and transfers them across the network also does this. So it may be interesting to have the test stream keys from an RFile. bq. I think it would be worth revisiting the comparison mechanism in isEqual, too, doing something like the Unsafe method used in Hadoop's FastByteComparisons class but going in reverse. Thats sounds like an interesting line of investigation. The compare methods could also leverage this technique. We may have an issue open for this already. > accumulo.core.data.Key.equals(Key, PartialKey) improvement > -- > > Key: ACCUMULO-4468 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4468 > Project: Accumulo > Issue Type: Improvement > Components: core >Affects Versions: 1.8.0 >Reporter: Will Murnane >Priority: Trivial > Labels: newbie, performance > Attachments: benchmark.tar.gz, key_comparison.patch > > > In the Key.equals(Key, PartialKey) overload, the current method compares > starting at the beginning of the key, and works its way toward the end. This > functions correctly, of course, but one of the typical uses of this method is > to compare adjacent rows to break them into larger chunks. For example, > accumulo.core.iterators.Combiner repeatedly calls this method with subsequent > pairs of keys. > I have a patch which reverses the comparison order. That is, if the method is > called with ROW_COLFAM_COLQUAL_COLVIS, it will compare visibility, cq, cf, > and finally row. This (marginally) improves the speed of comparisons in the > relatively common case where only the last part is changing, with less > complex code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ACCUMULO-4468) accumulo.core.data.Key.equals(Key, PartialKey) improvement
[ https://issues.apache.org/jira/browse/ACCUMULO-4468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513435#comment-15513435 ] Will Murnane commented on ACCUMULO-4468: [~elserj] I'm not sure why customVanilla and standardEquals behave differently. The difference is small, and perhaps it's due to variance in the JDK used to compile the standard Accumulo JAR versus the one used to compile the benchmark code? Maybe there are effects from having the code loaded from a small JAR versus a large one? Maybe the custom WillKey class gets laid out in memory differently, and it hits instruction cache differently? This is the problem of benchmarking... RE: generation of data, yeah, the current test data... leaves something to be desired. This was basically the least-worst mechanism I could come up with in 5 minutes to generate some test data that kinda-sorta resembles our production data. If anyone has a better strategy I'm willing to do a little legwork testing other data sets. [~kturner] The parts of the key are stored on the heap somewhere, so the problem of row equality is somewhat different than the problem of comparing two contiguous byte arrays. That said, maybe there would be benefits to storing all the pieces of the Key in a single byte array, and maintaining indices into it to track the individual parts, rather than several smaller arrays... That's a big refactor, though, for an unknown change in performance. I think it would be worth revisiting the comparison mechanism in isEqual, too, doing something like the Unsafe method used in Hadoop's FastByteComparisons class but going in reverse. The CPU's speculative prefetch should work in either direction, but doing the comparison byte-at-a-time is going to be more expensive than the 64-bit comparisons that FastByteComparisons does. But that's a topic for another ticket ;) > accumulo.core.data.Key.equals(Key, PartialKey) improvement > -- > > Key: ACCUMULO-4468 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4468 > Project: Accumulo > Issue Type: Improvement > Components: core >Affects Versions: 1.8.0 >Reporter: Will Murnane >Priority: Trivial > Labels: newbie, performance > Attachments: benchmark.tar.gz, key_comparison.patch > > > In the Key.equals(Key, PartialKey) overload, the current method compares > starting at the beginning of the key, and works its way toward the end. This > functions correctly, of course, but one of the typical uses of this method is > to compare adjacent rows to break them into larger chunks. For example, > accumulo.core.iterators.Combiner repeatedly calls this method with subsequent > pairs of keys. > I have a patch which reverses the comparison order. That is, if the method is > called with ROW_COLFAM_COLQUAL_COLVIS, it will compare visibility, cq, cf, > and finally row. This (marginally) improves the speed of comparisons in the > relatively common case where only the last part is changing, with less > complex code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
