[jira] [Resolved] (ACCUMULO-624) iterators may open lots of compressors
[ https://issues.apache.org/jira/browse/ACCUMULO-624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-624. Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > iterators may open lots of compressors > -- > > Key: ACCUMULO-624 > URL: https://issues.apache.org/jira/browse/ACCUMULO-624 > Project: Accumulo > Issue Type: Bug > Components: tserver >Reporter: Eric C. Newton >Priority: Major > > A large iterator tree may create many instances of Compressors. These > instances are pulled from a pool that never decreases in size. So, if 50 > simultaneous queries are run over dozens of files, each with a complex > iterator stack, there will be thousands of compressors created. Each of > these holds a large buffer. This can cause the server to run out of memory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (ACCUMULO-4022) Create a concept of multi-homed tablets
[ https://issues.apache.org/jira/browse/ACCUMULO-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17222602#comment-17222602 ] Christopher Tubbs edited comment on ACCUMULO-4022 at 10/28/20, 11:18 PM: - Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo was (Author: ctubbsii): Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Create a concept of multi-homed tablets > --- > > Key: ACCUMULO-4022 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4022 > Project: Accumulo > Issue Type: Wish > Components: client, tserver >Reporter: Marc Parisi >Priority: Major > Labels: performance > > I'm an accumulo newbie, but I wish to see the concept of multi-homed tablets. > This allows us to have tablets hosted by multiple servers, with only one > being writable against it. This concept would allow n receiver servers for a > tablet. An example might be a tablet that has become a hot spot could be > dynamically hosted elsewhere, and clients could pick this up as a potential. > Consistency must be kept between the hosts, as the initial read/write host > may compact or write to that tablet. > To me the larger problem may come from live ingest in which the write ahead > log has not been flushed. To avoid having to write to the read only servers > in a pipeline, we would likely need to create a model of enforcing reads only > after a flush of that tablet or a thrift interface to allow reading only the > data in memory to ensure consistency is enforced. I haven't given great > thought to solving this yet. > Please comment with ideas and pitfalls as I would like to see this wish come > to fruition with actionable tickets after some community thought. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4408) Improve the agitator
[ https://issues.apache.org/jira/browse/ACCUMULO-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4408. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Improve the agitator > > > Key: ACCUMULO-4408 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4408 > Project: Accumulo > Issue Type: Task >Reporter: Dave Marion >Priority: Major > > It looks like code is duplicated in the perl scripts. We should be able to > reduce this and add some host specific actions regarding cpu and network like > Netflix's ChaosMonkey does. See > https://github.com/Netflix/SimianArmy/tree/master/src/main/resources/scripts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-884) Insight into short circuit read for local files
[ https://issues.apache.org/jira/browse/ACCUMULO-884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-884. Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Insight into short circuit read for local files > --- > > Key: ACCUMULO-884 > URL: https://issues.apache.org/jira/browse/ACCUMULO-884 > Project: Accumulo > Issue Type: Sub-task > Components: docs >Reporter: Billie Rinaldi >Priority: Major > > This is a new feature in hadoop 1.0.x and some versions of 0.22 and 0.23. It > allows a client to read directly from disk instead of through a DataNode when > the data is stored locally. Enabling it involves setting two configuration > parameters, the first in hdfs-site.xml and the second in accumulo-site.xml. > We should make sure this works with Accumulo and recommend it in the > documentation. > - dfs.block.local-path-access.user is the key in datanode configuration to > specify the user allowed to do short circuit read. > - dfs.client.read.shortcircuit is the key to enable short circuit read at the > client side configuration. > See HDFS-2246 and http://hbase.apache.org/book/perf.hdfs.configs.html for > more information. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4022) Create a concept of multi-homed tablets
[ https://issues.apache.org/jira/browse/ACCUMULO-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4022. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Create a concept of multi-homed tablets > --- > > Key: ACCUMULO-4022 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4022 > Project: Accumulo > Issue Type: Wish > Components: client, tserver >Reporter: Marc Parisi >Priority: Major > Labels: performance > > I'm an accumulo newbie, but I wish to see the concept of multi-homed tablets. > This allows us to have tablets hosted by multiple servers, with only one > being writable against it. This concept would allow n receiver servers for a > tablet. An example might be a tablet that has become a hot spot could be > dynamically hosted elsewhere, and clients could pick this up as a potential. > Consistency must be kept between the hosts, as the initial read/write host > may compact or write to that tablet. > To me the larger problem may come from live ingest in which the write ahead > log has not been flushed. To avoid having to write to the read only servers > in a pipeline, we would likely need to create a model of enforcing reads only > after a flush of that tablet or a thrift interface to allow reading only the > data in memory to ensure consistency is enforced. I haven't given great > thought to solving this yet. > Please comment with ideas and pitfalls as I would like to see this wish come > to fruition with actionable tickets after some community thought. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3808) Sequential Bulk RW modules might error
[
https://issues.apache.org/jira/browse/ACCUMULO-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3808.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> Sequential Bulk RW modules might error
> --
>
> Key: ACCUMULO-3808
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3808
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.7.0
>Reporter: Josh Elser
>Priority: Minor
> Labels: 1.7.0_QA
>
> Noticed the following while running RandomWalk LongClean.xml:
> We ran the Bulk.xml file for a while, eventually, the 3600s timeout was hit
> and the module was torn stopped:
> {noformat}
> 11 22:29:20,566 [randomwalk.Module] INFO : Timed out waiting for Split to
> complete (waited 30 seconds). Exiting.
> java.util.concurrent.TimeoutException
> at java.util.concurrent.FutureTask.get(FutureTask.java:201)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:298)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We went right back into Bulk after that.
> {noformat}
> 11 22:29:20,567 [bulk.Setup] INFO : Starting bulk test on
> bulk_cn020_l42scl_hortonworks_com_23476_1431408560567
> {noformat}
> We near immediately errored out after that
> {noformat}
> 11 22:29:24,466 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Bulk.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at
> org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:59)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:119)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$2.run(Main.java:130)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Error running node bulk.BulkPlusOne
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:346)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:283)
> at org.apache.accumulo.test.randomwalk.Module$1.call(Module.java:278)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> ... 1 more
> Caused by: java.lang.IllegalStateException: Should not have a skipped import
> marker for a class other than
> org.apache.accumulo.test.randomwalk.bulk.BulkMinusOne but was
> org.apache.accumulo.test.randomwalk.bulk.BulkPlusOne
> at
> org.apache.accumulo.test.randomwalk.bulk.BulkImportTest.visit(BulkImportTest.java:44)
> ... 9 more
> {noformat}
> I believe the error is due to the State containing the old value for
> SKIPPED_IMPORT from the previous Bulk.xml invocation. Bulk.Setup should
> likely remove any state from previous runs since a new State is not passed in
> for each invocation of a new module.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2256) RW stops with commitsHeld
[
https://issues.apache.org/jira/browse/ACCUMULO-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2256.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> RW stops with commitsHeld
> -
>
> Key: ACCUMULO-2256
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2256
> Project: Accumulo
> Issue Type: Improvement
> Components: test
>Reporter: Eric C. Newton
>Priority: Major
>
> Random walker stopped with this error:
> {noformat}
> java.lang.Exception: Error running node Conditional.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at
> org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:65)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.accumulo.start.Main$1.run(Main.java:137)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.Exception: Error running node ct.Transfer
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 8 more
> Caused by: org.apache.accumulo.core.client.AccumuloException:
> org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server
> node8:55823
> at
> org.apache.accumulo.core.client.ConditionalWriter$Result.getStatus(ConditionalWriter.java:63)
> at
> org.apache.accumulo.test.randomwalk.conditional.Transfer.visit(Transfer.java:124)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 9 more
> Caused by: org.apache.accumulo.core.client.impl.AccumuloServerException:
> Error on server nebula8.cerulean.tycho.ncsc.mil:55823
> at
> org.apache.accumulo.core.client.impl.ConditionalWriterImpl.sendToServer(ConditionalWriterImpl.java:611)
> at
> org.apache.accumulo.core.client.impl.ConditionalWriterImpl.access$4(ConditionalWriterImpl.java:552)
> at
> org.apache.accumulo.core.client.impl.ConditionalWriterImpl$SendTask.run(ConditionalWriterImpl.java:448)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> conditionalUpdate
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_conditionalUpdate(TabletClientService.java:521)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.conditionalUpdate(TabletClientService.java:505)
> at
> org.apache.accumulo.core.client.impl.ConditionalWriterImpl.sendToServer(ConditionalWriterImpl.java:575)
> ... 11 more
> {noformat}
> Going over to node8, this is the only error near the same timestamp. The
> servers are not time synced.
> {noformat}
> 2014-01-24 21:09:08,207 [thrift.ProcessFunction] ERROR: Internal error
> processing applyUpdates
> org.apache.accumulo.tserver.HoldTimeoutException: Commits are held
> at
> org.apache.accumulo.tserver.TabletServerResourceManager.waitUntilCommitsAreEnabled(TabletServerResourceManager.java:412)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.flush(TabletServer.java:1552)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1531)
> at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMet
[jira] [Resolved] (ACCUMULO-3171) Table props not equal after import/export during RW
[
https://issues.apache.org/jira/browse/ACCUMULO-3171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3171.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> Table props not equal after import/export during RW
> ---
>
> Key: ACCUMULO-3171
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3171
> Project: Accumulo
> Issue Type: Bug
>Affects Versions: 1.6.1
>Reporter: Keith Turner
>Priority: Major
>
> {noformat}
> 24 21:48:15,861 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Shard.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:63)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$1.run(Main.java:141)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.Exception: Error running node shard.ExportIndex
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 8 more
> Caused by: java.lang.Exception: Props not equals
> ST_index_ip_10_1_2_15_ec2_internal_11483_1411593063602
> ST_index_ip_10_1_2_15_ec2_internal_11483_1411593063602_tmp
> at
> org.apache.accumulo.test.randomwalk.shard.ExportIndex.visit(ExportIndex.java:105)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 9 more
> {noformat}
> I thought this may be a problem w/ comparing eventually consistent config too
> quickly. so I diffed the two tables config in the shell and saw the
> following.
> {noformat}
> [cluster@ip-10-1-2-10 accumulo-1.6.1]$ ./bin/accumulo shell -u root -p secret
> -e 'config -t ST_index_ip_10_1_2_15_ec2_internal_11483_1411593063602' >
> t1_config.txt
> [cluster@ip-10-1-2-10 accumulo-1.6.1]$ ./bin/accumulo shell -u root -p secret
> -e 'config -t ST_index_ip_10_1_2_15_ec2_internal_11483_1411593063602_tmp' >
> t2_config.txt
> [cluster@ip-10-1-2-10 accumulo-1.6.1]$ diff -u t1_config.txt t2_config.txt
> --- t1_config.txt 2014-09-25 19:17:46.347161854 +
> +++ t2_config.txt 2014-09-25 19:17:57.392161854 +
> @@ -1,4 +1,4 @@
> -2014-09-25 19:17:44,032 [util.NativeCodeLoader] WARN : Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> +2014-09-25 19:17:55,037 [util.NativeCodeLoader] WARN : Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
>
> ---+-+---
> SCOPE | NAME| VALUE
>
> ---+-+---
> @@ -21,6 +21,7 @@
> default| table.failures.ignore . | false
> default| table.file.blocksize .. | 0B
> default| table.file.compress.blocksize . | 100K
> +table |@override .. | 2117999
> default| table.file.compress.blocksize.index ... | 128K
> default| table.file.compress.type .. | gz
> default| table.file.max | 15
> {noformat}
> I think the 2nd table was created by import. Somehow it ended up with
> table.file.compress.blocksize=2117999 when the original did not have that
> set. Maybe another walker set this on the namespace?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3170) RW failed with Unexpected table state mo DELETING != OFFLINE
[
https://issues.apache.org/jira/browse/ACCUMULO-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3170.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> RW failed with Unexpected table state mo DELETING != OFFLINE
>
>
> Key: ACCUMULO-3170
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3170
> Project: Accumulo
> Issue Type: Bug
>Affects Versions: 1.6.0, 1.6.1
>Reporter: Keith Turner
>Priority: Major
>
> Saw the following when running random walk, may just be a test bug.
> {noformat}
> 24 19:45:47,001 [randomwalk.Framework] ERROR: Error during random walk
> java.lang.Exception: Error running node Concurrent.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:63)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:122)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$1.run(Main.java:141)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.Exception: Error running node ct.OfflineTable
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 8 more
> Caused by: org.apache.accumulo.core.client.AccumuloException: Unexpected
> table state mo DELETING != OFFLINE
> at
> org.apache.accumulo.core.client.impl.TableOperationsImpl.waitForTableStateTransition(TableOperationsImpl.java:1196)
> at
> org.apache.accumulo.core.client.impl.TableOperationsImpl.offline(TableOperationsImpl.java:1330)
> at
> org.apache.accumulo.test.randomwalk.concurrent.OfflineTable.visit(OfflineTable.java:43)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2698) Improve data loss detection ability of Conditional RW test
[ https://issues.apache.org/jira/browse/ACCUMULO-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2698. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Improve data loss detection ability of Conditional RW test > -- > > Key: ACCUMULO-2698 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2698 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Assignee: Keith Turner >Priority: Major > > The Conditional RW test has roughly two phases. First it inserts a lot of > "banks" where each bank is a row. The columns in the row represent account > balances. After insertion of all banks is complete, then it starts > transferring money within a bank. If a mutation is silently lost in the > insert phase, the test can detect this. If a mutation is silently lost in > the transfer phase, then the test can not always detect this. The test > spends most of its time transferring, so it would be good to modify the test > to better detect data loss there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1020) Make it easy to run randomwalk against MiniAccumuloCluster
[ https://issues.apache.org/jira/browse/ACCUMULO-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1020. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Make it easy to run randomwalk against MiniAccumuloCluster > -- > > Key: ACCUMULO-1020 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1020 > Project: Accumulo > Issue Type: Improvement > Components: test >Reporter: Keith Turner >Priority: Major > > Being able to easily spin up N random walkers against a mini accumulo cluster > would be useful. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2769) Concurrent randomwalk fail with multiple walkers: mutations rejected
[
https://issues.apache.org/jira/browse/ACCUMULO-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2769.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> Concurrent randomwalk fail with multiple walkers: mutations rejected
>
>
> Key: ACCUMULO-2769
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2769
> Project: Accumulo
> Issue Type: Test
> Components: test
>Affects Versions: 1.6.0
>Reporter: Bill Havanki
>Priority: Minor
>
> Running concurrent randomwalk with two walkers, one dies with:
> {noformat}
> java.lang.Exception: Error running node ct.BatchWrite
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:286)
> ...
> Caused by: org.apache.accumulo.core.client.MutationsRejectedException: #
> constraint violations : 0 security codes: {} # server errors 1 # exceptions > 0
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.close(TabletServerBatchWriter.java:354)
> at
> org.apache.accumulo.core.client.impl.BatchWriterImpl.close(BatchWriterImpl.java:54)
> at
> org.apache.accumulo.test.randomwalk.concurrent.BatchWrite.visit(BatchWrite.java:63)
> ...
> {noformat}
> And earlier in the walker log:
> {noformat}
> 01 09:43:35,917 [impl.TabletServerBatchWriter] ERROR: Server side error on
> a1234.cloudera.com:10011: org.apache.thrift.TApplicationException: Internal
> error processing applyUpdates
> 01 09:43:35,918 [impl.TabletServerBatchWriter] ERROR: Failed to send tablet
> server a1234.cloudera.com:10011 its batch : Error on server
> a1234.cloudera.com:10011
> org.apache.accumulo.core.client.impl.AccumuloServerException: Error on server
> a1234.cloudera.com:10011
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:937)
> ...
> {noformat}
> On that tablet server:
> {noformat}
> 2014-05-01 09:43:35,863 [thrift.ProcessFunction] ERROR: Internal error
> processing applyUpdates
> java.lang.IllegalArgumentException: Table with id evr does not exist
> at
> org.apache.accumulo.core.client.impl.Tables.getNamespaceId(Tables.java:307)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.setUpdateTablet(TabletServer.java:1483)
> at
> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1527)
> ...
> {noformat}
> Logs on tablet servers indicate the tablets for evr had just been unloaded
> and closed. Maybe the table was deleted by one walker and the other one
> hadn't noticed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2236) Add table cleanup to Concurrent randomwalk test
[ https://issues.apache.org/jira/browse/ACCUMULO-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2236. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Add table cleanup to Concurrent randomwalk test > --- > > Key: ACCUMULO-2236 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2236 > Project: Accumulo > Issue Type: Improvement > Components: test >Affects Versions: 1.4.4 >Reporter: Bill Havanki >Priority: Trivial > Labels: newbie, randomwalk, test > > When the Concurrent randomwalk test ends, it will almost always leave behind > some of its test tables (ctt_000, ctt_001, and so on). Add a finishing step, > on the way to the "END" state, to delete any leftover tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2610) Improve logger.xml example for randomwalk
[ https://issues.apache.org/jira/browse/ACCUMULO-2610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2610. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Improve logger.xml example for randomwalk > - > > Key: ACCUMULO-2610 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2610 > Project: Accumulo > Issue Type: Test > Components: test >Reporter: Bill Havanki >Priority: Trivial > Labels: newbie > > Make the example logger.xml for randomwalk a good default: > * comment out usage of the remote log appender in the loggers, since the > appender itself is already commented out > * bump up the log level for o.a.a.core.file.rfile.bcfile.Compression to at > least INFO, to keep it from spamming the logs with unhelpful debug messages > * whatever else seems like a good idea -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2081) Add configuration to HDFS location for randomwalk config tarball distribution
[ https://issues.apache.org/jira/browse/ACCUMULO-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2081. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Add configuration to HDFS location for randomwalk config tarball distribution > - > > Key: ACCUMULO-2081 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2081 > Project: Accumulo > Issue Type: Improvement > Components: test >Reporter: Josh Elser >Priority: Minor > Labels: newbie > > Randomwalk distributes a tarball of configuration files to each node > participating in the test using HDFS. There is no configuration item for the > directory used in HDFS to do this so it is required that the user running the > test can write to the root of HDFS (or you have to edit the scripts). > It would be nice to provide a directory that can be used or write to the > current user's home directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2193) Check for concurrency issue in randomwalk State
[
https://issues.apache.org/jira/browse/ACCUMULO-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2193.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> Check for concurrency issue in randomwalk State
> ---
>
> Key: ACCUMULO-2193
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2193
> Project: Accumulo
> Issue Type: Bug
> Components: test
>Affects Versions: 1.4.4, 1.5.0
>Reporter: Bill Havanki
>Priority: Minor
> Labels: concurrency, randomwalk, test
>
> Check for any concurrency issues with the {{State}} class used for randomwalk
> testing, as called out by [~mdrob] in this review:
> https://reviews.apache.org/r/16857/.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2095) Shard randomwalk test fails w/ multiple namenodes
[
https://issues.apache.org/jira/browse/ACCUMULO-2095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2095.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo-testing
> Shard randomwalk test fails w/ multiple namenodes
> -
>
> Key: ACCUMULO-2095
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2095
> Project: Accumulo
> Issue Type: Bug
> Components: test
> Environment: db746960fbcafb1651c15ec2e5493d56acb5065c
>Reporter: Keith Turner
>Priority: Major
>
> Was running random walk w/ two namenodes configured for Accumulo and saw the
> following error. I think this is just a bug in the test code not handling
> multiple namenodes.
> {noformat}
> java.lang.Exception: Error running node Shard.xml
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at
> org.apache.accumulo.test.randomwalk.Framework.run(Framework.java:65)
> at
> org.apache.accumulo.test.randomwalk.Framework.main(Framework.java:125)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.accumulo.start.Main$1.run(Main.java:137)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: java.lang.Exception: Error running node shard.ExportIndex
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:285)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> ... 8 more
> Caused by: java.lang.IllegalArgumentException: Wrong FS:
> hdfs://ip-10-1-3-11:9000/accumulo/tables/1l/t-9xq/F867.rf, expected:
> hdfs://ip-10-1-3-10:9000
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:181)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at
> org.apache.accumulo.test.randomwalk.shard.ExportIndex.visit(ExportIndex.java:74)
> at org.apache.accumulo.test.randomwalk.Module.visit(Module.java:254)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3705) Update security randomwalk module for Kerberos
[ https://issues.apache.org/jira/browse/ACCUMULO-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3705. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Update security randomwalk module for Kerberos > -- > > Key: ACCUMULO-3705 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3705 > Project: Accumulo > Issue Type: Improvement > Components: test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > > The security randomwalk module doesn't play well with Kerberos enabled > because it assumes passwords all over the place. > Need to update it so that it actually works with keytabs instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2286) Add namespace ops to security randomwalk
[ https://issues.apache.org/jira/browse/ACCUMULO-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2286. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Add namespace ops to security randomwalk > > > Key: ACCUMULO-2286 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2286 > Project: Accumulo > Issue Type: Bug > Components: test >Reporter: John Vines >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3974) Modify Randomwalk Bulk module to catch ACCUMULO-3967
[ https://issues.apache.org/jira/browse/ACCUMULO-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3974. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Modify Randomwalk Bulk module to catch ACCUMULO-3967 > > > Key: ACCUMULO-3974 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3974 > Project: Accumulo > Issue Type: Test > Components: test >Reporter: Josh Elser >Priority: Major > > [~ecn] asked me after I committed a fix for ACCUMULO-3967 "why didn't > randomwalk catch this bug?" > I think we could potentially have caught this if in Setup we pre-split the > table to be a large collection of sequentially increasing tablets. It's not a > guarantee catch (since the bug itself was only shown in the case of import > failures and the tablet distribution hasn't changed). > Alternatively, we could copy the existing bulk module into a new module. In > this module, we remove the splitting and merging, instead keeping a static > split distribution. This would be almost guaranteed to eventually recreate > the scenario. Adding in a chaotic balancer, even more likely to reproduce. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4723) Add HADOOP_CONF_DIR to classpath when running Random Walk tests in YARN
[ https://issues.apache.org/jira/browse/ACCUMULO-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4723. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-testing > Add HADOOP_CONF_DIR to classpath when running Random Walk tests in YARN > --- > > Key: ACCUMULO-4723 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4723 > Project: Accumulo > Issue Type: Bug > Components: test >Reporter: Mike Walch >Priority: Major > > HADOOP_CONF_DIR needs to be set on the classpath for running RW tests in > YARN. See ACCUMULO-4718 for background on in issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4692) CompactionDriver leaves abandoned metadata scans
[ https://issues.apache.org/jira/browse/ACCUMULO-4692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4692. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > CompactionDriver leaves abandoned metadata scans > > > Key: ACCUMULO-4692 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4692 > Project: Accumulo > Issue Type: Bug > Components: fate >Reporter: Adam Fuchs >Priority: Major > > We wrote a tool to kick off tablet compactions in the background while > minimizing compaction load per-server. The tool uses range compaction on one > tablet per call. We're seeing a high number of scans on the metadata table > (~7,000 on a ~100 node cluster). > The metadata query in the isReady() method of CompactionDriver that is used > to see if the compaction has completed uses a range that goes to the end of > the metadata entries for the given table, but it stops consuming the results > of the scanner at the end of the compaction range. isReady gets called in a > pretty tight loop, especially with hundreds of compactions running > concurrently. Seems like we should limit the scan to the metadata range > associated with the compaction so that the scan can get cleaned up quickly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-607) BatchScanner sometimes logs error when closed
[ https://issues.apache.org/jira/browse/ACCUMULO-607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-607. Resolution: Cannot Reproduce Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > BatchScanner sometimes logs error when closed > - > > Key: ACCUMULO-607 > URL: https://issues.apache.org/jira/browse/ACCUMULO-607 > Project: Accumulo > Issue Type: Bug > Components: client >Affects Versions: 1.3.5-incubating, 1.4.0 >Reporter: Keith Turner >Priority: Major > > The batch scanner has a threadpool. When the batch scanner is closed it > calls shutdownNow() on this threadpool, which interrupts all of the threads. > If a user does not read all data from a batch scanner and calls close, thens > its likely that the background threads are still busy and will be > interrupted. This is a normal course of business, so there is no need to log > an exception for these interrupted excpetions. Some places in the code do > log an error when these exceptions occur. > One place is where the background code calls UtilWaitThread.sleep(). There > may be other places. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4323) Experiment with key shortening in rfile index
[ https://issues.apache.org/jira/browse/ACCUMULO-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4323. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Experiment with key shortening in rfile index > - > > Key: ACCUMULO-4323 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4323 > Project: Accumulo > Issue Type: Improvement >Reporter: Keith Turner >Priority: Major > > In ACCUMULO-1124 shorter keys that may not exist in data were generated to > add to RFile indexes. The work done for ACCUMULO-1124 only covered the lowest > level of the RFile tree. > As keys are pushed up the RFile index tree, its possible that more shortening > could be done. Not sure if this is possible. Thought of it while working on > ACCUMULO-1124 and didn't have time to pursue. Opening this issue before I > forget about it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1016) Investigate additional knowledge that can be derived from continuous ingest tests
[ https://issues.apache.org/jira/browse/ACCUMULO-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1016. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Investigate additional knowledge that can be derived from continuous ingest > tests > - > > Key: ACCUMULO-1016 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1016 > Project: Accumulo > Issue Type: New Feature > Components: test >Reporter: Josh Elser >Priority: Major > > It's well known that the continuous-ingest suite is often a good place for > verifying the validity of an Accumulo installation (much akin to Terasort for > Hadoop). > The onus is on the user to actually extrapolate the necessary information > from the generated reports to say "I need to change property X because I saw > feature Y in a graph". I think we could do some automatic introspection on > the results of the test to report some concrete recommended changes that > could be made to increase performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2784) Experiment with adjusting default # of compaction files
[
https://issues.apache.org/jira/browse/ACCUMULO-2784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2784.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Experiment with adjusting default # of compaction files
> ---
>
> Key: ACCUMULO-2784
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2784
> Project: Accumulo
> Issue Type: Task
>Reporter: Keith Turner
>Priority: Major
> Labels: newbie
>
> The default for {{tserver.compaction.major.thread.files.open.max}} was chosen
> arbitrarily. Need to experiment with other default values for tablet that
> have lots of files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1522) investigate HBASE-8755 for improved WAL performance
[ https://issues.apache.org/jira/browse/ACCUMULO-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1522. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > investigate HBASE-8755 for improved WAL performance > --- > > Key: ACCUMULO-1522 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1522 > Project: Accumulo > Issue Type: Bug > Components: tserver >Reporter: Eric C. Newton >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3739) Investigate adding more information to Monitor
[ https://issues.apache.org/jira/browse/ACCUMULO-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3739. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Investigate adding more information to Monitor > -- > > Key: ACCUMULO-3739 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3739 > Project: Accumulo > Issue Type: Improvement >Reporter: Christopher Tubbs >Priority: Major > Labels: newbie > > Commit a194c0b00b7a927e938da72350916f30b0a0707d for ACCUMULO-3204 removed > some unused code in the Monitor. Some of this could possibly have been useful > to display. > These included things like: > * online tablet count > * total ingest byte rate > * total query byte rate > * recoveries over time > Some of this may be redundant to what's already displayed, and it is unknown > what degree of functionality the code removed provided, and what would need > to be supplemented. > An interested party could examine that code, and find a way to place it on > the monitor if it is useful to do so. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-932) Investigate setting thead context classloader
[ https://issues.apache.org/jira/browse/ACCUMULO-932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-932. Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-classloaders > Investigate setting thead context classloader > - > > Key: ACCUMULO-932 > URL: https://issues.apache.org/jira/browse/ACCUMULO-932 > Project: Accumulo > Issue Type: Task > Components: start >Reporter: Keith Turner >Priority: Major > > Does the thread context classloader for threads executing scans need to be > set? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3740) Investigate split/assignment threadpool shutdown
[ https://issues.apache.org/jira/browse/ACCUMULO-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3740. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Investigate split/assignment threadpool shutdown > > > Key: ACCUMULO-3740 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3740 > Project: Accumulo > Issue Type: Task >Reporter: Christopher Tubbs >Priority: Major > > Commit a194c0b00b7a927e938da72350916f30b0a0707d for ACCUMULO-3204 removed > some unused code in the TabletServerResourceManager. > It is curious why this code was unused, because it seemed like it may have > needed to be used. > The methods were named: > * stopSplits > * stopNormalAssignments > * stopMetadataAssignments > The underlying thread pools these methods were supposed to shut down do not > appear to be stopped anywhere else. > We should investigate whether it is a problem that these thread pools are > never shut down, and if it is, reintroduce the deleted code, with additional > code to appropriately call it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1693) Add the ability to determine Table states in Accumulo Shell
[ https://issues.apache.org/jira/browse/ACCUMULO-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1693. - Resolution: Duplicate Closing as duplicate of https://github.com/apache/accumulo/pull/1735 > Add the ability to determine Table states in Accumulo Shell > --- > > Key: ACCUMULO-1693 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1693 > Project: Accumulo > Issue Type: Improvement >Reporter: Aaron Glahe >Priority: Minor > > Currently, there is no way to retrieve the various Table states (eg. Offline, > "going offline", exporting, cloning, etc) from the Shell. May be helpful > when trying to determine what actions can be performed on a table based upon > its current state. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1266) Automatically determine when a full major compaction would benefit scans
[
https://issues.apache.org/jira/browse/ACCUMULO-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-1266.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Automatically determine when a full major compaction would benefit scans
>
>
> Key: ACCUMULO-1266
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1266
> Project: Accumulo
> Issue Type: New Feature
>Reporter: Keith Turner
>Priority: Major
>
> For the following situation, there is a tipping point where it becomes
> beneficial to do a full major compaction.
> * a tablet is frequently scanned
> * scan time iterators supress a lot of data
> * a full major compaction would also supress that data
> Examples of this are tablets with lots of deletes, versions that are
> suppressed, data thats combined, and data thats filtered.
> If tablet servers kept track of statistics about scans, could this be used to
> determine when its beneficial to automatically compact? In the following
> simple example, it seems obvious that a major compaction would be beneficial.
> In this example scans over the last hour have had to examine and throw away
> 20 million uneeded keys. Alot of scan work could have been saved by doing a
> major compaction.
> * all scans over tabletA within the last hour have read 30 million keys and
> returned 10 million keys
> * TabletA has 3 million keys
> * a major compaction would reduce tabletA to 1 million keys and result in
> future scans returning all keys read.
> One complicating factor is that major compaction may have a different set of
> iterators configured. Therefore its possible that scan may filter a lot of
> data, and major compactions may not. Could possibly keep track of ratio of
> data dropped by compactions and the ratio of data dropped by scans. This
> could be used when deciding if a major compaction should be done to improve
> scan performance.
> What other situation can cause unnecessary major compactions and need to be
> defended against?
> In the case where a compaction of just the data in memory would benefit
> scans, ACCUMULO-519 may solve the problem that this ticket is looking to
> solve.
> So what should the formula be?
> {code:java}
> // k/v : key values
> // recentlyRead: total number of k/v read before applying iterators by
> recent scans (recentlyRead - recentlyDropped equals # of k/v returned to
> users)
> // majcDropRatio : ratio of k/v dropped by recent major compactions
> // totalKeyValues : total # of k/v in tablet
> // R a user configurable ratio, like the current major compaction ratio
> that is based on files
> if((recentlyRead * majcDropRatio > R * totalKeyValues)){
> doFullMajorCompaction()
> resetScanStats()
> }
> {code}
> The example formula above has an issue, it may initiate a major compaction
> when scans are not reading a part of the tablet that drops data. The formula
> below tries to remedy this.
> {code:java}
> // k/v : key values
> // recentlyDropped : number of k/v dropped by recent scans
> // recentlyRead: total number of k/v read before applying iterators by
> recent scans (recentlyRead - recentlyDropped equals # of k/v returned to
> users)
> // majcDropRatio : ratio of k/v dropped by recent major compactions
> // totalKeyValues : total # of k/v in tablet
> // R a user configurable ratio, like the current major compaction ratio
> that is based on files
> if((recentlyDropped > R * totalKeyValues) && (recentlyRead * majcDropRatio
> > R * totalKeyValues)){
> doFullMajorCompaction()
> resetScanStats()
> }
> {code}
> An issue with the above is that the total # of key values for a tablet may
> not be accurate because of bulk import and splits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2185) determine how rfile block location impacts performance
[ https://issues.apache.org/jira/browse/ACCUMULO-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2185. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > determine how rfile block location impacts performance > -- > > Key: ACCUMULO-2185 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2185 > Project: Accumulo > Issue Type: Task > Components: tserver >Reporter: Arshak Navruzyan >Priority: Major > > We should investigate and clearly spell out in the documentation: > 1. Location of RFile HDFS blocks vs. tserver assignments (is it true that > "compaction leading to eventual locality" and if not, why this isn't > important) > 2. Recommended setting HDFS balancer > 3. Recommended setting for HDFS short circuit reads > Good (but somewhat inconclusive) discussion here: > http://mail-archives.apache.org/mod_mbox/accumulo-user/201401.mbox/%3C13A9718A-F829-4DFE-ACD6-C5195FF4C253%40clearedgeit.com%3E -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3056) Determine what desired repeated start/stop calls are on MAC
[ https://issues.apache.org/jira/browse/ACCUMULO-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3056. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Determine what desired repeated start/stop calls are on MAC > --- > > Key: ACCUMULO-3056 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3056 > Project: Accumulo > Issue Type: Bug > Components: mini >Affects Versions: 1.5.1, 1.6.0 >Reporter: Josh Elser >Priority: Major > > In 1.5, multiple calls to MiniAccumuloCluster.start() result in an > IllegalStateException. > In 1.6, while still advertised, the Exception was removed. > In both versions, multiple calls to stop() pass gracefully. > What exactly do we want: exceptions when trying to re-enter the state which > we are already in, or not? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1453) Track tablet migrations and failed loads
[ https://issues.apache.org/jira/browse/ACCUMULO-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1453. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Track tablet migrations and failed loads > > > Key: ACCUMULO-1453 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1453 > Project: Accumulo > Issue Type: Improvement > Components: master, tserver >Affects Versions: 1.4.3, 1.5.0 >Reporter: Mike Drob >Priority: Major > > If a bad RFile or Tablet somehow gets in the system and brings down a > tserver, then as the master migrates it to other servers it will likely cause > cascading failures. > It might be a good idea to keep track of how many consecutive failures to > load there are for a given tablet, and either warn or refuse to host the > tablet if this value exceeds a given threshold. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3590) Rolling upgrade support
[ https://issues.apache.org/jira/browse/ACCUMULO-3590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3590. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Rolling upgrade support > --- > > Key: ACCUMULO-3590 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3590 > Project: Accumulo > Issue Type: Improvement >Reporter: Josh Elser >Priority: Major > > We really lack any support for rolling upgrades between versions of Accumulo. > Need to have a way to provide seamless upgrades between certain versions of > Accumulo which minimize any downtime/impact on users of the system. > Need to cover fundamental control features (like a rolling upgrade), changes > to our serialized data structures (help prevent us from goofing up > serialization across versions -- thrift/protobuf/etc), backwards/forward > compatible RFile and WAL support (including control), lots of tests, and lots > of documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3591) Update FATE serialization to use a serialization library
[ https://issues.apache.org/jira/browse/ACCUMULO-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3591. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Update FATE serialization to use a serialization library > > > Key: ACCUMULO-3591 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3591 > Project: Accumulo > Issue Type: Sub-task > Components: fate >Reporter: Josh Elser >Priority: Major > > FATE presently relies on Java serialization to store things in the TStore > (ZooKeeper). We should swap out that approach for something a bit more robust > to drifting schemas like protobuf or thrift. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3595) Clean up ZooKeeper serialized data structures
[ https://issues.apache.org/jira/browse/ACCUMULO-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3595. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Clean up ZooKeeper serialized data structures > - > > Key: ACCUMULO-3595 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3595 > Project: Accumulo > Issue Type: Sub-task > Components: zookeeper >Reporter: Josh Elser >Priority: Major > > We have a lot of different kinds of data in ZooKeeper, we have a lot of > different ways of getting at that data, and we have lots of way of parsing > that data. > Try to centralize access to data stored in ZK and the means of > (de)serializing it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1454) Need good way to perform a rolling restart of all tablet servers
[ https://issues.apache.org/jira/browse/ACCUMULO-1454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1454. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Need good way to perform a rolling restart of all tablet servers > > > Key: ACCUMULO-1454 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1454 > Project: Accumulo > Issue Type: Sub-task > Components: tserver >Affects Versions: 1.4.3, 1.5.0 >Reporter: Mike Drob >Priority: Major > Attachments: ACCUMULO-1454-proposal-01.adoc, > ACCUMULO-1454-proposal-01.html > > > When needing to change a tserver parameter (e.g. java heap space) across the > entire cluster, there is not a graceful way to perform a rolling restart. > The naive approach of just killing tservers one at a time causes a lot of > churn on the cluster as tablets move around and zookeeper tries to maintain > current state. > Potential solutions might be via a fancy fate operation, with coordination by > the master. Ideally, the master would know which servers are 'safe' to > restart and could minimize overall impact during the operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3598) Address varied file versions by servers over RPC layer
[ https://issues.apache.org/jira/browse/ACCUMULO-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3598. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Address varied file versions by servers over RPC layer > -- > > Key: ACCUMULO-3598 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3598 > Project: Accumulo > Issue Type: Sub-task > Components: tserver >Reporter: Josh Elser >Priority: Major > > There's an issue of handling newer versions of RFile and WALs in the middle > of a rolling restart. > 1. Server1 is restarted as the new version > 2. Server1 writes some new data > 3. Server1 dies > 4. Server2 (still old version) gets the tablets from Server1 > We need to ensure that there is control to limit the new software from > writing out new versions of persistent files while there are still old > versions of the software participating in the instance. It's similar to > finalizing an upgrade: after we're sure that all of the servers have been > upgraded and are functioning well, we can flip them over to using new > messages/serialization that the old versions aren't aware of. > This problem gets much easier after we adopt Thrift/PB for serializing things > because both of those can naturally read newer versions of messages they know > about, ignoring the new fields. > Ideally, we should define an API which rolling restart (ACCUMULO-1454) can > leverage, but there are many ways we could go about the "feature". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2924) Begin replication before WAL is marked as "closed"
[ https://issues.apache.org/jira/browse/ACCUMULO-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2924. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Begin replication before WAL is marked as "closed" > -- > > Key: ACCUMULO-2924 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2924 > Project: Accumulo > Issue Type: Improvement > Components: replication >Reporter: Josh Elser >Priority: Minor > > Presently, we wait for a WAL to be closed before we start replicating the > data contained within. > In an "active" system (one that is continually ingesting a steady stream of > data), you don't really notice this lag. In a "fresh" system, there is a > noticeable wait before you start to get any replication flowing. This tends > to be a pain when you want to verify the correct configuration before walking > away for a coffee. > I don't believe there is anything that would break from removing this check, > but it will require a little testing to make sure the edge cases are solid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2580) Modify "bulk ingest" code path to create replication entries
[ https://issues.apache.org/jira/browse/ACCUMULO-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2580. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Modify "bulk ingest" code path to create replication entries > > > Key: ACCUMULO-2580 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2580 > Project: Accumulo > Issue Type: New Feature > Components: replication >Reporter: Josh Elser >Priority: Major > > When a file import is requested, we also need to create a key to replicate so > we know to replicate this file and not allow the GC to delete it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4156) Tunable replication frequency
[ https://issues.apache.org/jira/browse/ACCUMULO-4156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4156. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Tunable replication frequency > - > > Key: ACCUMULO-4156 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4156 > Project: Accumulo > Issue Type: Improvement > Components: core, replication >Affects Versions: 1.7.1 >Reporter: William Slacum >Priority: Major > > Currently, replication happens when a write ahead log file is closed. The > only parameter to toggle when this event occurs is write ahead log size, and > is only applicable to the tablet servers themselves. > By default this means that when replication happens isn't tied to the table > it is configured on, but also exogenous factors such as total write load and > failures. If a system receives ~100MB/day/TServer, and the WAL size is its > default 1GB, it will take 10 days for any replication event to occur. Another > possibility is that an unreplicated table is receiving many writes, which > will cause more frequent replication events, but proportionally the work will > involve less data for the table being replicated. > I don't have a specific implementation in mind, but I'd like to see a > solution that involves isolating the work down to specific table events such > as time-since-last-replication and data-added-since-last-replication. > [~elserj] has had some ideas about doing things incrementally within WAL > files (ie, replicating between two sync points) that can also help with this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3232) Improve consumption of WAL header in partial replication case
[
https://issues.apache.org/jira/browse/ACCUMULO-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3232.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Improve consumption of WAL header in partial replication case
> -
>
> Key: ACCUMULO-3232
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3232
> Project: Accumulo
> Issue Type: Improvement
> Components: replication
>Reporter: Josh Elser
>Priority: Major
>
> Consider a system that is actively replicating from one instance to another.
> Specifically, assume there is one WAL that is currently being replicated to
> the destination and the source instance is shutdown.
> When the source instance is restarted, it will notice that the WAL has read
> through N {{LogFileKey}}/{{LogFileValue}} pairs (from before it was shutdown)
> and while proceed past these records to get to the data in the file which it
> needs to read.
> We have to re-read each of these pairs from the file because the WAL is an
> append-only structure, and we can't efficiently seek to some point in the
> file, as we wouldn't know how to correlate the byte offset to entries.
> As we read the WAL, in addition (or perhaps instead of) tracking the offset
> in the WAL, it would be good to track the correlation of N bytes read to M
> records consumed which would help us better resume replication.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2860) Support ConditionalMutations with Replication
[ https://issues.apache.org/jira/browse/ACCUMULO-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2860. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Support ConditionalMutations with Replication > - > > Key: ACCUMULO-2860 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2860 > Project: Accumulo > Issue Type: Wish > Components: replication >Reporter: Josh Elser >Priority: Major > > ConditionalMutations are presently unsupported with a table that is being > replicated. It would be nice if we could figure out a way to make them work, > although I have absolutely no idea what such a solution would look like. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2990) BatchWriter never recovers from failure(s)
[
https://issues.apache.org/jira/browse/ACCUMULO-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2990.
-
Resolution: Cannot Reproduce
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> BatchWriter never recovers from failure(s)
> --
>
> Key: ACCUMULO-2990
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2990
> Project: Accumulo
> Issue Type: Bug
> Components: client
>Affects Versions: 1.5.1, 1.6.0
>Reporter: Josh Elser
>Priority: Critical
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In trying to understand what's happening in ACCUMULO-2964, I noticed that I
> had similar exceptions from two different threads. One of the threads
> starting working after the unexplained thrift exceptions from a tserver
> restart, and the other continued to repeatedly fail for the lifetime of the
> test.
> I repeatedly saw this exception:
> {noformat}
> 2014-07-11 04:14:41,591 [replication.WorkMaker] WARN : Failed to write work
> mutations for replication, will retry
> org.apache.accumulo.core.client.MutationsRejectedException: # constraint
> violations : 0 security codes:
> {accumulo.metadata(ID:!0)=[DEFAULT_SECURITY_ERROR]} # server errors 0 #
> exceptions 0
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.checkForFailures(TabletServerBatchWriter.java:537)
> at
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter.addMutation(TabletServerBatchWriter.java:249)
> at
> org.apache.accumulo.core.client.impl.BatchWriterImpl.addMutation(BatchWriterImpl.java:45)
> at
> org.apache.accumulo.master.replication.WorkMaker.addWorkRecord(WorkMaker.java:184)
> at
> org.apache.accumulo.master.replication.WorkMaker.run(WorkMaker.java:124)
> at
> org.apache.accumulo.master.replication.ReplicationDriver.run(ReplicationDriver.java:91)
> {noformat}
> The part that struck me as odd was that the BatchWriter wasn't against the
> metadata table, but the replication table.
> I looked into the TabletServerBatchWriter. It appears that once the client
> sees a MutationsRejectedException, that BatchWriter becomes useless as the
> internal member {{somethingFailed}} is never reset back to {{false}} after
> the failure is reported. Same goes for {{serverSideErrors}},
> {{unknownErrors}}, {{lastUnknownErrors}}, too.
> If this is the case, this is a bug because the BatchWriter should be
> resilient in this regard and not force the client to create a new Instance.
> If that's infeasible to do, we should add exceptions to the BatchWriter that
> fail fast when a BatchWriter is used that will report repeatedly report the
> same failure over and over again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2937) Cache columnvisibilities in replication pipeline
[ https://issues.apache.org/jira/browse/ACCUMULO-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2937. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Cache columnvisibilities in replication pipeline > > > Key: ACCUMULO-2937 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2937 > Project: Accumulo > Issue Type: Improvement > Components: replication >Reporter: Josh Elser >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > In the BatchWriterReplicationReplayer, we have to re-create the Mutation > objects (see ACCUMULO-2925 for more details). > As such, we have to re-parse the bytes for the cv into a ColumnVisibility > object for Mutation. Since those ColumnVisibilities are immutable, and > expensive to create, we should cache them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3949) getFileStatus called too often?
[ https://issues.apache.org/jira/browse/ACCUMULO-3949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3949. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > getFileStatus called too often? > --- > > Key: ACCUMULO-3949 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3949 > Project: Accumulo > Issue Type: Task > Components: tserver > Environment: very large cluster >Reporter: Eric C. Newton >Assignee: Eric C. Newton >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > In a recent analysis of a large accumulo installation, it was found that > getFileStatus represented 40% of the calls to the individual NNs. > This is surprising. Investigate the calls to getFileStatus and see if they > can be reduced. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (ACCUMULO-4316) Move website social buttons to footer
[ https://issues.apache.org/jira/browse/ACCUMULO-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller closed ACCUMULO-4316. > Move website social buttons to footer > - > > Key: ACCUMULO-4316 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4316 > Project: Accumulo > Issue Type: Task > Components: website >Reporter: Mike Walch >Priority: Minor > > From the review of ACCUMULO-4313, [~ctubbsii] mentioned the following... > "I think the left-side social buttons shouldn't appear on the first page > either. Instead, it should be moved to the footer. Or, at the very least, > they should be made smaller without the large text next to them (hover text > is sufficient)." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4316) Move website social buttons to footer
[ https://issues.apache.org/jira/browse/ACCUMULO-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-4316. -- Resolution: Fixed There are no social buttons on the left-side. The ones on the front page look fine. > Move website social buttons to footer > - > > Key: ACCUMULO-4316 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4316 > Project: Accumulo > Issue Type: Task > Components: website >Reporter: Mike Walch >Priority: Minor > > From the review of ACCUMULO-4313, [~ctubbsii] mentioned the following... > "I think the left-side social buttons shouldn't appear on the first page > either. Instead, it should be moved to the footer. Or, at the very least, > they should be made smaller without the large text next to them (hover text > is sufficient)." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3953) BulkIngest fetches the file size 2x, not once
[ https://issues.apache.org/jira/browse/ACCUMULO-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3953. - Resolution: Not A Problem If this is still relevant, it is likely only relevant to the old bulk import code, and not the new bulk import code in 2.x. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > BulkIngest fetches the file size 2x, not once > - > > Key: ACCUMULO-3953 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3953 > Project: Accumulo > Issue Type: Sub-task > Components: tserver >Reporter: Eric C. Newton >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > While investigating parent ticket, I found that a bulk import fetches the > size of file twice: once to compute the estimated size to send to importers, > and once to open the file and read the index. Ideally this would only be done > once. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3970) Generating multiple views of a value at scan time
[
https://issues.apache.org/jira/browse/ACCUMULO-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3970.
-
Resolution: Won't Fix
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
Also, this can be done by storing different representations, or scanning with a
proxy user who is authorized to view the data and manipulate it in an iterator
at scan time.
> Generating multiple views of a value at scan time
> -
>
> Key: ACCUMULO-3970
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3970
> Project: Accumulo
> Issue Type: New Feature
>Reporter: Russ Weeks
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> It would be useful to have the ability to generate different representations
> of a key-value pair at scan time, based on the scan authorizations.
> For example, consider [HIPPA safe harbour
> de-identification|http://www.hhs.gov/ocr/privacy/hipaa/understanding/coveredentities/De-identification/guidance.html#dates].
> One of the rules for de-identifying a patient's date of birth is that if a
> patient is 89 years old or younger, you can disclose his exact year of birth.
> If a patient is 90 years old or over, you pretend that he's 90 years old.
> You can imagine implementing this as a key/value mapping in accumulo like,
> {{(pt_id, demographic, pt_dob, PII_DOB) -> "1925-08-22"}}
> {{(pt_id, demographic, pt_dob, SHD_DOB) -> "1925"}}
> Where the value corresponding to visibility SHD_DOB is produced at scan-time,
> depending on the patient's current age.
> Another example would be the ability to produce a salted hash of a unique
> identifier like a social security number or medical record number, where the
> salt (or the hash algorithm, or the work factor...) could be specified
> dynamically without having to re-code all the values in the system.
> More broadly speaking, this feature would give organizations more flexibility
> to change how they deidentify, transform or anonymize data to suit different
> access levels.
> Of course, to do this you'd need to have a pluggable component that can
> process key/value pairs before visibilities are evaluated. I can see why this
> might give a lot of people the heeby-jeebies but I'd like to gather as much
> feedback as possible. Looking forward to hearing your thoughts!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4031) consistency check failure
[ https://issues.apache.org/jira/browse/ACCUMULO-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4031. - Resolution: Cannot Reproduce The code has changed a lot since this issue was created. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > consistency check failure > - > > Key: ACCUMULO-4031 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4031 > Project: Accumulo > Issue Type: Bug > Components: tserver >Affects Versions: 1.6.4 > Environment: Very large production cluster >Reporter: Eric C. Newton >Priority: Major > > Sorry for the lack of concrete details, but my logs are not online. > This system does a lot of bulk ingest. When it was shut down, a few tablets > complained about inconsistency of their file list with what was in the > metadata table. > I tracked one down, the others appear to be similar. > First, the inconsistency was an "extra" bulk import file in the metadata > table, which was missing from the in-memory list. > The file was attempted to bulk loaded into the tablet, but the bulk-load > failed. It failed due to a constraint violation: the bulk transaction was no > longer running. > Except, really, it was. The constraint fired during the update of the > tablets' metadata. The server of the metadata tablet was having a (brief) > connection problem with zookeeper, which is where the bulk transaction status > is stored. > The importing tablet server saw the constraint violation, and didn't add the > file to the in-memory list. However, 2 minutes later, the bulk import was > retried, and (consulting the metadata table), it claimed the file *was* > imported already. > So, in the intervening 2 minutes, somehow the update was made to the metadata > tablet. > * No splitting of either tablet occurred during this event. > * Neither tablet was moved during this event. > * No recovery of the metadata table took place. > * The tablet server never reported the file imported. > I reviewed the handling of constraints, and it looks correct (despite > ACCUMULO-4029). > With ACCUMULO-3327, the tablet server will not reject retries, because it > does not re-consult the metadata table. > I don't know how the mutations would be applied without the tablet server > reporting the file as loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4683) Create IT for monitor REST endpoints
[ https://issues.apache.org/jira/browse/ACCUMULO-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4683. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Create IT for monitor REST endpoints > > > Key: ACCUMULO-4683 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4683 > Project: Accumulo > Issue Type: Test > Components: monitor >Reporter: Christopher Tubbs >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > With the changes to the monitor in ACCUMULO-3005, it'd be nice to have an > integration test to verify the REST endpoints as API. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4034) agitate zookeepers
[ https://issues.apache.org/jira/browse/ACCUMULO-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4034. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > agitate zookeepers > -- > > Key: ACCUMULO-4034 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4034 > Project: Accumulo > Issue Type: Sub-task > Components: scripts >Reporter: Eric C. Newton >Assignee: Eric C. Newton >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The parent issue occurred during brief zookeeper disconnect. Model that in > continuous ingest. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2495) OOM exception didn't bring down tserver
[
https://issues.apache.org/jira/browse/ACCUMULO-2495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2495.
-
Resolution: Duplicate
Closing as presumed duplicate of https://github.com/apache/accumulo/issues/1689
If this is a separate issue, then please open a new issue or PR at
https://github.com/apache/accumulo
> OOM exception didn't bring down tserver
> ---
>
> Key: ACCUMULO-2495
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2495
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
>Affects Versions: 1.5.1
>Reporter: John Vines
>Priority: Major
> Attachments: Test.java
>
>
> Got
> {code}Thread "acu-problem-reporter 2" died Direct buffer memory
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:659)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:113)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
> at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:75)
> at sun.nio.ch.IOUtil.read(IOUtil.java:223)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> at
> java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at
> java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
> at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.readAll(ThriftTransportPool.java:271)
> at
> org.apache.thrift.protocol.TCompactProtocol.readByte(TCompactProtocol.java:601)
> at
> org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:470)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_update(TabletClientService.java:443)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.update(TabletClientService.java:427)
> at
> org.apache.accumulo.core.client.impl.Writer.updateServer(Writer.java:69)
> at
> org.apache.accumulo.core.client.impl.Writer.update(Writer.java:97)
> at
> org.apache.accumulo.server.problems.ProblemReport.saveToMetadataTable(ProblemReport.java:134)
> at
> org.apache.accumulo.server.problems.ProblemReports$1.run(ProblemReports.java:92)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:701){code}
> while hammering a single node setup with table creates and delete. First the
> master went down with an OOM after about an hour, which is strange since I
> gave it a gig and was only creating and dropping tables in 64 count chunks.
> When I brought the master back up, I saw that stack trace in the monitor, but
> nothing in the tserver logs.
> Initial logging was
> {code}
> 2014-03-18 16:59:43,977 [impl.TabletServerBatchWriter] ERROR: Failed to send
> tablet server 127.0.0.1:9997 its batch : Direct buffer memory
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:659)
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:113)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:305)
> at sun.nio.ch.Util.get
[GitHub] [accumulo] ctubbsii commented on issue #1689: Tserver in bad state may be writing corrupted files during compactions.
ctubbsii commented on issue #1689: URL: https://github.com/apache/accumulo/issues/1689#issuecomment-718248255 Another related JIRA issue: https://issues.apache.org/jira/browse/ACCUMULO-2495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-2485) HDFS permissions error leaves root_tablet in inconsistent state
[ https://issues.apache.org/jira/browse/ACCUMULO-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2485. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > HDFS permissions error leaves root_tablet in inconsistent state > --- > > Key: ACCUMULO-2485 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2485 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.4.5 > Environment: 1.4.5-SNAPSHOT on 6faac42, Hadoop 1 via CDH3u6 >Reporter: Sean Busbey >Priority: Major > Labels: 14_qa_bug > Attachments: ACCUMULO-2485-tserver.log > > > While setting up a test cluster, I mistakenly left out a home directory for > the Accumulo user on a cluster set up with HDFS Trash enabled. > After successfully writing out the new file for a MajC on the root tablet, > some part of the cleanup of old files failed when moving to Trash didn't > work. The result was that the tablet still listed one of the original files, > but that file was no longer in HDFS. > All future MajCs failed with file not found. Attaching representative log > sample. > I don't know if this impacts future versions. > Workaround: ACCUMULO-1219 (copied the result of the initial MajC to the files > named as its source) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-818) User Management API should support appending authorizations
[ https://issues.apache.org/jira/browse/ACCUMULO-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-818. Resolution: Won't Fix Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > User Management API should support appending authorizations > --- > > Key: ACCUMULO-818 > URL: https://issues.apache.org/jira/browse/ACCUMULO-818 > Project: Accumulo > Issue Type: Sub-task > Components: client, shell >Affects Versions: 1.4.1 >Reporter: Mike Drob >Priority: Major > Attachments: ACCUMULO-818-1.patch > > > It would be useful to have a functionality for appending authorizations to a > user instead of having to "get and set" and dealing with potential > concurrency issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3272) tserver breaks in a bad way when it can't write to hdfs trash
[
https://issues.apache.org/jira/browse/ACCUMULO-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3272.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> tserver breaks in a bad way when it can't write to hdfs trash
> -
>
> Key: ACCUMULO-3272
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3272
> Project: Accumulo
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Adam Fuchs
>Priority: Major
> Attachments: ProposedRootCompactionStateTransition.png
>
>
> When installing and testing a vanilla install of HDP 2.1 the HDFS setting for
> fs.trash.interval is set to 360 by default. Accumulo takes this to mean that
> it should move deleted files to the .Trash directory in accumulo's hdfs home
> directory. In this instance, the home directory did not exist, which caused a
> major compaction to fail. The failure happened in such a way that the
> internal state of the tserver became inconsistent, preventing automatic
> recovery after an admin solved the trash problem.
> The first stack trace below shows the initial problem. The second shows a
> secondary problem caused by the poor failure mode.
> {code}
> 2014-10-28 15:05:19,353 [tserver.Tablet] DEBUG: Major compaction plan:
> [hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/0_0.rf,
>
> hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/F03q.rf]
> propogate deletes : false
> 2014-10-28 15:05:19,353 [tserver.Tablet] DEBUG: MajC initiate lock 0.00 secs,
> wait 0.00 secs
> 2014-10-28 15:05:19,356 [tserver.Tablet] DEBUG: Starting MajC +r<< (NORMAL)
> [hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/F03q.rf,
>
> hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/0_0.rf]
> --> hdfs://n1.sqrrl-lab.
> net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/A03r.rf_tmp []
> 2014-10-28 15:05:19,376 [tserver.TabletServer] DEBUG: Got getScans message
> from user: !SYSTEM
> 2014-10-28 15:05:19,432 [tserver.Compactor] DEBUG: Compaction +r<< 28 read |
> 18 written |608 entries/sec | 0.046 secs
> 2014-10-28 15:05:19,482 [fs.TrashPolicyDefault] INFO : Namenode trash
> configuration: Deletion interval = 360 minutes, Emptier interval = 0 minutes.
> 2014-10-28 15:05:19,491 [fs.TrashPolicyDefault] WARN : Can't create trash
> directory:
> hdfs://n1.sqrrl-lab.net:8020/user/accumulo/.Trash/Current/accumulo_1.6_perf_test/tables/+r/root_tablet
> 2014-10-28 15:05:19,491 [tserver.Tablet] ERROR: MajC Failed, extent = +r<<
> 2014-10-28 15:05:19,491 [tserver.Tablet] ERROR: MajC Failed, message = Failed
> to move to trash:
> hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/delete+A03r.rf+F03q.rf
> java.io.IOException: Failed to move to trash:
> hdfs://n1.sqrrl-lab.net:8020/accumulo_1.6_perf_test/tables/+r/root_tablet/delete+A03r.rf+F03q.rf
> at
> org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:160)
> at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:109)
> at
> org.apache.accumulo.server.fs.VolumeManagerImpl.moveToTrash(VolumeManagerImpl.java:364)
> at
> org.apache.accumulo.tserver.RootFiles.finishReplacement(RootFiles.java:64)
> at
> org.apache.accumulo.tserver.RootFiles.replaceFiles(RootFiles.java:75)
> at
> org.apache.accumulo.tserver.Tablet$DatafileManager.bringMajorCompactionOnline(Tablet.java:1001)
> at org.apache.accumulo.tserver.Tablet._majorCompact(Tablet.java:3239)
> at org.apache.accumulo.tserver.Tablet.majorCompact(Tablet.java:3340)
> at org.apache.accumulo.tserver.Tablet.access$4800(Tablet.java:172)
> at
> org.apache.accumulo.tserver.Tablet$CompactionRunner.run(Tablet.java:2804)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:42)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at
> org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:42)
> at
> org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
> at java.lang.Thread.run(Thread.java:744)
> Caused by: org.apache.hadoop.security.AccessControlException: Permission
> denied: user=accumulo, access=WRITE, inode="/user":hdfs:hdfs:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)
> at
> org.apache.hadoop.hdfs.server.nam
[jira] [Resolved] (ACCUMULO-3067) scan performance degrades after compaction
[
https://issues.apache.org/jira/browse/ACCUMULO-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3067.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> scan performance degrades after compaction
> --
>
> Key: ACCUMULO-3067
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3067
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Environment: Macbook Pro 2.6 GHz Intel Core i7, 16GB RAM, SSD, OSX
> 10.9.4, single tablet server process, single client process
>Reporter: Adam Fuchs
>Priority: Major
> Attachments: Screen Shot 2014-08-19 at 4.19.37 PM.png,
> accumulo_query_perf_test.tar.gz, jit_log_during_compaction.txt
>
>
> I've been running some scan performance tests on 1.6.0, and I'm running into
> an interesting situation in which query performance starts at a certain level
> and then degrades by ~15% after an event. The test follows roughly the
> following scenario:
> # Single tabletserver instance
> # Load 100M small (~10byte) key/values into a tablet and let it finish major
> compacting
> # Disable the garbage collector (this makes the time to _the event_ longer)
> # Restart the tabletserver
> # Repeatedly scan from the beginning to the end of the table in a loop
> # Something happens on the tablet server, like one of {idle compaction of
> metadata table, forced flush of metadata table, forced compaction of metadata
> table, forced flush of trace table}
> # Observe that scan rates dropped by 15-20%
> # Observe that restarting the scan will not improve performance back to
> original level. Performance only gets better upon restarting the tablet
> server.
> I've been able to get this not to happen by removing iterators from the
> iterator tree. It doesn't seem to matter which iterators, but removing a
> certain number both improves performance (significantly) and eliminates the
> degradation problem. The default iterator tree includes:
> * SourceSwitchingIterator
> ** VersioningIterator
> *** SynchronizedIterator
> VisibilityFilter
> * ColumnQualifierFilter
> ** ColumnFamilySkippingIterator
> *** DeletingIterator
> StatsIterator
> * MultiIterator
> ** MemoryIterator
> ** ProblemReportingIterator
> *** HeapIterator
> RFile.LocalityGroupReader
> We can eliminate the weird condition by narrowing the set of iterators to:
> * SourceSwitchingIterator
> ** VisibilityFilter
> *** ColumnFamilySkippingIterator
> DeletingIterator
> * StatsIterator
> ** MultiIterator
> *** MemoryIterator
> *** ProblemReportingIterator
> HeapIterator
> * RFile.LocalityGroupReader
> There are other combinations that also perform much better than the default.
> I haven't been able to isolate this problem to a single iterator, despite
> removing each iterator one at a time.
> Anybody know what might be happening here? Best theory so far: the JVM learns
> that iterators can be used in a different way after a compaction, and some
> JVM optimization like JIT compilation, branch prediction, or automatic
> inlining stops happening.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [accumulo] ctubbsii commented on a change in pull request #1757: Create Ample interface method for createDeleteMutation
ctubbsii commented on a change in pull request #1757:
URL: https://github.com/apache/accumulo/pull/1757#discussion_r513791394
##
File path:
server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##
@@ -283,17 +283,18 @@ public static void finishSplit(KeyExtent extent,
* datafilesToDelete are strings because they can be a TabletFile or
directory
*/
public static void addDeleteEntries(KeyExtent extent, Set
datafilesToDelete,
- ServerContext context) {
+ ServerContext context, Ample ample) {
Review comment:
Can't you just create a local object before that loop?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-809) display Fate status in the monitor
[ https://issues.apache.org/jira/browse/ACCUMULO-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-809. - Resolution: Duplicate https://github.com/apache/accumulo/issues/1249 > display Fate status in the monitor > -- > > Key: ACCUMULO-809 > URL: https://issues.apache.org/jira/browse/ACCUMULO-809 > Project: Accumulo > Issue Type: New Feature > Components: master, monitor >Reporter: Eric C. Newton >Priority: Major > Labels: gsoc2013, mentor > > The Fate Admin class can print a crude status for the running Fate > operations. It would be nice to clean this information up and display it in > the monitor. > For example: > |Requester|Request|Table Locks|Executing|Details > |192.168.1.1|BulkImport|Index(R)|LoadFiles|Directory > /accumulo/imports/import-2010-09-01| > |192.168.1.2|RangeOp|Documents (W, waiting)| |Merging Range (200809, 200810]| > |192.168.1.1|BulkImport| |Successful|Directory > /accumulo/imports/import-2010-08-31| > | *Clear Successful Entries* | -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4048) Regexp in org.apache.accumulo.core.conf.PropertyType.isValidFormat() eats too much CPU
[ https://issues.apache.org/jira/browse/ACCUMULO-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-4048. - Resolution: Not A Problem The frequency of calls to validate configuration has changed in recent code. This is probably no longer relevant. Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Regexp in org.apache.accumulo.core.conf.PropertyType.isValidFormat() eats too > much CPU > -- > > Key: ACCUMULO-4048 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4048 > Project: Accumulo > Issue Type: Bug > Components: tserver >Affects Versions: 1.7.0 >Reporter: Volth >Priority: Major > Attachments: yUf0KA4.png > > > org.apache.accumulo.core.conf.PropertyType.isValidFormat() is called too many > times so the regexp eats too much CPU: http://i.imgur.com/yUf0KA4.png -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [accumulo] milleruntime commented on a change in pull request #1757: Create Ample interface method for createDeleteMutation
milleruntime commented on a change in pull request #1757:
URL: https://github.com/apache/accumulo/pull/1757#discussion_r513790531
##
File path:
server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##
@@ -283,17 +283,18 @@ public static void finishSplit(KeyExtent extent,
* datafilesToDelete are strings because they can be a TabletFile or
directory
*/
public static void addDeleteEntries(KeyExtent extent, Set
datafilesToDelete,
- ServerContext context) {
+ ServerContext context, Ample ample) {
Review comment:
Yes but each call to `context.getAmple()` creates a new object. I added
it as a param to prevent creating objects in the loop over `datafilesToDelete`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-4625) Monitor gets slow with lots of migrations
[ https://issues.apache.org/jira/browse/ACCUMULO-4625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-4625. -- Resolution: Fixed This should be fixed with the new monitor in 2.x > Monitor gets slow with lots of migrations > - > > Key: ACCUMULO-4625 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4625 > Project: Accumulo > Issue Type: Bug > Components: master, monitor >Affects Versions: 1.6.6 >Reporter: Michael Wall >Priority: Major > > An anecdotal ticket until I can dig further. The monitor gets really slow > when there is a large number of migrations that need to happen. For example, > create a table. Then add 1000 splits. All tablets are assigned to one > tserver until they are migrated. May even queue up needed migrations from > other tables behind those 1000 migrations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1463) Server components don't use standard command-line options parsing
[ https://issues.apache.org/jira/browse/ACCUMULO-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-1463. -- Resolution: Fixed We should be using jcommander for all command line parsing now. If not, an issue or PR specific to that part of the code can be opened in GitHub. > Server components don't use standard command-line options parsing > - > > Key: ACCUMULO-1463 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1463 > Project: Accumulo > Issue Type: Improvement > Components: gc, master, monitor, tserver >Reporter: Christopher Tubbs >Priority: Major > > Master/Monitor/TServer/etc. use custom command-line parsing. They should use > jcommander like the utilities. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1708) Error during minor compaction left tserver in bad state
[ https://issues.apache.org/jira/browse/ACCUMULO-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1708. - Resolution: Duplicate Closing as duplicate of what appears to be the same issue reported more recently on GitHub: https://github.com/apache/accumulo/issues/1689 > Error during minor compaction left tserver in bad state > --- > > Key: ACCUMULO-1708 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1708 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.4.0 >Reporter: Keith Turner >Priority: Critical > Attachments: ThreadTest.java > > > A tserver experienced a OOME during minor compaction. This OOME was thrown > because java could not create a native thread. Minor compactions only catch > declared exceptions and RuntimeExceptions. This left the system in a state > where the compaction was not running but the tserver thought it was. This > cause"flush -w" to hang and prevented the tserver from reclaiming memory. > For whatever reason the OOME handler that kills the process did not kick in > (seems it only kicks in w/ OOME related to heap allocation). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [accumulo] ctubbsii commented on issue #1689: Tserver in bad state may be writing corrupted files during compactions.
ctubbsii commented on issue #1689: URL: https://github.com/apache/accumulo/issues/1689#issuecomment-718234478 I ran into this old JIRA issue that seems to be the same issue: https://issues.apache.org/jira/browse/ACCUMULO-1708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-4307) semver compliance maven plugin
[
https://issues.apache.org/jira/browse/ACCUMULO-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-4307.
-
Resolution: Won't Fix
> semver compliance maven plugin
> --
>
> Key: ACCUMULO-4307
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4307
> Project: Accumulo
> Issue Type: New Feature
> Components: build
>Reporter: Dave Marion
>Assignee: Christopher Tubbs
>Priority: Major
> Attachments: ACCUMULO-4307-1.patch
>
>
> Found https://siom79.github.io/japicmp/MavenPlugin.html today and tested it
> out. Was wondering what thoughts are for incorporating this into the build. I
> tested it on 1.6.6-SNAPSHOT by dropping the following into the test pom file:
> {noformat}
>
> semver-compliance
>
>
>
> com.github.siom79.japicmp
> japicmp-maven-plugin
> 0.7.2
>
>
> client compliance
>
> cmp
>
> verify
>
>
>
> org.apache.accumulo
> accumulo-core
> 1.6.5
> jar
>
>
>
>
> org.apache.accumulo
> accumulo-core
> ${project.version}
> jar
>
>
>
>
> org.apache.accumulo.core.client
> org.apache.accumulo.core.data
> org.apache.accumulo.core.security
>
>
> *crypto*
> *impl*
> *thrift*
>
> protected
>
> breakBuildBasedOnSemanticVersioning>true
> true
>
>
>
>
>
>
>
>
> {noformat}
> I tried getting the previous release version number using the
> build-helper-maven-plugin, but it found the wrong version. If we use this we
> would also have to include an execution for minicluster and determine whether
> or not we want to use the reporting feature of the plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [accumulo] ctubbsii commented on a change in pull request #1757: Create Ample interface method for createDeleteMutation
ctubbsii commented on a change in pull request #1757:
URL: https://github.com/apache/accumulo/pull/1757#discussion_r513765003
##
File path:
server/base/src/main/java/org/apache/accumulo/server/util/MetadataTableUtil.java
##
@@ -283,17 +283,18 @@ public static void finishSplit(KeyExtent extent,
* datafilesToDelete are strings because they can be a TabletFile or
directory
*/
public static void addDeleteEntries(KeyExtent extent, Set
datafilesToDelete,
- ServerContext context) {
+ ServerContext context, Ample ample) {
Review comment:
Is ample not available off of ServerContext? Not sure we need to pass in
another param here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (ACCUMULO-3722) DFSInputStream being closed multiple times
[
https://issues.apache.org/jira/browse/ACCUMULO-3722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3722.
-
Resolution: Cannot Reproduce
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> DFSInputStream being closed multiple times
> --
>
> Key: ACCUMULO-3722
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3722
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
>Reporter: Josh Elser
>Priority: Minor
> Attachments:
> 0001-ACCUMULO-3722-Synchronize-check-for-closed-BlockRead.patch
>
>
> Running on top of Hadoop 2.7.0-SNAPSHOT (or a version with HDFS-7494
> applied), will result in lots of warnings from DFSInputStream in the
> TabletServer.
> {noformat}
> 2015-04-10 13:03:21,220 [hdfs.DFSClient] WARN : DFSInputStream has been
> closed already
> {noformat}
> We should figure out why we're closing the input stream multiple times and
> stop doing that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-652) support block-based filtering within RFile
[ https://issues.apache.org/jira/browse/ACCUMULO-652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-652. Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > support block-based filtering within RFile > -- > > Key: ACCUMULO-652 > URL: https://issues.apache.org/jira/browse/ACCUMULO-652 > Project: Accumulo > Issue Type: Improvement > Components: tserver >Reporter: Adam Fuchs >Assignee: Adam Fuchs >Priority: Major > Attachments: ACCUMULO-652-patches.tar.gz > > Time Spent: 1h 40m > Remaining Estimate: 0h > > If we keep some stats about what is in an RFile block, we might be able to > efficiently [O(log N)], with high probability, implement filters that > currently require linear table scans. Two use cases of this include timestamp > range filtering (i.e. give me everything from last Tuesday) and cell-level > security filtering (i.e. give me everything that I can see with my > authorizations). > For the timestamp range filter, we can keep minimum and maximum timestamps > across all keys used in a block within the index entry for that block. For > the cell-level security filter, we can keep an aggregate label. This could be > done using a simplified disjunction of all of the labels in the block. The > extra block statistics information can propagate up the index hierarchy as > well, giving nice performance characteristics for finding the next matching > entry in a file. > In general, this is a heuristic technique that is good if data tends to > naturally cluster in blocks with respect to the way it is queried. Testing > its efficacy will require closely emulating real-world use cases -- tests > like the continuous ingest test will not be sufficient. We will have to test > for a few things: > # The cost for storing the extra stats in the index are not too expensive. > # The performance benefit for common use cases is significant. > # We shouldn't introduce any unacceptable worst-case behavior, like bloating > the index to ridiculous proportions for any data set. > Eventually this will all need to be exposed through the Iterator API to be > useful, which will be another ticket. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1188) Sandbox iterators
[ https://issues.apache.org/jira/browse/ACCUMULO-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1188. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Sandbox iterators > - > > Key: ACCUMULO-1188 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1188 > Project: Accumulo > Issue Type: Bug >Reporter: Keith Turner >Priority: Major > Attachments: ACCUMULO-1188_fig1.png > > > It's possible that a user iterator can bring down a tablet server. For > example if it has an OOM or creates too many threads. It would be nice if > iterators could be sandboxed in some way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2699) Conditional writer lacks trace level logging
[ https://issues.apache.org/jira/browse/ACCUMULO-2699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2699. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Conditional writer lacks trace level logging > > > Key: ACCUMULO-2699 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2699 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.6.0 >Reporter: Keith Turner >Priority: Major > > The conditional writer code lacks trace logging about what its doing. Other > client code like BatchWriter, BatchScanner, and Scanner have this. > It should like events like : > * sending X to tserver Y > * requeued X failed mutations > These are not per mutations, but identify what its doing w/ batches of > mutations. > Noticed this while debugging ACCUMULO-2695 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3717) Add trace instrumentation around recovery
[ https://issues.apache.org/jira/browse/ACCUMULO-3717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3717. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Add trace instrumentation around recovery > - > > Key: ACCUMULO-3717 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3717 > Project: Accumulo > Issue Type: Improvement > Components: trace, tserver >Reporter: Josh Elser >Priority: Major > > Noticed this when looking into some tracing things with Billie: it doesn't > appear that we have recovery instrumented with tracing. > It would be nice to know what the long pole in the tent is for recovery since > it typically represents a period of unavailability of some data for users. We > should be aware of why it takes as long as it does and try to reduce it as > much as possible. > Because spans are delivered via ZK, I *think* it will be ok if we're > performing recovery on a WAL which contains updates for the trace table. As > long as the serialization to the trace table doesn't cause problems (it > should just create back-pressure in the tracer, but not throw exceptions), I > think it should be fine. Some testing would be needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2644) Improve tracing event names and information in writes
[
https://issues.apache.org/jira/browse/ACCUMULO-2644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-2644.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Improve tracing event names and information in writes
> -
>
> Key: ACCUMULO-2644
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2644
> Project: Accumulo
> Issue Type: Improvement
> Components: trace
>Affects Versions: 1.5.1
>Reporter: John Vines
>Assignee: John Vines
>Priority: Major
>
> There is some ambiguity in the trace names for some events, which can lead to
> confusion. We should rename trace events to make sure there isn't collisions.
> This is specific to the trace path for writes, which could also use some more
> information about quantity of data.
> {code} 3457+0 shell@john-P15xEMx shell:root
> 9+1603 shell@john-P15xEMx close
> 9+1603 shell@john-P15xEMx
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter 1
> 8+1604 shell@john-P15xEMx
> org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter 1
> 8+1604 shell@john-P15xEMx sendMutations
> 7+1605 tserver@localhost update
> 7+1605 tserver@localhost wal
> 4+1607 tserver@localhost update
> 4+1607 tserver@localhost wal
> 1+1608 tserver@localhost client:update
> 1+1609 tserver@localhost update
> 1+1609 tserver@localhost wal{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3270) TabletServerBatchReader needs a better error message to accompany it's stack trace
[ https://issues.apache.org/jira/browse/ACCUMULO-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3270. - Resolution: Not A Problem Finalizers were removed in the newest code base. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > TabletServerBatchReader needs a better error message to accompany it's stack > trace > -- > > Key: ACCUMULO-3270 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3270 > Project: Accumulo > Issue Type: Bug >Affects Versions: 1.6.0 >Reporter: John Vines >Priority: Major > > It creates cryptic stack traces that are made to help users find their > dangling BatchReader, but doesn't specify that. It should. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1198) Accumulo trace visualization
[ https://issues.apache.org/jira/browse/ACCUMULO-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1198. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Accumulo trace visualization > > > Key: ACCUMULO-1198 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1198 > Project: Accumulo > Issue Type: Sub-task > Components: monitor, trace >Reporter: Josh Elser >Priority: Trivial > Labels: gsoc2013, mentor > Attachments: minc-trace.png, trace-summary.png > > > The Accumulo monitor will show the recent traces to the user, typically for > minor and major compactions. This shows a basic tabbed hierarchy of each > action performed and the duration of the invocation. There is often some > additional data with context about what the action was performing (for > example, the tablet which was compacted). > It would be nice to have a nice graph/chart which displays the same > information but in a much prettier format. > Basic Java and HTML/Javascript understanding required. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3903) Add ability for Accumulo to store and display external Htrace traces in the Monitor pages
[ https://issues.apache.org/jira/browse/ACCUMULO-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3903. - Resolution: Fixed Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Add ability for Accumulo to store and display external Htrace traces in the > Monitor pages > - > > Key: ACCUMULO-3903 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3903 > Project: Accumulo > Issue Type: Improvement > Components: monitor, trace >Reporter: Philip Young >Priority: Minor > Labels: features > > I would like to use Accumulo as the central store of trace information for my > whole system (of which Accumulo is also a component). I am currently > integrating the existing Accumulo HTrace trace implementation into one of the > other components of my system. I would imagine that this would be also useful > for other people to use in tracing their whole system. > As such, I would like to be able to use the existing trace pages in the > Monitor for displaying all these traces. I would like to add the ability to > filter the traces that are displayed to particular system components > (services) and/or hosts. This could be implemented using either a drop down > or filter textbox to filter client-side the already de-serialised records. > There are a number of existing tickets that relate to this work that could be > done as part of this work which would centrally position Accumulo as an > enabler for tracing functionality for a whole system, they include: > # [ACCUMULO-1198|https://issues.apache.org/jira/browse/ACCUMULO-1198] > # [ACCUMULO-1013|https://issues.apache.org/jira/browse/ACCUMULO-1013] > # [ACCUMULO-3005|https://issues.apache.org/jira/browse/ACCUMULO-3005] > Also, even though the Accumulo Trace implementation can be used by other > components pretty much straight out of the box, it does require the > accumulo-site.xml file for it to work. This probably could be refactored to > not require all the site.xml for the trace server to work but some other > trace config file. > Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (ACCUMULO-3903) Add ability for Accumulo to store and display external Htrace traces in the Monitor pages
[ https://issues.apache.org/jira/browse/ACCUMULO-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs reopened ACCUMULO-3903: - > Add ability for Accumulo to store and display external Htrace traces in the > Monitor pages > - > > Key: ACCUMULO-3903 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3903 > Project: Accumulo > Issue Type: Improvement > Components: monitor, trace >Reporter: Philip Young >Priority: Minor > Labels: features > > I would like to use Accumulo as the central store of trace information for my > whole system (of which Accumulo is also a component). I am currently > integrating the existing Accumulo HTrace trace implementation into one of the > other components of my system. I would imagine that this would be also useful > for other people to use in tracing their whole system. > As such, I would like to be able to use the existing trace pages in the > Monitor for displaying all these traces. I would like to add the ability to > filter the traces that are displayed to particular system components > (services) and/or hosts. This could be implemented using either a drop down > or filter textbox to filter client-side the already de-serialised records. > There are a number of existing tickets that relate to this work that could be > done as part of this work which would centrally position Accumulo as an > enabler for tracing functionality for a whole system, they include: > # [ACCUMULO-1198|https://issues.apache.org/jira/browse/ACCUMULO-1198] > # [ACCUMULO-1013|https://issues.apache.org/jira/browse/ACCUMULO-1013] > # [ACCUMULO-3005|https://issues.apache.org/jira/browse/ACCUMULO-3005] > Also, even though the Accumulo Trace implementation can be used by other > components pretty much straight out of the box, it does require the > accumulo-site.xml file for it to work. This probably could be refactored to > not require all the site.xml for the trace server to work but some other > trace config file. > Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1078) Batch writer latency flushes aren't included in trace
[ https://issues.apache.org/jira/browse/ACCUMULO-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1078. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Batch writer latency flushes aren't included in trace > - > > Key: ACCUMULO-1078 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1078 > Project: Accumulo > Issue Type: Bug > Components: trace >Affects Versions: 1.4.2 >Reporter: Brian Loss >Assignee: Brian Loss >Priority: Major > > When using the batch writer, when it starts processing mutations due to the > latency timer expiring, any activity on that thread is not included in the > current distributed trace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3903) Add ability for Accumulo to store and display external Htrace traces in the Monitor pages
[ https://issues.apache.org/jira/browse/ACCUMULO-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3903. - Resolution: Abandoned > Add ability for Accumulo to store and display external Htrace traces in the > Monitor pages > - > > Key: ACCUMULO-3903 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3903 > Project: Accumulo > Issue Type: Improvement > Components: monitor, trace >Reporter: Philip Young >Priority: Minor > Labels: features > > I would like to use Accumulo as the central store of trace information for my > whole system (of which Accumulo is also a component). I am currently > integrating the existing Accumulo HTrace trace implementation into one of the > other components of my system. I would imagine that this would be also useful > for other people to use in tracing their whole system. > As such, I would like to be able to use the existing trace pages in the > Monitor for displaying all these traces. I would like to add the ability to > filter the traces that are displayed to particular system components > (services) and/or hosts. This could be implemented using either a drop down > or filter textbox to filter client-side the already de-serialised records. > There are a number of existing tickets that relate to this work that could be > done as part of this work which would centrally position Accumulo as an > enabler for tracing functionality for a whole system, they include: > # [ACCUMULO-1198|https://issues.apache.org/jira/browse/ACCUMULO-1198] > # [ACCUMULO-1013|https://issues.apache.org/jira/browse/ACCUMULO-1013] > # [ACCUMULO-3005|https://issues.apache.org/jira/browse/ACCUMULO-3005] > Also, even though the Accumulo Trace implementation can be used by other > components pretty much straight out of the box, it does require the > accumulo-site.xml file for it to work. This probably could be refactored to > not require all the site.xml for the trace server to work but some other > trace config file. > Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2938) Investigate logging on KeyExtent to ensure no data leakage
[ https://issues.apache.org/jira/browse/ACCUMULO-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2938. - Resolution: Not A Problem I recently refactored KeyExtent and did not observe any relevant problems of the type described here. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Investigate logging on KeyExtent to ensure no data leakage > -- > > Key: ACCUMULO-2938 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2938 > Project: Accumulo > Issue Type: Bug > Components: master, tserver >Reporter: Josh Elser >Priority: Critical > > The KeyExtent class identifies a Tablet in Accumulo. Of interest to this > issue, KeyExtent may contain the endRow of the Tablet and/or the endRow of > the previous Tablet (or neither). > If we log the extent, we have the potential to be leaking some data that > might need to be protected (visibilities, encryption) to a medium only > protected by filesystem restrictions. > This may be difficult since the extent is included in things like MinC and > MajC log messages and can be helpful when diagnosing problems on the system. > Can we abstract away what might be potentially sensitive data in some way > that we still provide useful data for debugging purposes? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3594) Consider message replacement for Writable classes internally
[
https://issues.apache.org/jira/browse/ACCUMULO-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3594.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Consider message replacement for Writable classes internally
>
>
> Key: ACCUMULO-3594
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3594
> Project: Accumulo
> Issue Type: Sub-task
>Reporter: Josh Elser
>Priority: Major
>
> We have a number of writable classes, some of which are non-trivial and
> serialized in tables or files:
> * {{org.apache.accumulo.core.tabletserver.log.LogEntry}}
> * {{org.apache.accumulo.core.metadata.schema.DataFileValue}}
> * {{org.apache.accumulo.server.master.state.TServerInstance}}
> Consider converting this (and possibly others) to better message classes to
> easily add more functionality in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2613) Take advantage of HDFS caching to improve MTTR
[ https://issues.apache.org/jira/browse/ACCUMULO-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-2613. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo > Take advantage of HDFS caching to improve MTTR > -- > > Key: ACCUMULO-2613 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2613 > Project: Accumulo > Issue Type: Improvement >Reporter: Sean Busbey >Priority: Critical > Labels: recovery > > Hadoop 2.3.0 added [HDFS > caching|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html]. > We should use this for small internal use tables (like !METADATA) and we > should probably have a configurable option to use it for tables, with a stern > warning that it should only be enabled on small tables that will be > frequently used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3619) GENERAL_RPC_TIMEOUT not exposed via ClientProperty's
[
https://issues.apache.org/jira/browse/ACCUMULO-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-3619.
-
Resolution: Auto Closed
Closing this stale issue. It may be OBE by new client code in 2.x. If this is
still a problem, please open a new issue or PR at
https://github.com/apache/accumulo
> GENERAL_RPC_TIMEOUT not exposed via ClientProperty's
>
>
> Key: ACCUMULO-3619
> URL: https://issues.apache.org/jira/browse/ACCUMULO-3619
> Project: Accumulo
> Issue Type: Bug
> Components: client
>Reporter: Josh Elser
>Priority: Critical
>
> I noticed the following issue digging through ACCUMULO-3616.
> {code:title=ClientContext.java|borderStyle=solid}
> public long getClientTimeoutInMillis() {
> return getConfiguration().getTimeInMillis(Property.GENERAL_RPC_TIMEOUT);
> }
> {code}
> In the client API impl, we extract the RPC timeout using a server-side
> Property with no property in existence in the client API through
> ClientProperty.
> While it would be best to change this in 1.6 (as it's a bug IMO), semver
> would keep us from doing that. Given that it's gone a few releases without
> anyone noticing, it's probably not critical to buck the processes. Fix it
> before 1.7.0 comes out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1541) User manual, section 4.5, should mention why the proxy might be useful.
[ https://issues.apache.org/jira/browse/ACCUMULO-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1541. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-proxy > User manual, section 4.5, should mention why the proxy might be useful. > > > Key: ACCUMULO-1541 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1541 > Project: Accumulo > Issue Type: Improvement > Components: docs >Affects Versions: 1.5.0 >Reporter: David Medinets >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-4243) Kerberos thrift servers don't adhere to general.server.message.size.max
[
https://issues.apache.org/jira/browse/ACCUMULO-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Tubbs resolved ACCUMULO-4243.
-
Resolution: Abandoned
Closing this stale issue. If this is still a problem, please open a new issue
or PR at https://github.com/apache/accumulo
> Kerberos thrift servers don't adhere to general.server.message.size.max
> ---
>
> Key: ACCUMULO-4243
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4243
> Project: Accumulo
> Issue Type: Bug
> Components: rpc
>Affects Versions: 1.7.0, 1.7.1
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
>
> It looks like when I implemented Kerberos client authentication in
> ACCUMULO-2815, I botched the use of general.server.message.size.max.
> This property is meant to guard us against trying to allocate a very large
> buffer from an RPC call. Typically, the first four bytes of data sent to one
> of our thrift servers (tserver, master, proxy) is treated as the number of
> bytes to read for some RPC. The problem is that if some garbage data (often,
> a security scan by some admins) may inadvertently cause the server to try to
> allocate a very large buffer (GBs in size) which will cause the process to
> ultimately crash while instantiating the buffer.
> It seems like something might be handled differently in the TSaslServer in
> Thrift. I'll have to dig more into whether it's an Accumulo or Thrift bug.
> To reproduce this, I was able to use telnet:
> {noformat}
> $ telnet `hostname -f` 9997
> <...waiting to get connected...>
> 00
> ..
> {noformat}
> This will try to create a very large buffer (~2.1GB). I observed this by
> hooking up jvisualvm to the tabletserver.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-1887) Proxy README is unclear.
[ https://issues.apache.org/jira/browse/ACCUMULO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-1887. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-proxy > Proxy README is unclear. > > > Key: ACCUMULO-1887 > URL: https://issues.apache.org/jira/browse/ACCUMULO-1887 > Project: Accumulo > Issue Type: Improvement > Components: docs >Affects Versions: 1.5.0 >Reporter: David Medinets >Priority: Trivial > Labels: newbie > > The Proxy README says: > The proxy server is installed during the Accumulo installation process. Read > ../README for more information. > It is unclear to me if the "more information" refers to the Proxy process or > to the Accumulo installation process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-2348) Consolidate process ZooKeeper Lock[Watcher] code
[ https://issues.apache.org/jira/browse/ACCUMULO-2348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-2348. -- Resolution: Won't Do Too vague and may no longer be relevant. > Consolidate process ZooKeeper Lock[Watcher] code > > > Key: ACCUMULO-2348 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2348 > Project: Accumulo > Issue Type: Improvement > Components: gc, master, monitor, tserver >Reporter: Josh Elser >Priority: Major > > The master, monitor, GC and tserver all get a ZooLock for themselves and set > a similar Watcher. We should consolidate the watchers into one class that can > be re-used across the processes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3506) Create integration test for proxy running over SSL
[ https://issues.apache.org/jira/browse/ACCUMULO-3506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Christopher Tubbs resolved ACCUMULO-3506. - Resolution: Abandoned Closing this stale issue. If this is still a problem, please open a new issue or PR at https://github.com/apache/accumulo-proxy > Create integration test for proxy running over SSL > -- > > Key: ACCUMULO-3506 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3506 > Project: Accumulo > Issue Type: Sub-task > Components: proxy, test >Reporter: Josh Elser >Priority: Major > > ACCUMULO-3452 indirectly added support for starting the proxy with SSL (by > virtue of leveraging the existing code). > We need to write tests to ensure that it is functional and nothing is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (ACCUMULO-3206) New Public API: Approximate data counts of Tablets and Tablet Servers
[ https://issues.apache.org/jira/browse/ACCUMULO-3206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miller resolved ACCUMULO-3206. -- Resolution: Fixed The new Summaries that have been added in 2.x should meet this need. https://accumulo.apache.org/docs/2.x/development/summaries Any additional requirements can be opened as an issue on github. > New Public API: Approximate data counts of Tablets and Tablet Servers > - > > Key: ACCUMULO-3206 > URL: https://issues.apache.org/jira/browse/ACCUMULO-3206 > Project: Accumulo > Issue Type: New Feature > Components: client, monitor >Affects Versions: 1.6.1 >Reporter: Shana Hutchison >Priority: Minor > > The broader picture is public programmatic access to information in the > Accumulo monitor. Specifically I'm looking to obtain the number of entries > per tablet and per tablet server for a given table. The use case is to > verify that manually set (or automatically set I suppose) table splits are > effectively dividing Accumulo data among many tablets, that is, verifying > load balancing. > I wrote Accumulo 1.5 code which uses non-public API to obtain this > information in the same way the Monitor does via TabletStats. The tricky part > was cross-referencing the Metadata table to find the assignment of tablets to > tablet servers for a given table. I rewrote that code for 1.6, switching the > name of the Metadata table to "accumulo.metadata" and other associated > changes, but it would be great to make this part of the public API so that > people don't have to use non-public methods to obtain data that Accumulo has > in the Monitor and Metadata table anyway. > We could approach this by adding to the TableOperations class or something > similar. A request could go to an Accumulo master which gathers the > necessary information from the tablet servers just as the Monitor does, so > that the client does not have to do it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [accumulo] tynyttie commented on issue #1628: Add test where compaction selector sporadically throws an exception.
tynyttie commented on issue #1628: URL: https://github.com/apache/accumulo/issues/1628#issuecomment-718127854 @keith-turner Is this ticket still relevant base on your new updates to Compaction? If so, what components do I directly affect to accomplish the task? Plus, what was your expected result? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] milleruntime merged pull request #1754: Update jvm option for JDWP in Mini
milleruntime merged pull request #1754: URL: https://github.com/apache/accumulo/pull/1754 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] milleruntime opened a new pull request #1757: Create Ample interface method for createDeleteMutation
milleruntime opened a new pull request #1757: URL: https://github.com/apache/accumulo/pull/1757 * Drop static method in favor of method on the Ample interface for creating delete mutations * Change GarbageCollectorIT to use Ample for one of the tests * Reuse Ample object in loops to prevent extra object creation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] EdColeman commented on issue #1007: ZooReaderWriter.putData may create data, timeout, then fail
EdColeman commented on issue #1007: URL: https://github.com/apache/accumulo/issues/1007#issuecomment-718086806 I can look at this as part of the configuration changes that I'm going to work on when my current task completes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] ctubbsii merged pull request #1755: ZooReader/ZooReaderWriter cleanup
ctubbsii merged pull request #1755: URL: https://github.com/apache/accumulo/pull/1755 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] ctubbsii commented on pull request #1756: Configure surefire/failsafe forkCount separately
ctubbsii commented on pull request #1756: URL: https://github.com/apache/accumulo/pull/1756#issuecomment-718073544 Merged to main in c4e47bab9d86eca894317efcb19fa11e6225425f This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [accumulo] ctubbsii merged pull request #1756: Configure surefire/failsafe forkCount separately
ctubbsii merged pull request #1756: URL: https://github.com/apache/accumulo/pull/1756 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
