Re: reading rfiles directly

[Keith Turner]

Could the Accumulo Map Reduce input format and enable scanning an
offline table. This will read the tables rfiles directly excluding any
data falling outside of tablet boundaries.  Since this is a Hadoop
input format, it should work easily with Spark.  I can point to
examples of this if interested.

Another option is using the RFile class (added in 1.8) in the public
API to directly read individual RFiles, this is useful when tables and
tablets are not a concern.  I have not used this with Spark, but I
think it would work easily by  partitioning a list of files into task
and having each task read a set of rfiles directly.

On Mon, Aug 3, 2020 at 4:46 PM Bulldog20630405
 wrote:
>
>
> we would like to read rfiles directly outside an active accumulo instance 
> using spark.  is there a example to do this?
>
> note: i know there is an utility to print rfiles and i could start there and 
> build my own; but was hoping to leverage something already there.

Re: Unit/Integration tests with Mini Accumulo Cluster

[Keith Turner]

One more point about Fluo.  ITBase relies on maven setting a property.
That is done by the following link.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/pom.xml#L333

On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes  wrote:
>
> Hi all,
>
> I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> MockAccumulo infrastructure for our unit/integration tests which run in
> a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> looking at using the MiniAccumulo cluster infrastructure.
>
> Are there best practices or examples of how to best use and reuse such a
> cluster between unit tests?
>
> I see two broad options:  First, using Maven's integration test
> framework to start a cluster 'outside' of a unit test.  Second, we've
> build a test runner for HBase which starts a cluster and then runs the
> various tests.  This second approach has some small downsides, and
> that's why I'm asking if the you all have any great ideas!
>
> Cheers,
>
> Jim
>

Re: Unit/Integration tests with Mini Accumulo Cluster

[Keith Turner]

You could use the Accumulo Maven Plugin which starts an Accumulo
cluster for using MiniAccumulo.

https://accumulo.apache.org/accumulo2-maven-plugin/

Fluo, uses this and below are a few pointers.

The following is a base for Fluo ITs that uses the mini Accumulo
started by the maven plugin if it exists, otherwise it starts a mini
(useful for running in an IDE).

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/modules/integration-tests/src/main/java/org/apache/fluo/integration/ITBase.java#L74

Fluo is a multi module project.  The link below shows configuring the
plugin in the parent pom.  Plugins do not inherit the dependencies of
the project, that is why the plugin explicitly configures the
Accumulo, Hadoop, and ZK versions.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/pom.xml#L279

The link below shows activating the plugin in the IT module.  It
inherits the config from the module.  Not all modules activate the
plugin.

https://github.com/apache/fluo/blob/549d645addb330f4ae2e074447428cb86b5a9a3f/modules/integration-tests/pom.xml#L103

On Fri, Dec 13, 2019 at 10:30 AM Jim Hughes  wrote:
>
> Hi all,
>
> I work on GeoMesa and for the Accumulo 1.x line, we have been using the
> MockAccumulo infrastructure for our unit/integration tests which run in
> a Maven build.  In Accumulo 2.x, since MockAccumulo is gone, we're
> looking at using the MiniAccumulo cluster infrastructure.
>
> Are there best practices or examples of how to best use and reuse such a
> cluster between unit tests?
>
> I see two broad options:  First, using Maven's integration test
> framework to start a cluster 'outside' of a unit test.  Second, we've
> build a test runner for HBase which starts a cluster and then runs the
> various tests.  This second approach has some small downsides, and
> that's why I'm asking if the you all have any great ideas!
>
> Cheers,
>
> Jim
>

Re: Why may "tablet read ahead" take long time? (was: Profile a (batch) scan)

[Keith Turner]

Maxim,

Below are some pointers into the code you asked about.  Let me know if
you have any questions about the code.

The following tablet server code puts a task on a thread pool for a batch scan.

https://github.com/apache/accumulo/blob/rel/1.9.2/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java#L776

This thread pool is created here.

https://github.com/apache/accumulo/blob/rel/1.9.2/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java#L227

The function that creates the thread pool calls the following code to
wrap the thread pool

https://github.com/apache/accumulo/blob/rel/1.9.2/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java#L111

Keith

On Wed, Jan 16, 2019 at 4:09 AM Maxim Kolchin  wrote:
>
> Hi Adam,
>
> Thank you for the list!
>
> In my case, there is only one client which runs a single query which I'm 
> tracing, so I hope the thread pools have free threads.
>
> I'd like to look at the code which runs in the scope of the "table read 
> ahead". Do you know where I should look at? I tried to search by keywords on 
> GitHub, but it wasn't able to find a string "table read ahead" which is given 
> to the tracer.
>
> Cheers,
> Maxim
>
> On Tue, Jan 15, 2019 at 9:01 PM Adam Fuchs  wrote:
>>
>> Hi Maxim,
>>
>> What you're seeing is an artifact of the threading model that Accumulo uses. 
>> When you launch a query, Accumulo tablet servers will coordinate RPCs via 
>> Thrift in one thread pool (which grows unbounded) and queue up scans (rfile 
>> lookups, decryption/decompression, iterators, etc.) in another threadpool 
>> known as the readahead pool (which has a fixed number of threads). You're 
>> seeing everything that happens in that readahead thread in one big chunk. 
>> You may need to look a bit deeper into profiling/sampling tablet server CPU 
>> to get insights into how to improve your query performance. If you want to 
>> speed up queries in general you might try (in no particular order):
>> 1. Increase parallelism by bumping up the readahead threads 
>> (tserver.readahead.concurrent.max). This will still be bounded by the number 
>> of parallel scans clients are driving.
>> 2. Increase parallelism driven by clients by querying more, smaller ranges, 
>> or by splitting tablets.
>> 3. Increase scan batch sizes if the readahead thread or thrift coordination 
>> overhead is high.
>> 4. Optimize custom iterators if that is a CPU bottleneck.
>> 5. Increase cache sizes or otherwise modify queries to improve cache hit 
>> rates.
>> 6. Change compression settings if that is a CPU bottleneck. Try snappy 
>> instead of gz.
>>
>> Cheers,
>> Adam
>>
>> On Tue, Jan 15, 2019, 10:45 AM Maxim Kolchin >>
>>> Hi all,
>>>
>>> I try to trace some scans with Zipkin and see that quite often the trace 
>>> called "tablet read ahead" takes 10x or 100x more time than the other 
>>> similar traces.
>>>
>>> Why it may happen? What could be done to reduce the time? I found a similar 
>>> discussion on the list, but it doesn't have an answer. I'd be great to have 
>>> a how-to article listing some steps which could be done.
>>>
>>> Attaching a screenshot of one of the traces having this issue.
>>>
>>> Maxim Kolchin
>>>
>>> E-mail: kolchin...@gmail.com
>>> Tel.: +7 (911) 199-55-73
>>> Homepage: http://kolchinmax.ru
>>>
 Below you can find a good example of what I'm struggling to understand
 right now. It's a trace for a simple scan over some columns with a
 BatchScanner using 75 threads. The scan takes 877 milliseconds and the main
 contributor is the entry "tablet read ahead 1", which starts at 248 ms.
 These are the questions that I cannot answer with this trace:

1. why this heavy operation starts after 248ms? By summing up the delay
before this operation you get a number which is not even close to 248ms.
2. what does "tablet read ahead 1" means? In general, how to map the
entries of a trace to their meaning? Is there a guide about this?
3. why "tablet read ahead 1" takes 600ms? It's clearly not the sum of
the entries under this one but that's the important part.
4. I may be naive but...how much data have been read by this scan? How
many entries? That's very important to understand what's going on.

 Thanks for the help,

 Mario

 877+ 0 Dice@h01 counts
 2+ 7 tserver@h12 startScan
 6+ 10 tserver@h15 startScan
 5+ 11 tserver@h15 metadata tablets read ahead 4
 843+ 34 Dice@h01 batch scanner 74- 1
 620+ 230 tserver@h09 startMultiScan
 600+ 248 tserver@h09 tablet read ahead 1
 22+ 299 tserver@h09 newDFSInputStream
 22+ 299 tserver@h09 getBlockLocations
 2+ 310 tserver@h09 ClientNamenodeProtocol#getBlockLocations
 1+ 321 tserver@h09 getFileInfo
 1+ 321 tserver@h09 ClientNamenodeProtocol#getFileInfo
 2+ 322 tserver@h09 DFSInputStream#byteArrayRead
 1+ 324 

Re: Accumulo on Google Cloud Storage

[Keith Turner]

Maxim,

This is very interesting.  Would you be interested in writing an
Accumulo blog post about your experience?  If you are interested I can
help.

Keith

On Tue, Jan 15, 2019 at 10:03 AM Maxim Kolchin  wrote:
>
> Hi,
>
> I just wanted to leave intermediate feedback on the topic.
>
> So far, Accumulo works pretty well on top of Google Storage. The 
> aforementioned issue still exists, but it doesn't break anything. However, I 
> can't give you any useful performance numbers at the moment.
>
> The cluster:
>
>  - master (with zookeeper) (n1-standard-1) + 2 tservers (n1-standard-4)
>  - 32+ billlion entries
>  - 5 tables (excluding system tables)
>
> Some averaged numbers from two use cases:
>
>  - batch write into pre-splitted tables with 40 client machines + 4 tservers 
> (n1-standard-4) - max speed 1.5M entries/sec.
>  - sequential read with 2 client iterators (1 - filters by key, 2- filters by 
> timestamp), with 5 client machines +  2 tservers (n1-standard-4 ) and less 
> than 60k entries returned - max speed 1M+ entries/sec.
>
> Maxim
>
> On Mon, Jun 25, 2018 at 12:57 AM Christopher  wrote:
>>
>> Ah, ok. One of the comments on the issue led me to believe that it was the 
>> same issue as the missing custom log closer.
>>
>> On Sat, Jun 23, 2018, 01:10 Stephen Meyles  wrote:
>>>
>>> > I'm not convinced this is a write pattern issue, though. I commented on..
>>>
>>> The note there suggests the need for a LogCloser implementation; in my 
>>> (ADLS) case I've written one and have it configured - the exception I'm 
>>> seeing involves failures during writes, not during recovery (though it then 
>>> leads to a need for recovery).
>>>
>>> S.
>>>
>>> On Fri, Jun 22, 2018 at 4:33 PM, Christopher  wrote:

 Unfortunately, that feature wasn't added until 2.0, which hasn't yet been 
 released, but I'm hoping it will be later this year.

 However, I'm not convinced this is a write pattern issue, though. I 
 commented on 
 https://github.com/GoogleCloudPlatform/bigdata-interop/issues/103#issuecomment-399608543

 On Fri, Jun 22, 2018 at 1:50 PM Stephen Meyles  wrote:
>
> Knowing that HBase has been run successfully on ADLS, went looking there 
> (as they have the same WAL write pattern). This is informative:
>
> 
> https://www.cloudera.com/documentation/enterprise/5-12-x/topics/admin_using_adls_storage_with_hbase.html
>
> which suggests a need to split the WALs off on HDFS proper versus ADLS 
> (or presumably GCS) barring changes in the underlying semantics of each. 
> AFAICT you can't currently configure Accumulo to send WAL logs to a 
> separate cluster - is this correct?
>
> S.
>
>
> On Fri, Jun 22, 2018 at 9:07 AM, Stephen Meyles  wrote:
>>
>> > Did you try to adjust any Accumulo properties to do bigger writes less 
>> > frequently or something like that?
>>
>> We're using BatchWriters and sending reasonable larges batches of 
>> Mutations. Given the stack traces in both our cases are related to WAL 
>> writes it seems like batch size would be the only tweak available here 
>> (though, without reading the code carefully it's not even clear to me 
>> that is impactful) but if there others have suggestions I'd be happy to 
>> try.
>>
>> Given we have this working well and stable in other clusters atop 
>> traditional HDFS I'm currently pursuing this further with the MS to 
>> understand the variance to ADLS. Depending what emerges from that I may 
>> circle back with more details and a bug report and start digging in more 
>> deeply to the relevant code in Accumulo.
>>
>> S.
>>
>>
>> On Fri, Jun 22, 2018 at 6:09 AM, Maxim Kolchin  
>> wrote:
>>>
>>> > If somebody is interested in using Accumulo on GCS, I'd like to 
>>> > encourage them to submit any bugs they encounter, and any patches (if 
>>> > they are able) which resolve those bugs.
>>>
>>> I'd like to contribute a fix, but I don't know where to start. We tried 
>>> to get any help from the Google Support about [1] over email, but they 
>>> just say that the GCS doesn't support such write pattern. In the end, 
>>> we can only guess how to adjust the Accumulo behaviour to minimise 
>>> broken connections to the GCS.
>>>
>>> BTW although we observe this exception, the tablet server doesn't fail, 
>>> so it means that after some retries it is able to write WALs to GCS.
>>>
>>> @Stephen,
>>>
>>> > as discussions with MS engineers have suggested, similar to the GCS 
>>> > thread, that small writes at high volume are, at best, suboptimal for 
>>> > ADLS.
>>>
>>> Did you try to adjust any Accumulo properties to do bigger writes less 
>>> frequently or something like that?
>>>
>>> [1]: https://github.com/GoogleCloudPlatform/bigdata-interop/issues/103
>>>
>>> Maxim
>>>

Re: Accumulo performance on various hardware configurations

[Keith Turner]

On Wed, Aug 29, 2018 at 9:37 AM, guy sharon  wrote:
> hi Marc,
>
> Just ran the test again with the changes you suggested. Setup: 5 tservers on
> CentOS 7, 4 CPUs and 16 GB RAM, Accumulo 1.7.4, table with 6M rows.
> org.apache.accumulo.examples.simple.helloworld.ReadData now uses a
> BatchScanner with 10 threads. I got:
>
> $ time install/accumulo-1.7.4/bin/accumulo
> org.apache.accumulo.examples.simple.helloworld.ReadData -i muchos -z
> localhost:2181 -u root -t hellotable -p secret

Doing performance test this way has overhead that would not be seen in
a long running java process.  I think the accumulo script runs a java
process to check something before running the java process for
ReadData.  Also, Java is really slow when it first starts because it
is interpreting and loading classes.   When the classes are loaded and
the byte code is compiled, execution is much faster.   For performance
test I usually loop a few times in java running the same test and
printing out the time for each iteration.  The first run is usually
really slow, then it speeds up until the times stabilize.


>
> real0m16.979s
> user0m13.670s
> sys0m0.599s
>
> So this doesn't really improve things. That looks strange to me as I'd
> expect Accumulo to use the threads to speed things up. Unless the full scan
> makes it use just one thread with the assumption that the entries are next
> to each other on the disk making it faster to read them sequentially rather
> than jump back and forth with threads. What do you think?
>
> BR,
> Guy.
>
>
>
>
> On Wed, Aug 29, 2018 at 3:25 PM Marc  wrote:
>>
>> Guy,
>>   To clarify :
>>
>> [1] If you have four tablets it's reasonable to suspect that the RPC
>> time to access those servers may increase a bit if you access them
>> sequentially versus in parallel.
>> On Wed, Aug 29, 2018 at 8:16 AM Marc  wrote:
>> >
>> > Guy,
>> >   The ReadData example appears to use a sequential scanner. Can you
>> > change that to a batch scanner and see if there is improvement [1]?
>> > Also, while you are there can you remove the log statement or set your
>> > log level so that the trace message isn't printed?
>> >
>> > In this case we are reading the entirety of that data. If you were to
>> > perform a query you would likely prefer to do it at the data instead
>> > of bringing all data back to the client.
>> >
>> > What are your expectations since it appears very slow. Do you want
>> > faster client side access to the data? Certainly improvements could be
>> > made -- of that I have no doubt -- but the time to bring 6M entries to
>> > the client is a cost you will incur if you use the ReadData example.
>> >
>> > [1] If you have four tablets it's reasonable to suspect that the RPC
>> > time to access those servers may increase a bit.
>> >
>> > On Wed, Aug 29, 2018 at 8:05 AM guy sharon 
>> > wrote:
>> > >
>> > > hi,
>> > >
>> > > Continuing my performance benchmarks, I'm still trying to figure out
>> > > if the results I'm getting are reasonable and why throwing more hardware 
>> > > at
>> > > the problem doesn't help. What I'm doing is a full table scan on a table
>> > > with 6M entries. This is Accumulo 1.7.4 with Zookeeper 3.4.12 and Hadoop
>> > > 2.8.4. The table is populated by
>> > > org.apache.accumulo.examples.simple.helloworld.InsertWithBatchWriter
>> > > modified to write 6M entries instead of 50k. Reads are performed by
>> > > "bin/accumulo org.apache.accumulo.examples.simple.helloworld.ReadData -i
>> > > muchos -z localhost:2181 -u root -t hellotable -p secret". Here are the
>> > > results I got:
>> > >
>> > > 1. 5 tserver cluster as configured by Muchos
>> > > (https://github.com/apache/fluo-muchos), running on m5d.large AWS 
>> > > machines
>> > > (2vCPU, 8GB RAM) running CentOS 7. Master is on a separate server. Scan 
>> > > took
>> > > 12 seconds.
>> > > 2. As above except with m5d.xlarge (4vCPU, 16GB RAM). Same results.
>> > > 3. Splitting the table to 4 tablets causes the runtime to increase to
>> > > 16 seconds.
>> > > 4. 7 tserver cluster running m5d.xlarge servers. 12 seconds.
>> > > 5. Single node cluster on m5d.12xlarge (48 cores, 192GB RAM), running
>> > > Amazon Linux. Configuration as provided by Uno
>> > > (https://github.com/apache/fluo-uno). Total time was 26 seconds.
>> > >
>> > > Offhand I would say this is very slow. I'm guessing I'm making some
>> > > sort of newbie (possibly configuration) mistake but I can't figure out 
>> > > what
>> > > it is. Can anyone point me to something that might help me find out what 
>> > > it
>> > > is?
>> > >
>> > > thanks,
>> > > Guy.
>> > >
>> > >

[ANNOUNCE] Apache Accumulo 1.9.2 (Critical Bug Fixes)

[Keith Turner]

The Apache Accumulo project is pleased to announce the release
of Apache Accumulo 1.9.2! This release contains fixes for **critical**
bugs, which could result in unexpected problems during recovery from
a previous failure. (See the release notes linked below for details.)

Versions 1.8.0, 1.8.1, 1.9.0, and 1.9.1 are affected, and users of those
versions are encouraged to upgrade to this version immediately to
avoid serious problems. Users of earlier versions who are planning to
upgrade to one of the affected versions are encouraged to upgrade
directly to this version instead.

***

Apache Accumulo® is a sorted, distributed key/value store that
provides robust, scalable data storage and retrieval. With
Apache Accumulo, users can store and manage large data sets
across a cluster. Accumulo uses Apache Hadoop's HDFS to store
its data and Apache ZooKeeper for consensus.

This version is now available in Maven Central, and at:
https://accumulo.apache.org/downloads/

The full release notes can be viewed at:
https://accumulo.apache.org/release/accumulo-1.9.2/

--
The Apache Accumulo Team

Re: Corrupt WAL

[Keith Turner]

On Tue, Jun 12, 2018 at 12:10 PM, Adam J. Shook  wrote:
> Yes, that is the error.  I'll inspect the logs and report back.

Ok.  The LogReader command has a mechanism to filter which tablet is
displayed.  If the walog has  alot of data in it, may need to use
this.

Also, be aware that only 5 mutations are shown for a "many mutations"
objects in the walog.   The -m options changes this.  May want to see
more when deciding if the info in the log is important.


>
> On Tue, Jun 12, 2018 at 10:14 AM, Keith Turner  wrote:
>>
>> Is the message you are seeing "COMPACTION_FINISH (without preceding
>> COMPACTION_START)" ?  That messages indicates that the WALs are
>> incomplete, probably as a result of the NN problems.  Could do the
>> following :
>>
>> 1) Run the following command to see whats in the log.  Need to see
>> what is there for the root tablet.
>>
>>accumulo org.apache.accumulo.tserver.logger.LogReader
>>
>> 2) Replace the log file with an empty file after seeing if there is
>> anything important in it.
>>
>> I think the list of WALs for the root tablet is stored in ZK at
>> /accumulo//walogs
>>
>> On Mon, Jun 11, 2018 at 5:26 PM, Adam J. Shook 
>> wrote:
>> > Hey all,
>> >
>> > The root tablet on one of our dev systems isn't loading due to an
>> > illegal
>> > state exception -- COMPACTION_FINISH preceding COMPACTION_START.  What'd
>> > be
>> > the best way to mitigate this issue?  This was likely caused due to both
>> > of
>> > our NameNodes failing.
>> >
>> > Thank you,
>> > --Adam
>
>

Re: Corrupt WAL

[Keith Turner]

Is the message you are seeing "COMPACTION_FINISH (without preceding
COMPACTION_START)" ?  That messages indicates that the WALs are
incomplete, probably as a result of the NN problems.  Could do the
following :

1) Run the following command to see whats in the log.  Need to see
what is there for the root tablet.

   accumulo org.apache.accumulo.tserver.logger.LogReader

2) Replace the log file with an empty file after seeing if there is
anything important in it.

I think the list of WALs for the root tablet is stored in ZK at
/accumulo//walogs

On Mon, Jun 11, 2018 at 5:26 PM, Adam J. Shook  wrote:
> Hey all,
>
> The root tablet on one of our dev systems isn't loading due to an illegal
> state exception -- COMPACTION_FINISH preceding COMPACTION_START.  What'd be
> the best way to mitigate this issue?  This was likely caused due to both of
> our NameNodes failing.
>
> Thank you,
> --Adam

Re: Large numbers of authorizations

[Keith Turner]

I found and fixed this issue.

https://github.com/apache/accumulo/pull/410

On Fri, Mar 23, 2018 at 1:10 PM, Michael Ladakos  wrote:
>
> -- Forwarded message --
> From: Michael Ladakos 
> Date: Fri, Mar 23, 2018 at 12:32 PM
> Subject: Large numbers of authorizations
> To: user@accumulo.apache.org
>
>
> I am somewhat new to Accumulo and was doing some experimentation on
> consequences for using large numbers of authorizations.
>
> I found that a user with a large set of authorizations would take a great
> deal of time to perform a scan. I tested at various increments up to 100,000
> authorizations. At that point, it would take at least 25 seconds to perform
> the scan, even if the table was newly created with no rows.
>
> Performing a scan with a small subset of authorizations is equivalent to
> performing a query with a user that only has a small number of
> authorizations.
>
> I attempted to find the place in the code where whatever is being done,
> because I wanted to understand what caused this, but I wasn't able to track
> down the exact class. Any chance I could get an explanation or pointed in
> the right direction?
>
> Thanks!
>

Re: Large numbers of authorizations

[Keith Turner]

On Fri, Mar 23, 2018 at 11:55 PM, Keith Turner <ke...@deenlo.com> wrote:
> On Fri, Mar 23, 2018 at 4:06 PM, mdladakos <mdlada...@gmail.com> wrote:
>> Keith, thanks for your quick response!
>>
>> Maybe I wasn't clear enough or I am not understanding your explanation.
>>
>> What I was exploring was performing a scan with a large number of
>> authorizations. While I did use tables with thousands of rows, I also ran
>> scans against empty tables and still performed at ~25 Seconds. So shouldn't
>> VisibilityEvaluator not be in involved?
>>
>> I don't think the actual filtering is the problem. Is there some work done
>> by the tablet servers when receiving the scan request, specifically in
>> regard to user authorizations?
>>
>> Again, if I used -s to pass a subset of authorizations for the user with
>> 10 authorizations, this increase in return time would be equivalent to a
>> user with that number of authorizations (i.e.: If I scanned with 100
>> authorizations out of the 10, it would be the normal, fast speed)
>
> I think the following code may be the problem. The collection
> userauths is a list, so performance will O(M*N).  Is M and N are 100K,
> then its not good.  If userauths were a set this would be much faster
> for the case you are testing.
>
> https://github.com/apache/accumulo/blob/17bc708dcabd17824a8378597e0542002470ed18/server/base/src/main/java/org/apache/accumulo/server/security/handler/ZKAuthorizor.java#L166

This code is called on the server side to check if the auth passed by
a scan are valid.

>>
>>
>>
>> --
>> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Users-f2.html

Re: Large numbers of authorizations

[Keith Turner]

On Fri, Mar 23, 2018 at 4:06 PM, mdladakos  wrote:
> Keith, thanks for your quick response!
>
> Maybe I wasn't clear enough or I am not understanding your explanation.
>
> What I was exploring was performing a scan with a large number of
> authorizations. While I did use tables with thousands of rows, I also ran
> scans against empty tables and still performed at ~25 Seconds. So shouldn't
> VisibilityEvaluator not be in involved?
>
> I don't think the actual filtering is the problem. Is there some work done
> by the tablet servers when receiving the scan request, specifically in
> regard to user authorizations?
>
> Again, if I used -s to pass a subset of authorizations for the user with
> 10 authorizations, this increase in return time would be equivalent to a
> user with that number of authorizations (i.e.: If I scanned with 100
> authorizations out of the 10, it would be the normal, fast speed)

I think the following code may be the problem. The collection
userauths is a list, so performance will O(M*N).  Is M and N are 100K,
then its not good.  If userauths were a set this would be much faster
for the case you are testing.

https://github.com/apache/accumulo/blob/17bc708dcabd17824a8378597e0542002470ed18/server/base/src/main/java/org/apache/accumulo/server/security/handler/ZKAuthorizor.java#L166
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Users-f2.html

Re: Large numbers of authorizations

[Keith Turner]

On Fri, Mar 23, 2018 at 4:06 PM, mdladakos  wrote:
> Keith, thanks for your quick response!
>
> Maybe I wasn't clear enough or I am not understanding your explanation.
>
> What I was exploring was performing a scan with a large number of
> authorizations. While I did use tables with thousands of rows, I also ran
> scans against empty tables and still performed at ~25 Seconds. So shouldn't
> VisibilityEvaluator not be in involved?

Gotcha.  So one possibility is its just taking a while to send the
auths from the client to the tserver.  The following code is the
thrift RPC to start a scan.  client.startScan() is passed
scanState.authorizations.getAuthorizationsBB() which is the auths.
The getAuthorizationsBB() method does a copy.  So there is a copy,
then thrift has to serialize auths, send them, and then deserialize on
server side.. and this is done for each startScan RPC.  The startScan
call happens once per tablet, subsequent batches of key/vals from a
tablet are fetched using contunueScan RPC which does not pass auths
again.

https://github.com/apache/accumulo/blob/1e4d4827096bd0047c7de3e0b672263defe66634/core/src/main/java/org/apache/accumulo/core/client/impl/ThriftScanner.java#L429

It would be interesting to see how long the call to startScan takes
for your case.  Enabling trace logging for ThriftScanner will give
some insight into this.

>
> I don't think the actual filtering is the problem. Is there some work done
> by the tablet servers when receiving the scan request, specifically in
> regard to user authorizations?
>
> Again, if I used -s to pass a subset of authorizations for the user with
> 10 authorizations, this increase in return time would be equivalent to a
> user with that number of authorizations (i.e.: If I scanned with 100
> authorizations out of the 10, it would be the normal, fast speed)
>
>
>
> --
> Sent from: http://apache-accumulo.1065345.n5.nabble.com/Users-f2.html

Re: Large numbers of authorizations

[Keith Turner]

This is the code that scans use to filter based on column visibility
and authorizations.  It has a cache of previously seen column
visibilities and the decision that was made for those.

  
https://github.com/apache/accumulo/blob/2e171cdb8420f817ff9ebeb23f9d8a70b0878ca5/core/src/main/java/org/apache/accumulo/core/iterators/system/VisibilityFilter.java

The following code does the evaluation.

  
https://github.com/apache/accumulo/blob/f81a8ec7410e789d11941351d5899b8894c6a322/core/src/main/java/org/apache/accumulo/core/security/VisibilityEvaluator.java

On Fri, Mar 23, 2018 at 1:10 PM, Michael Ladakos  wrote:
>
> -- Forwarded message --
> From: Michael Ladakos 
> Date: Fri, Mar 23, 2018 at 12:32 PM
> Subject: Large numbers of authorizations
> To: user@accumulo.apache.org
>
>
> I am somewhat new to Accumulo and was doing some experimentation on
> consequences for using large numbers of authorizations.
>
> I found that a user with a large set of authorizations would take a great
> deal of time to perform a scan. I tested at various increments up to 100,000
> authorizations. At that point, it would take at least 25 seconds to perform
> the scan, even if the table was newly created with no rows.
>
> Performing a scan with a small subset of authorizations is equivalent to
> performing a query with a user that only has a small number of
> authorizations.
>
> I attempted to find the place in the code where whatever is being done,
> because I wanted to understand what caused this, but I wasn't able to track
> down the exact class. Any chance I could get an explanation or pointed in
> the right direction?
>
> Thanks!
>

Re: Adding empty string split causes consistency check to failed for merge command

[Keith Turner]

On Mon, Mar 5, 2018 at 2:16 PM, Dong Zhou <dzho...@gmail.com> wrote:
> Hi Keith,
>
> Thanks for the remedy steps, I tried the instruction and it worked. Yay.
>
> I was trying to fix the issue using similar steps, but I missed out offline
> and online the table part, which leads me to not able to run merge command
> with following error:
>> 2018-03-05 19:10:06,037 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,038 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>> 2018-03-05 19:10:06,151 [master.Master] INFO : Asking
>> tserver1:10011[161d36db90563c3] to chop 3;a<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Computing next merge
>> state for 3<< which is presently WAITING_FOR_CHOPPED isDelete : false
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : 21 tablets are chopped
>> 3<<
>> 2018-03-05 19:10:06,151 [state.MergeStats] INFO : Waiting for 21 chopped
>> tablets to be 22 3<<
>
> One more question for the supplementary part that the "" split has files.
> Would it be a good idea to run hdfs command to physically move the file into

That may be helpful because I am not sure if the Accumulo Garbage
Collector will ever delete the directory when it becomes empty.  You
could compact the tablets referencing the file and then delete the dir
when its empty.

If moving the file, then you need to be careful because multiple files
may reference the tablet.  You can grep the metadata table for the
file to be sure.

> the next tablet folder, and then insert an entry in metadata for it.
> Or it does not matter, I can just reuse the folder that was created for ""
> split.
>
> Thanks,
>
> -Dong Zhou
>
>
>
> On Mon, Mar 5, 2018 at 9:34 AM, Keith Turner <ke...@deenlo.com> wrote:
>>
>> I omitted something in with my previous email. I assumed the the
>> tablet with an empty row could have no data.  However, it can have
>> data.  If the tablet has files, they should not be deleted without
>> inserting those files into the next tablet.
>>
>>   root@uno accumulo.metadata> addsplits -t foo '' abc
>>   root@uno foo> insert '' f q v3
>>   root@uno foo> scan
>>f:q []v3
>>   a f:q []v1
>>   b f:q []v2
>>
>> Now we see the empty end row tablet has a file.  This file would need
>> to be inserted into the next tablet before deleting.
>>
>>   root@uno foo> flush -t foo
>>   root@uno foo> scan -t accumulo.metadata -c file -r 2;
>>   2; file:hdfs://localhost:8020/accumulo/tables/2/t-015/F016.rf
>> []185,1
>>
>>
>> On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <ke...@deenlo.com> wrote:
>> > I replicated this locally and then did the following surgery to fix it.
>> >
>> > Create a table and determine its table id.  The table foo is table id
>> > 2.  This tableid is used for rows in metadata table.
>> >
>> >   root@uno> createtable foo
>> >   root@uno foo> tables -l
>> >   accumulo.metadata=>!0
>> >   accumulo.replication =>  +rep
>> >   accumulo.root=>+r
>> >   foo  => 2
>> >   trace=> 1
>> >
>> > Add the empty split and another split.
>> >
>> >   root@uno foo> addsplits -t foo '' abc
>> >
>> > Inpect the metdata table.
>> >
>> >   root@uno foo> scan -t accumulo.metadata -b 2;
>> >   2; loc:1000174d243 []localhost:9997
>> >   2; srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-005
>> >   2; srv:lock []
>> > tservers/localhost:9997/zlock-00$1000174d243
>> >   2; srv:time []M0
>> >   2; ~tab:~pr []\x00
>> >   2;abc loc:1000174d243 []localhost:9997
>> >   2;abc srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-006
>> >   2;abc srv:lock []
>> > tservers/localhost:9997/zlock-00$1000174d243
>> >   2;abc srv:time []M0
>> >   2;abc ~tab:~pr []\x01
>> >   2< loc:1000174d243 []localhost:9997
>> >   2< srv:dir []
>> > hdfs://localhost:8020/accumulo/tables/2/default_tablet
>> >   2< srv:lock []
>>

Re: Adding empty string split causes consistency check to failed for merge command

[Keith Turner]

I omitted something in with my previous email. I assumed the the
tablet with an empty row could have no data.  However, it can have
data.  If the tablet has files, they should not be deleted without
inserting those files into the next tablet.

  root@uno accumulo.metadata> addsplits -t foo '' abc
  root@uno foo> insert '' f q v3
  root@uno foo> scan
   f:q []v3
  a f:q []v1
  b f:q []v2

Now we see the empty end row tablet has a file.  This file would need
to be inserted into the next tablet before deleting.

  root@uno foo> flush -t foo
  root@uno foo> scan -t accumulo.metadata -c file -r 2;
  2; file:hdfs://localhost:8020/accumulo/tables/2/t-015/F016.rf
[]185,1


On Mon, Mar 5, 2018 at 12:27 PM, Keith Turner <ke...@deenlo.com> wrote:
> I replicated this locally and then did the following surgery to fix it.
>
> Create a table and determine its table id.  The table foo is table id
> 2.  This tableid is used for rows in metadata table.
>
>   root@uno> createtable foo
>   root@uno foo> tables -l
>   accumulo.metadata=>!0
>   accumulo.replication =>  +rep
>   accumulo.root=>+r
>   foo  => 2
>   trace=> 1
>
> Add the empty split and another split.
>
>   root@uno foo> addsplits -t foo '' abc
>
> Inpect the metdata table.
>
>   root@uno foo> scan -t accumulo.metadata -b 2;
>   2; loc:1000174d243 []localhost:9997
>   2; srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-005
>   2; srv:lock []tservers/localhost:9997/zlock-00$1000174d243
>   2; srv:time []M0
>   2; ~tab:~pr []\x00
>   2;abc loc:1000174d243 []localhost:9997
>   2;abc srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-006
>   2;abc srv:lock []
> tservers/localhost:9997/zlock-00$1000174d243
>   2;abc srv:time []M0
>   2;abc ~tab:~pr []\x01
>   2< loc:1000174d243 []localhost:9997
>   2< srv:dir []hdfs://localhost:8020/accumulo/tables/2/default_tablet
>   2< srv:lock []tservers/localhost:9997/zlock-00$1000174d243
>   2< srv:time []M0
>   2< ~tab:~pr []\x01abc
>
> Insert some data
>
>   root@uno foo> insert a f q v1
>   root@uno foo> insert b f q v2
>
> Offline table and grant write permission to metadata.
>
>   root@uno foo> offline -t foo -w
>   root@uno foo> grant Table.WRITE -u root -t accumulo.metadata
>
> Delete the tablet with an empty end row
>
>   root@uno foo> deletemany -r 2; -t accumulo.metadata
>   Delete { 2; srv:dir [] } ? y
>   [DELETED] 2; srv:dir []
>   Delete { 2; srv:lock [] } ? y
>   [DELETED] 2; srv:lock []
>   Delete { 2; srv:time [] } ? y
>   [DELETED] 2; srv:time []
>   Delete { 2; ~tab:~pr [] } ? y
>   [DELETED] 2; ~tab:~pr []
>
> Fix the tablet with end row "abc" to indicate no previous tablet
>
>   root@uno foo> table accumulo.metadata
>   root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00
>
> Bring table online and scan it
>
>   root@uno accumulo.metadata> online foo
>   root@uno accumulo.metadata> scan -t foo
>   a f:q []v1
>   b f:q []v2
>
> Remove write permission to metadata table.
>
>   root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
>
> On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou <dzho...@gmail.com> wrote:
>> Hi all,
>>
>> I noticed that adding an empty string to as a split will cause merge command
>> failed the consistency check.
>>
>> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441
>> unassigned 442 v.unassigned 441 verify.total 441
>> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check
>> failed 1b<<
>>
>> Also, is there a way to remove an unwanted split from table. Merge is one
>> way, but with empty string does not work.
>>
>> Cheers,
>>
>> -Dong Zhou

Re: Adding empty string split causes consistency check to failed for merge command

[Keith Turner]

I replicated this locally and then did the following surgery to fix it.

Create a table and determine its table id.  The table foo is table id
2.  This tableid is used for rows in metadata table.

  root@uno> createtable foo
  root@uno foo> tables -l
  accumulo.metadata=>!0
  accumulo.replication =>  +rep
  accumulo.root=>+r
  foo  => 2
  trace=> 1

Add the empty split and another split.

  root@uno foo> addsplits -t foo '' abc

Inpect the metdata table.

  root@uno foo> scan -t accumulo.metadata -b 2;
  2; loc:1000174d243 []localhost:9997
  2; srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-005
  2; srv:lock []tservers/localhost:9997/zlock-00$1000174d243
  2; srv:time []M0
  2; ~tab:~pr []\x00
  2;abc loc:1000174d243 []localhost:9997
  2;abc srv:dir []hdfs://localhost:8020/accumulo/tables/2/t-006
  2;abc srv:lock []tservers/localhost:9997/zlock-00$1000174d243
  2;abc srv:time []M0
  2;abc ~tab:~pr []\x01
  2< loc:1000174d243 []localhost:9997
  2< srv:dir []hdfs://localhost:8020/accumulo/tables/2/default_tablet
  2< srv:lock []tservers/localhost:9997/zlock-00$1000174d243
  2< srv:time []M0
  2< ~tab:~pr []\x01abc

Insert some data

  root@uno foo> insert a f q v1
  root@uno foo> insert b f q v2

Offline table and grant write permission to metadata.

  root@uno foo> offline -t foo -w
  root@uno foo> grant Table.WRITE -u root -t accumulo.metadata

Delete the tablet with an empty end row

  root@uno foo> deletemany -r 2; -t accumulo.metadata
  Delete { 2; srv:dir [] } ? y
  [DELETED] 2; srv:dir []
  Delete { 2; srv:lock [] } ? y
  [DELETED] 2; srv:lock []
  Delete { 2; srv:time [] } ? y
  [DELETED] 2; srv:time []
  Delete { 2; ~tab:~pr [] } ? y
  [DELETED] 2; ~tab:~pr []

Fix the tablet with end row "abc" to indicate no previous tablet

  root@uno foo> table accumulo.metadata
  root@uno accumulo.metadata> insert 2;abc ~tab ~pr \x00

Bring table online and scan it

  root@uno accumulo.metadata> online foo
  root@uno accumulo.metadata> scan -t foo
  a f:q []v1
  b f:q []v2

Remove write permission to metadata table.

  root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata

On Thu, Mar 1, 2018 at 8:46 PM, Dong Zhou  wrote:
> Hi all,
>
> I noticed that adding an empty string to as a split will cause merge command
> failed the consistency check.
>
> 2018-03-01 19:58:46,724 [state.MergeStats] DEBUG: chopped 442 v.chopped 441
> unassigned 442 v.unassigned 441 verify.total 441
> 2018-03-01 19:58:46,724 [state.MergeStats] INFO : Merge consistency check
> failed 1b<<
>
> Also, is there a way to remove an unwanted split from table. Merge is one
> way, but with empty string does not work.
>
> Cheers,
>
> -Dong Zhou

[ANNOUNCE] Apache Fluo 1.2.0 released

[Keith Turner]

The Apache Fluo team is happy to announce the release of Fluo 1.2.0:

  http://fluo.apache.org/release/fluo-1.2.0/

We would like to thank everyone who contributed to this release for their time
and effort.

Apache Fluo is a distributed processing system built on Apache Accumulo.
Fluo users can easily setup workflows that execute cross node transactions
when data changes. These workflows enable users to continuously join new
data into large existing data sets with low latency while avoiding
reprocessing all data.

In this release, Fluo's YARN code was moved to a separate project called
fluo-yarn, which will release shortly. This release needed to happen first.

Re: Failed to assign map files to tablets during Bulk Import

[Keith Turner]

On Fri, Feb 9, 2018 at 5:13 PM, Dong Zhou <dz...@phemi.com> wrote:
> The bulk import operation did finish, but it was marked as failed.
> The failed to import rfile was moved into the failure directory.
>
> We tried manually to import this rfile using importdirectory, and then the
> exact same outcome occurred and error log appeared in the tserver log file.

Can you try running : accumulo rfile-info 

I am curious if that command can read the file.


>
>
> On Fri, Feb 9, 2018 at 1:46 PM Keith Turner <ke...@deenlo.com> wrote:
>>
>> On Fri, Feb 9, 2018 at 2:26 PM, Dong Zhou <dz...@phemi.com> wrote:
>> > Hi All,
>> >
>> > We were trying to write some data into an Accumulo table that contains
>> > roughly 3.7 Trillion entries using Bulk Import.
>> >
>> > Our code generates RFiles in distributed fashion, and most of them go
>> > into
>> > the table perfectly, excepting for 1 RFile.
>> >
>> > The bulk import failed with following error message. Please note that I
>> > have
>> > shortened the error message and also masked out some of the data.
>> >
>> >
>>
>> This message is from an intermediate tserver.  It possible this
>> intermediate thread is running after the bulk import completed, more
>> on this below.
>>
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
>> > exception in assignMapFiles.
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport
>> > at
>> >
>> > org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
>> > at
>> > org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
>> > at
>> >
>> > org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
>> > at
>> >
>> > org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
>> > at
>> >
>> > org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
>> > at
>> >
>> > org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
>> > at
>> >
>> > org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
>> > at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at
>> >
>> > org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> > at java.lang.Thread.run(Thread.java:745)
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
>> > map
>> > files to tablet o;SESSION_ID|;SESSION_ID| because :
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport .  Will retry ...
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
>> > map
>> > files to tablet o;SESSION_ID|;SESSION_ID| because :
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport .  Will retry ...
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
>> > map
>> > files to tablet o;SESSION_ID|;SESSION_ID| because :
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport .  Will retry ...
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
>> > map
>> > files to tablet o;SESSION_ID|;SESSION_ID| because :
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport .  Will retry ...
>> > 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1
>> > map
>> > files to tablet o;SESSION_ID|;SESSION_ID| because :
>> > org.apache.thrift.TApplicationException: Internal error processing
>> > bulkImport .  Will retry ...
>> > ...
>> > ...
>> > ...
>>
>> Each bulk import has a transaction id in zookeeper.  When bulk import
>> finishes, it removes this

Re: Failed to assign map files to tablets during Bulk Import

[Keith Turner]

On Fri, Feb 9, 2018 at 2:26 PM, Dong Zhou  wrote:
> Hi All,
>
> We were trying to write some data into an Accumulo table that contains
> roughly 3.7 Trillion entries using Bulk Import.
>
> Our code generates RFiles in distributed fashion, and most of them go into
> the table perfectly, excepting for 1 RFile.
>
> The bulk import failed with following error message. Please note that I have
> shortened the error message and also masked out some of the data.
>
>

This message is from an intermediate tserver.  It possible this
intermediate thread is running after the bulk import completed, more
on this below.

> 2018-02-05 21:47:07,244 [client.BulkImporter] ERROR: Encountered unknown
> exception in assignMapFiles.
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.recv_bulkImport(TabletClientService.java:594)
> at
> org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.bulkImport(TabletClientService.java:577)
> at
> org.apache.accumulo.server.client.BulkImporter.assignMapFiles(BulkImporter.java:600)
> at
> org.apache.accumulo.server.client.BulkImporter.access$400(BulkImporter.java:77)
> at
> org.apache.accumulo.server.client.BulkImporter$AssignmentTask.run(BulkImporter.java:479)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|;SESSION_ID| because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|;SESSION_ID| because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|;SESSION_ID| because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|;SESSION_ID| because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> 2018-02-05 21:47:07,244 [client.BulkImporter] INFO : Could not assign 1 map
> files to tablet o;SESSION_ID|;SESSION_ID| because :
> org.apache.thrift.TApplicationException: Internal error processing
> bulkImport .  Will retry ...
> ...
> ...
> ...

Each bulk import has a transaction id in zookeeper.  When bulk import
finishes, it removes this transaction id from zookeeper (this is a
step in a bulk import FATE tx).  Writes are made to the metadata table
as part of bulk import. The metadata table has a constraint sanity
check that ensure the bulk import is active.  The exception below is
this check failing.

Below is the code that removes this Id from ZK.

https://github.com/apache/accumulo/blob/rel/1.8.1/server/master/src/main/java/org/apache/accumulo/master/tableOps/CompleteBulkImport.java#L42

Bulk import spawns work threads on intermediate tablet servers to
inspect rfiles indexes  to determine where the files should go.  These
intermediate tablet servers instruct other tablet servers to load
files.  The error message below is from one of these 2nd hop tservers.
Its possible that this spawned work thread is still running after the
bulk import has completed and this message could be ignored.  Did the
bulk import operation fail of succeed from the Accumulo client
perspective?   Its also possible that this error message is unrelated
to the reason the bulk import failed.  Was there one file in the fail
dir?

> 2018-02-05 21:47:07,494 [impl.Writer] ERROR: error sending update to
> ::
> ConstraintViolationException(violationSummaries:[TConstraintViolationSummary(constrainClass:org.apache.accumulo.server.constraints.MetadataConstraints,
> violationCode:8, violationDescription:Bulk load transaction no longer
> running, numberOfViolatingMutations:1)])
> 2018-02-05 21:47:07,494 [util.MetadataTableUtil] ERROR: null
> 

Re: Triggers or their equivalent

[Keith Turner]

On Wed, Dec 20, 2017 at 12:32 PM, Josh Elser  wrote:
> Accumulo doesn't provide any mechanism to build a traditional trigger like a
> traditional RDBMS or HBase. Can you share more details about what exactly
> you want this trigger to do? It's possible we can suggest an alternate
> approach which would fit naturally.
>
> If your trigger is causing other updates to Accumulo or creating some remote
> I/O operation, the general recommendation would be to handle this at a
> higher-level instead of in Accumulo itself. For generating "follow-on"
> updates in Accumulo from a single update, Apache Fluo is a good starting
> point. There are many other tools which you could use to build your own kind
> of system, but this is very dependent on your actual requirements.

Fluo is a good solution if you would like to join existing data with
new data and update a query table based on the results of the joins.
If no joins are required, something like Kafka may be a good solution.
Insert new data into Kafka and have a consumer group that inserts into
Accumulo.  Have other consumer groups for the actions you want to
Accumulo triggers to take.


>
>
> On 12/20/17 11:43 AM, Geoffry Roberts wrote:
>>
>> I have need of a trigger mechanism in Accumulo.  My first impulse was to
>> use an iterator, but the documentation advised against this.  I think I can
>> see why.
>>
>> Is there a best practice way of accomplishing this?  Is there any plan to
>> add such functionality in the future?
>>
>> --
>> There are ways and there are ways,
>>
>> Geoffry Roberts

Re: Major Compactions

[Keith Turner]

The command to run in the shell is "compact --cancel -t table"

On Mon, Dec 11, 2017 at 7:41 PM, Jeff Downton  wrote:
> Hi All,
>
> Is it possible to stop a major compaction once it has begun (Accumulo
> 1.7.0)?
>
> Manually kicked one off on a table containing ~20k tablets,  which
> subsequently queued up ~20k major compactions (I'm assuming one for each
> tablet in the table).
>
> It's running slower than I'd like so I'm looking to defer running it till
> another time.  The purpose for the major compaction is to permanently remove
> deleted key-value pairs in the table.
>
> Thanks in advance for any help.
>
> -Jeff
>

Re: Accumulo as a Column Storage

[Keith Turner]

On Thu, Oct 19, 2017 at 9:05 PM, Christopher  wrote:
> There's no expected scaling issue with having each column qualifier in its
> own unique column family, regardless of how large the number of these
> becomes. I've ingested random data like this before for testing, and it
> works fine.
>
> However, there may be an issue trying to create a very large number of
> locality groups. Locality groups are named, and you must explicitly
> configure them to store particular column families. That configuration is
> typically stored in ZooKeeper, and the configuration storage (in ZooKeeper,
> and/or in your conf/accumulo-site.xml file) does not scale as well as the
> data storage (HDFS) does. Where, and how, it will break, is probably
> system-dependent and not directly known (at least, not known by me). I would
> expect dozens, and possibly hundreds, of locality groups to work okay, but
> thousands seems like it's too many (but I haven't tried).

Seeking to a random location is O(F*L), where F is the number of files
and L is the number of locality groups used.  So if a tablet had 10
files and 10 locality groups were being used, then a seek on the
tablet would result in 100 seeks at the lowest levels.

After the initial seek, scanning over locality groups uses a heap of
heaps.  A heap that select the min key from all files.  Within each
file there is a heap that selects the min key from each loc group.
So scanning is O(log2(F) * log2(L)) or O(log2(F) + log2(L)) not sure.

Scanning over lots locality groups is probably pretty efficient, but
doing lots of random seeks over lots of loc groups may not be.

>
>
> On Thu, Oct 19, 2017 at 6:47 PM Mohammad Kargar  wrote:
>>
>> That makes sense. So this means that there's no limit or concerns on
>> having, potentially,  large number of column families (holing only one
>> column qualifier), right?
>>
>> On Thu, Oct 19, 2017 at 3:06 PM, Josh Elser  wrote:
>>>
>>> Yup, that's the intended use case. You have the flexibility to determine
>>> what column families make sense to group together. Your only "cost" in
>>> changing your mind is the speed at which you can re-compact your data.
>>>
>>> There is one concern which comes to mind. Though making many locality
>>> groups does increase the speed at which you can read from specific columns,
>>> it decreases the speed at which you can read from _all_ columns. So, you can
>>> do this trick to make Accumulo act more like a columnar database, but beware
>>> that you're going to have an impact if you still have a use-case where you
>>> read more than just one or two columns at a time.
>>>
>>> Does that make sense?
>>>
>>>
>>> On 10/19/17 5:50 PM, Mohammad Kargar wrote:

 AFAIK in Accumulo we can use "locality groups" to group sets of columns
 together on disk which would make it more like  a column-oriented database.
 Considering that "locality groups" are per column family, I was wondering
 what if we treat column families like column qualifiers (creating one 
 column
 family per each qualifier) and assigning each to a different locality 
 group.
 This way all the data in a given column will be next to each other on disk
 which makes it easier for analytical applications to query the data.

 Any thoughts?

 Thanks,
 Mohammad

>>
>

Re: problems restoring backups

[Keith Turner]

Are you scanning with the correct authorizations?  May be filtering
everything out on the server side if not.

On Wed, Oct 11, 2017 at 3:05 AM, vLex Systems  wrote:
> Hi,
>
> I'm doing backups of an accumulo database following the procedure in
> the readme (clonetable, offline and exporttable). I distcp to s3 and
> also upload the exportMetaddata.zip to s3.
> I do the restore by downloading the files from s3, distcping them to
> hadoopfs and then doing the importtable.
>
>
> When I restore the tables in a new accumulo server, all tables but one
> are correctly restored. When I do a scan -t of that table, the shell
> hangs approximately for a minute and it returns nothing.
>
> According to the accumulo web interface the table has data. How can I
> debug what's happening with that table?

[ANNOUCE] Apache Fluo is a TLP

[Keith Turner]

I am excited to share that Apache Fluo is now a top level project at the ASF.

  
https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces5
  http://fluo.apache.org/blog/2017/07/26/fluo-graduates-from-apache-incubator/

Keith

Re: maximize usage of cluster resources during ingestion

[Keith Turner]

On Thu, Jul 13, 2017 at 10:56 AM,   wrote:
> Regarding the referenced paper, pre-splitting the tables, using an optimized 
> zookeeper deployment, and increasing concurrent minor / major compactions are 
> good things. I'm not sure that we want to recommend turning off the write 
> ahead logs and replication for production deployments.


I wouldn't recommend turning off write ahead logs either.   However,
it can be useful to turn them off for performance testing to
understand their impact.

I noticed you set "table.durability": "flush" which is good for
performance.  However the metdata table may still be set to sync which
can cause performance problems.   The following article describes how
to set the metdata table to flush. The article also describes the
consequences for regular tables of having the metadata table set to
sync.

http://accumulo.apache.org/blog/2016/11/02/durability-performance.html

>
> -Original Message-
> From: Jeremy Kepner [mailto:kep...@ll.mit.edu]
> Sent: Thursday, July 13, 2017 10:05 AM
> To: user@accumulo.apache.org
> Subject: Re: maximize usage of cluster resources during ingestion
>
> https://arxiv.org/abs/1406.4923  contains a number of tricks for maximizing 
> ingest performance.
>
> On Thu, Jul 13, 2017 at 08:13:40AM -0400, Jonathan Wonders wrote:
>> Keep in mind that Accumulo puts a much different kind of load on HDFS
>> than the DFSIO benchmark.  It might be more appropriate to use a tool
>> like dstat to monitor HDD utilization and queue depth.  HDD throughput
>> benchmarks usually will involve high queue depths as disks are much
>> more effective when they can pipeline and batch updates. Accumulo's
>> WAL workload will typically call hflush or hsync periodically which
>> interrupts the IO pipeline much like memory barriers can interrupt CPU
>> pipelining except more severe.  This is necessary to provide
>> durability guarantees, but definitely comes at a cost to throughput.
>> Any database that has these durability guarantees will suffer
>> similarly to an extent.  For Accumulo, it is probably worse than for
>> non-distributed databases because the flush or sync must happen at
>> each replica prior to the mutation being added into the in-memory map.
>>
>> I think one of the reasons the recommendation was made to add more
>> tablet servers is because each tablet server only writes to one WAL at
>> a time and each block will live on N disk based on replication factor.
>> If you have a replication factor of 3, there will be 10x3 blocks being
>> appended to at any given time (excluding compactions).  Since you have
>> 120 disks, not all will be participating in write-ahead-logging, so
>> you should not count the IO capacity of these extra disks towards
>> expected ingest throughput.  10 tablet servers per node is probably
>> too many because there would likely be a lot of contention
>> flushing/syncing WALs.  I'm not sure how smart HDFS is about how it
>> distributes the WAL load.  You might see more benefit with 2-4
>> tservers per node.  This would mostly likely require more batch writer 
>> threads in the client as well.
>>
>> I'm not too surprised that snappy did not help because the WALs are
>> not compressed and are likely a bigger bottleneck than compaction
>> since you have many disks not participating in WAL.
>>
>>
>> On Wed, Jul 12, 2017 at 11:16 AM, Josh Elser  wrote:
>>
>> > You probably want to split the table further than just 4 tablets per
>> > tablet server. Try 10's of tablets per server.
>> >
>> > Also, merging the content from (who I assume is) your coworker on
>> > this stackoverflow post[1], I don't believe the suggestion[2] to
>> > verify WAL max size, minc threshold, and native maps size was brought up 
>> > yet.
>> >
>> > Also, did you look at the JVM GC logs for the TabletServers like was
>> > previously suggested to you?
>> >
>> > [1] https://stackoverflow.com/questions/44928354/accumulo-tablet
>> > -server-doesnt-utilize-all-available-resources-on-host-machine/
>> > [2] https://accumulo.apache.org/1.8/accumulo_user_manual.html#_n
>> > ative_maps_configuration
>> >
>> > On 7/12/17 10:12 AM, Massimilian Mattetti wrote:
>> >
>> >> Hi all,
>> >>
>> >> I ran a few experiments in the last days trying to identify what is
>> >> the bottleneck for the ingestion process.
>> >> - Running 10 tservers per node instead of only one gave me a very
>> >> neglectable performance improvement of about 15%.
>> >> - Running the ingestor processes from the two masters give the same
>> >> performance as running one ingestor process in each tablet server
>> >> (10
>> >> ingestors)
>> >> - neither the network limit (10 Gb network) nor the disk throughput
>> >> limit has been reached (1GB/s per node reached while running the
>> >> TestDFSIO benchmark on HDFS)
>> >> - CPU is always around 20% on each tserver
>> >> - changing compression from GZ to snappy did not provide any
>> >> benefit
>> >> - increasing the 

Re: New blog post

[Keith Turner]

The edit link that automagically creates a PR is really neat.

On Fri, Jun 30, 2017 at 12:26 PM, Mike Walch  wrote:
> I wrote a blog post about improvements that I made to the Accumulo
> documentation for 2.0:
>
> https://accumulo.apache.org/blog/2017/06/29/accumulo-documentation-improvements.html
>

[ANNOUNCE] Apache Fluo Recipes 1.1.0-incubating released

[Keith Turner]

The Apache Fluo (incubating) team is happy to announce the release of
Fluo Recipes 1.1.0-incubating:

https://fluo.apache.org/release/fluo-recipes-1.1.0-incubating/

This release includes significant API improvements that make writing
applications faster and easier.

Thanks,
The Apache Fluo team

=

*Disclaimer*

Apache Fluo is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
required of all newly accepted projects until a further review indicates
that the infrastructure, communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects. While
incubation status is not necessarily a reflection of the completeness or
stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.

Re: 'logs/' dir in the accumulo package

[Keith Turner]

On Mon, Jun 26, 2017 at 4:52 PM, Srikanth Viswanathan
 wrote:
> Thanks, Josh and Mike.
>
> This isn't causing any issues per se. We wanted to `chmod` the dir to be
> publicly readable (for log aggregation), so there were questions around
> where exactly to put the `chmod` call. I was then surprised to find that the
> directory already existed. Do you recommend we rely on the directory already
> existing after the distribution is untarred?

Could use mkdir -p $logdir after untar.

>
> Thanks,
> Srikanth
>
> On Mon, Jun 26, 2017 at 4:50 PM, Michael Wall  wrote:
>>
>> Srikanth,
>>
>> Like Josh said, I don't think we make that guarantee.  But it is part of
>> the maven assembly
>> (https://github.com/apache/accumulo/blob/rel/1.8.1/assemble/src/main/assemblies/component.xml#L91)
>> so I would expect the directory to continue to be there.  Is that causing
>> you issues?
>>
>>
>> On Mon, Jun 26, 2017 at 4:46 PM Josh Elser  wrote:
>>>
>>> Srikanth,
>>>
>>> I just checked 1.7.3 and 1.8.0, both of which have the logs/ directory
>>> included in the bin-tarball. It isn't a change in packaging -- it's been
>>> like this for some time.
>>>
>>> I don't expect that we would provide any guarantees about the presence
>>> of this directory (but I don't know why we would chose to omit it,
>>> either).
>>>
>>> - Josh
>>>
>>> On 6/26/17 4:27 PM, Srikanth Viswanathan wrote:
>>> > Hi all,
>>> >
>>> > Kind of a silly question, but I noticed the `logs/` directory being
>>> > packaged with the accumulo binary release. Do you mean to package this
>>> > and do you guarantee this will be packaged in future releases?
>>> >
>>> > $ tar -tvf accumulo-1.8.1-bin.tar.gz |grep logs
>>> > drwx-- mjwall/mjwall 0 2017-02-10 12:50 accumulo-1.8.1/logs/
>>> >
>>> >
>>> > Thanks,
>>> > Srikanth
>
>

Blog post about running Accumulo

[Keith Turner]

Mike Walch wrote a blog post about running Accumulo :

http://accumulo.apache.org/blog/2017/04/21/introducing-uno-and-muchos.html
https://twitter.com/ApacheAccumulo/status/855493002709659648

Re: Batchscanner - error closing output stream because stream is closed

[Keith Turner]

On Wed, Apr 19, 2017 at 12:38 PM, David Boyd  wrote:
> All:
>
>I am getting this stack trace periodically based on no pattern I can
> determine from my application.

If you are seeing this regularly, try to enable Accumulo debug
logging.  That may allow you see messages like [1] which may help
understand the cause.

https://github.com/apache/accumulo/blob/rel/1.7.2/core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java#L689

>
> Is this a message I should be worried about?
>
> What is a technique to trace this back to my code and the cause?
>
> Obviously something is closing things before the thread closes it.
>
>
>> 2017-04-19 12:33:53,423 |  WARN | [batch scanner 19824- 8 looking up 1
>> ranges at accumulodev:9997] | (TIOStreamTransport.java:112) - Error closing
>> output stream.
>> java.io.IOException: The stream is closed
>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>> at
>> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
>> at
>> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)
>> at
>> org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686)
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349)
>> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> at java.lang.Thread.run(Thread.java:745)
>
>
>
> --
> = mailto:db...@incadencecorp.com 
> David W. Boyd
> VP,  Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office:   +1-703-552-2862
> cell: +1-703-402-7908
> == http://www.incadencecorp.com/ 
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
> Board Member- USSTEM Foundation - www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited.  If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>

Re: Batchscanner - error closing output stream because stream is closed

[Keith Turner]

One possible cause of this is a communication error with a remote
tablet server.  For example if the thrift call at
TabletServerBatchReaderIterator.java line 665 [1] failed with a comm
error, then it would try to close the connection at line 686.  Then
when the close fails (because its a busted socket), it seems that
thrift logs a warn[2].  I think the batch scanner will retry in this
case.  I suspect line 665 would throw a TTransportException which
would cause a retry.   Do you know if the batch scanner continued to
operate?  Did any exceptions propagate to code using the batch
scanner?

[1]: 
https://github.com/apache/accumulo/blob/rel/1.7.2/core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java#L665
[2]: 
https://github.com/apache/thrift/blob/0.9.1/lib/java/src/org/apache/thrift/transport/TIOStreamTransport.java#L112

On Wed, Apr 19, 2017 at 12:38 PM, David Boyd  wrote:
> All:
>
>I am getting this stack trace periodically based on no pattern I can
> determine from my application.
>
> Is this a message I should be worried about?
>
> What is a technique to trace this back to my code and the cause?
>
> Obviously something is closing things before the thread closes it.
>
>
>> 2017-04-19 12:33:53,423 |  WARN | [batch scanner 19824- 8 looking up 1
>> ranges at accumulodev:9997] | (TIOStreamTransport.java:112) - Error closing
>> output stream.
>> java.io.IOException: The stream is closed
>> at
>> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
>> at
>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
>> at
>> org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110)
>> at
>> org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89)
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309)
>> at
>> org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571)
>> at
>> org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151)
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686)
>> at
>> org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349)
>> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>> at java.lang.Thread.run(Thread.java:745)
>
>
>
> --
> = mailto:db...@incadencecorp.com 
> David W. Boyd
> VP,  Data Solutions
> 10432 Balls Ford, Suite 240
> Manassas, VA 20109
> office:   +1-703-552-2862
> cell: +1-703-402-7908
> == http://www.incadencecorp.com/ 
> ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
> Chair ANSI/INCITS TC Big Data
> Co-chair NIST Big Data Public Working Group Reference Architecture
> First Robotic Mentor - FRC, FTC - www.iliterobotics.org
> Board Member- USSTEM Foundation - www.usstem.org
>
> The information contained in this message may be privileged
> and/or confidential and protected from disclosure.
> If the reader of this message is not the intended recipient
> or an employee or agent responsible for delivering this message
> to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication
> is strictly prohibited.  If you have received this communication
> in error, please notify the sender immediately by replying to
> this message and deleting the material from any computer.
>
>

Re: tserver.compaction.*.concurrent.max behavior in Accumulo 1.8.1

[Keith Turner]

On Thu, Mar 23, 2017 at 12:37 PM, Massimilian Mattetti
 wrote:
> I just tested the ingestion with 10 tablets per Server and now the system
> achieve up to 24 concurrent major compactions.
>
> I have another issue with the tserver.memory.maps.max property. I am play
> with the size of the native map to track how its size affect the ingestion
> performance. I set the values of table.compaction.minor.logs.threshold and
> tserver.walog.max.size big enough so I can increase the map size up to 40GB
> without problems. At the beginning changing the size of the map (using the
> shell) didn't bring any benefit to the ingestion process. Monitoring the
> servers I noticed that their RAM consumption was constant. Eventually I
> tried to restart the tablet servers after changing the
> tserver.memory.maps.max and the RAM usage on each node increased as I
> expected. From the documentation I know that any change to the
> tserver.memory.maps.max should take effect without restarting the tablet
> servers, is that always true? (tserver.memory.maps.native.enabled has always
> been true and from the logs I see that the shared library has been correctly
> loaded).
> Thanks!

I took a quick look at the 1.6.5 code and it does not seem to update
after tserver start in that version.

>
> Best Regards,
> Max
>
>
>
> From:Michael Wall 
> To:user@accumulo.apache.org
> Date:23/03/2017 17:49
>
> Subject:Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> 
>
>
>
> Yes, until you hit another constraint like Marc and Dave were asking about.
>
> Mike
>
> On Thu, Mar 23, 2017 at 11:34 AM Massimilian Mattetti 
> wrote:
> I wasn't aware of such constrain. So I can just increase the number of
> tablets per server and it will perform more major compactions.
> Thanks,
> Max
>
>
>
>
> From:Michael Wall 
> To:user@accumulo.apache.org
> Date:23/03/2017 17:09
> Subject:Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> 
>
>
>
> Max,
>
> So your max major compactions will be 3 per tablet server.  Accumulo will
> not run 2 majors on the same tablet concurrently.
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:37 AM Massimilian Mattetti 
> wrote:
> One table, 9 pre-split tablets, 3 tablets per server and the data is uniform
> distributed among each tablet.
> Max
>
>
>
>
> From:Michael Wall 
> To:user@accumulo.apache.org
> Date:23/03/2017 16:28
> Subject:Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> 
>
>
>
> Max,
>
> On you 3 node cluster, how many tables are you ingesting into?  How many
> tablets are in each table?  Are the tablets equally spread amongst the 3
> tablet servers?
>
> Mike
>
> On Thu, Mar 23, 2017 at 10:13 AM Massimilian Mattetti 
> wrote:
> With the configuration I presented before the concurrent major compactions
> are never more than 3 per tablet server while the minor are under the 4 per
> node. Can one of the other configurations be the cause of this behavior?
>
> Regards,
> Max
>
>
>
> From:Dave Marion 
> To:user@accumulo.apache.org, Massimilian Mattetti/Haifa/IBM@IBMIL
> Date:23/03/2017 14:55
> Subject:Re: tserver.compaction.*.concurrent.max behavior in Accumulo
> 1.8.1
> 
>
>
>
> Can you explain more what you mean by "My problem is that both the minor and
> major compactions do not overcome their default max values?" I have done
> some testing with 1.8.1 and specifically modifying
> tserver.compaction.major.concurrent.max to a higher number and I have seen
> it take effect.
>
> On March 23, 2017 at 7:54 AM Massimilian Mattetti 
> wrote:
>
> Hi All,
>
> I am running some heavy ingestion process on a 3 nodes cluster of Accumulo
> 1.8.1, using the following configuration:
>
> table.compaction.minor.logs.threshold=10
> table.durability=flush
> table.file.max=30
>
> tserver.wal.blocksize=2G
> tserver.walog.max.size=4G
> tserver.mutation.queue.max=2M
> tserver.memory.maps.max=4G
> tserver.compaction.minor.concurrent.max=50
> tserver.compaction.major.concurrent.max=8
>
> My problem is that both the minor and major compactions do not overcome
> their default max values. I checked the config from the shell and it looks
> fine to me:
>
> default| tserver.compaction.major.concurrent.max  | 3
> system| @override
> ... | 8
>
> default| tserver.compaction.minor.concurrent.max ... | 4
> system| @override
> ... | 50
>
> Is something changed from 1.8.0? I haven't seen such behavior with the
> previous 

Re: Fix "Table x has a hole" [SEC=UNOFFICIAL]

[Keith Turner]

On Wed, Mar 1, 2017 at 12:23 AM, Dickson, Matt MR
<matt.dick...@defence.gov.au> wrote:
> UNOFFICIAL
>
> Thanks for that Keith,
>
> That's got it working again.  As for the cause, I had an error in the logs 
> stating a tablet was hosted and assigned.  I've always removed the referenced 
> tablet in the metadata table to fix this and had no issues in the past.  It 
> looks like I fat fingered the deletion which removed the wrong entry so not 
> an issue with Accumulo.

Hosted and assigned, that's not good. What version of Accumulo are you
running?  Do you have any ideas on what may cause this? How often do
you see this?  Is there any event that usually precedes this?
>
> Thanks.
>
> -----Original Message-
> From: Keith Turner [mailto:ke...@deenlo.com]
> Sent: Wednesday, 1 March 2017 03:36
> To: user@accumulo.apache.org
> Subject: Re: Fix "Table x has a hole" [SEC=UNOFFICIAL]
>
> Below are some commands that show how to recreate this problem and how
> to fix it.   Each table in the metadata table has a pointer to the
> previous tablets.  Adding and removing splits to a table changes this.
>
>   root@uno> createtable test
>
> Get the tables ID below we will need it later.
>
>   root@uno test> tables -l
>   accumulo.metadata=>!0
>   accumulo.replication =>  +rep
>   accumulo.root=>+r
>   test => 3
>   trace=> 1
>
> Add some splits and then scan the metadata table.  The pointers to the 
> previous tablet are in the ~tab:~pr column.  The scan below uses the table id 
> above.
>
>   root@uno test> addsplits  333
>   root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
>   3; ~tab:~pr []\x00
>   3;333 ~tab:~pr []\x01
>   3< ~tab:~pr []\x01333
>
> Add another split and rescan the metadata table.
>
>   root@uno test> addsplits 222
>   root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
>   3; ~tab:~pr []\x00
>   3;222 ~tab:~pr []\x01
>   3;333 ~tab:~pr []\x01222
>   3< ~tab:~pr []\x01333
>
> Grant permission to write to the metadata table and then recreate the problem 
> you have.
>
>   root@uno test> grant Table.WRITE -u root -t accumulo.metadata
>   root@uno test> table accumulo.metadata
>   root@uno accumulo.metadata> insert 3;333 ~tab ~pr \x01
>   root@uno accumulo.metadata> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 
> 3<
>   3; ~tab:~pr []\x00
>   3;222 ~tab:~pr []\x01
>   3;333 ~tab:~pr []\x01
>   3< ~tab:~pr []\x01333
>
> If you ran check for metadata problems here, should see the error message you 
> saw.  Below, the pointer is fixed and write permission is revoked (to prevent 
> accidental writes in the future).
>
>   root@uno accumulo.metadata> insert 3;333 ~tab ~pr \x01222
>   root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
>   root@uno accumulo.metadata>
>
> After running the command above to fix the potiner, check for metadata 
> problems should be happy.
>
> It would be nice to try to track down the cause of this.  Spliting a tablet 
> involves three metadata operations.  For fault tolerance, the columns 
> ~tab:oldprevrow and ~tab:splitRatio are temporarily written.
> If a tablet server dies in the middle of splitting a tablet, then Accumulo 
> will see these temporary columns and attempt to continue the split.  So I am 
> curious if you see these columns?
>
> On Sun, Feb 26, 2017 at 6:49 PM, Dickson, Matt MR 
> <matt.dick...@defence.gov.au> wrote:
>> UNOFFICIAL
>>
>> Running the CheckForMetadataProblems on Accumulo is listing
>>
>> Table xxx has a hole  != 222
>>
>> Is there a correct way to repair this?
>>
>> Thanks in advance.

Re: Fix "Table x has a hole" [SEC=UNOFFICIAL]

[Keith Turner]

Below are some commands that show how to recreate this problem and how
to fix it.   Each table in the metadata table has a pointer to the
previous tablets.  Adding and removing splits to a table changes this.

  root@uno> createtable test

Get the tables ID below we will need it later.

  root@uno test> tables -l
  accumulo.metadata=>!0
  accumulo.replication =>  +rep
  accumulo.root=>+r
  test => 3
  trace=> 1

Add some splits and then scan the metadata table.  The pointers to the
previous tablet are in the ~tab:~pr column.  The scan below uses the
table id above.

  root@uno test> addsplits  333
  root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3; ~tab:~pr []\x00
  3;333 ~tab:~pr []\x01
  3< ~tab:~pr []\x01333

Add another split and rescan the metadata table.

  root@uno test> addsplits 222
  root@uno test> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3; ~tab:~pr []\x00
  3;222 ~tab:~pr []\x01
  3;333 ~tab:~pr []\x01222
  3< ~tab:~pr []\x01333

Grant permission to write to the metadata table and then recreate the
problem you have.

  root@uno test> grant Table.WRITE -u root -t accumulo.metadata
  root@uno test> table accumulo.metadata
  root@uno accumulo.metadata> insert 3;333 ~tab ~pr \x01
  root@uno accumulo.metadata> scan -t accumulo.metadata -c ~tab:~pr -b 3; -e 3<
  3; ~tab:~pr []\x00
  3;222 ~tab:~pr []\x01
  3;333 ~tab:~pr []\x01
  3< ~tab:~pr []\x01333

If you ran check for metadata problems here, should see the error
message you saw.  Below, the pointer is fixed and write permission is
revoked (to prevent accidental writes in the future).

  root@uno accumulo.metadata> insert 3;333 ~tab ~pr \x01222
  root@uno accumulo.metadata> revoke Table.WRITE -u root -t accumulo.metadata
  root@uno accumulo.metadata>

After running the command above to fix the potiner, check for metadata
problems should be happy.

It would be nice to try to track down the cause of this.  Spliting a
tablet involves three metadata operations.  For fault tolerance, the
columns ~tab:oldprevrow and ~tab:splitRatio are temporarily written.
If a tablet server dies in the middle of splitting a tablet, then
Accumulo will see these temporary columns and attempt to continue the
split.  So I am curious if you see these columns?

On Sun, Feb 26, 2017 at 6:49 PM, Dickson, Matt MR
 wrote:
> UNOFFICIAL
>
> Running the CheckForMetadataProblems on Accumulo is listing
>
> Table xxx has a hole  != 222
>
> Is there a correct way to repair this?
>
> Thanks in advance.

Re: data miss when use rowiterator

[Keith Turner]

On Thu, Feb 9, 2017 at 11:39 PM, Josh Elser  wrote:
> Just to be clear, Lu, for now stick to using a Scanner with the RowIterator
> :)
>
> It sounds like we might have to re-think how the RowIterator works with the
> BatchScanner...

I opened : https://issues.apache.org/jira/browse/ACCUMULO-4586

>
> Christopher wrote:
>>
>> I suspected that was the case. BatchScanner does not guarantee ordering
>> of entries, which is needed for the behavior you're expecting with
>> RowIterator. This means that the RowIterator could see the same row
>> multiple times with different subsets of the row's columns. This is
>> probably affecting your count.
>>
>> On Thu, Feb 9, 2017 at 10:29 PM Lu Q > > wrote:
>>
>> I use BatchScanner
>>
>>> 在 2017年2月10日，11:24，Christopher >> > 写道：
>>>
>>> Does it matter if your scanner is a BatchScanner or a Scanner?
>>> I wonder if this is due to the way BatchScanner could split rows up.
>>>
>>> On Thu, Feb 9, 2017 at 9:50 PM Lu Q >> > wrote:
>>>
>>>
>>> I use accumulo 1.8.0,and I develop a ORM framework for
>>> conversion the scan result to a object.
>>>
>>> Before,I use Rowiterator because it faster than direct to use
>>> scan
>>>
>>> RowIterator rows = new RowIterator(scan);
>>> rows.forEachRemaining(rowIterator -> {
>>> while (rowIterator.hasNext()) {
>>> Map.Entry entry = rowIterator.next();
>>> ...
>>> }
>>> }
>>>
>>> it works ok until I query 1000+ once .I found that when the
>>> range size bigger then 1000,some data miss.
>>> I think maybe I conversion it error ,so I change it to a map
>>> struct ,the row_id as the map key ,and other as the map value
>>> ,the problem still exists.
>>>
>>> Then I not use RowIterator,it works ok.
>>> for (Map.Entry entry : scan) {
>>> ...
>>> }
>>>
>>>
>>> Is the bug or my program error ?
>>> Thanks.
>>>
>>> --
>>> Christopher
>>
>>
>> --
>> Christopher

Re: Zookeeper Exception

[Keith Turner]

Accumulo has some static resources that can cause problems for web
applications.  You could try calling CleanUp.shutdownNow() [1]  before
the container unloads the web app.

[1]: 
https://github.com/apache/accumulo/blob/rel/1.7.2/core/src/main/java/org/apache/accumulo/core/util/CleanUp.java#L36

On Tue, Feb 7, 2017 at 9:39 AM, Cyrille Savelief  wrote:
> Hi,
>
>
>
> Since a few weeks, we are having the following recurring warnings (tens of
> megabytes) :
>
>
>
> 02-07-2017 15:17:39 - WARN - org.apache.zookeeper.ClientCnxn run - Session
> 0x159d0669ef70043 for server 10.91.176.82/10.91.176.82:2181, unexpected
> error, closing socket connection and attempting reconnect
>
> java.lang.IllegalStateException: Can't overwrite cause with
> java.lang.IllegalStateException: Illegal access: this web application
> instance has been stopped already.  Could not load
> org.apache.zookeeper.proto.SetWatches.  The eventual following stack trace
> is caused by an error thrown for debugging purposes as well as to attempt to
> terminate the thread which caused the illegal access, and has no functional
> impact.
>
> at java.lang.Throwable.initCause(Throwable.java:457)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1305)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1185)
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1146)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.primeConnection(ClientCnxn.java:926)
>
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
>
> Caused by: java.lang.ClassNotFoundException
>
> at
> org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1304)
>
> ... 5 more
>
>
>
> Apart from this warning, we have no other errors in the log and everything
> is running fine. Do you have any idea where it might come from ?
>
>
>
> Best,
>
>
>
> Cyrille SAVELIEF
>
> ��Z7�>� 4��

Re: accumulo enhancement suggestions

[Keith Turner]

On Mon, Jan 30, 2017 at 4:04 PM, John Vines <vi...@apache.org> wrote:
> From reading the ticket, they want to have different directories in hdfs

Oh yeah,  I should have read the email more closely.

> used. We already support this, this is the instance.volumes property
>
>
> On Mon, Jan 30, 2017 at 10:10 AM Keith Turner <ke...@deenlo.com> wrote:
>>
>> One norm we set in Fluo is expecting the user to add a path to the
>> zookeeper connection string.  For example a connection string like the
>> following :
>>
>>   zserver1:1234,zserver2:1234,zserver3:1234/foo9
>>
>> Would chroot the zookeeper connection to /foo9.  So if you listed
>> children a / using that connection, it would actually list children of
>> /foo9.
>>
>> I wonder if these types of connection strings would work with
>> Accumulo?  If not it may be worthwhile to fix the issues that prevent
>> it from working.  Then you could use this zookeeper feature to
>> accomplish your use case.
>>
>> On Sat, Jan 28, 2017 at 4:16 PM, Josh Elser <josh.el...@gmail.com> wrote:
>> > Please use the user@accumulo.apache.org mailing list in the future for
>> > these
>> > kinds of questions/request. It is not in good taste to send private
>> > emails
>> > to those involved in any Apache projects.
>> >
>> > --
>> >
>> > The functionality you're suggesting already exists for all versions you
>> > mentioned.
>> >
>> >
>> > http://accumulo.apache.org/1.8/accumulo_user_manual.html#_instance_volumes
>> >
>> > Information on contacting those who work on or with Apache Accumulo is
>> > documented at http://accumulo.apache.org/get_involved/.
>> >
>> > Matthew Purdy wrote:
>> >>
>> >> Josh,
>> >>
>> >> I have a simple feature enhancement for accumulo that would help
>> >> developers; is there a place to put in suggestions/requests?
>> >>
>> >>
>> >> =
>> >>
>> >> i develop on linux (normally a VM). i would like to install a few
>> >> accumulo versions (1.6.6, 1.7.2, 1.8.0) on the same VM running on the
>> >> same hadoop pseudo cluster all with different zookeeper instances and
>> >> different root directories in hdfs. this way i can turn on the accumulo
>> >> version i want and run my code on that version.
>> >>
>> >> i can configure multiple zookeeper instances; however, accumulo puts
>> >> all
>> >> its tables in hdfs under /accumulo. it would be nice to configure that
>> >> root path; then i could create /accumulo166, /accumulo172, accumulo180,
>> >> etc and only have to turn on/off different versions of accumulo (once
>> >> they were initially configured in zookeeper)
>> >>
>> >> i don’t think this would be a big change; should only need a new
>> >> property in accumulo-site.xml and expose the root directory using that
>> >> property
>> >>
>> >> =
>> >>
>> >> Thank You,
>> >> Matthew Purdy
>> >>
>> >>
>> >>
>> >> --
>> >> Matthew Purdy
>> >> Principle Software Engineer
>> >> Purdy Good Engineering
>> >> matthew.pu...@purdygoodengineering.com
>> >> <mailto:matthew.pu...@purdygoodengineering.com>
>> >> 443.848.1595
>> >> --
>> >> "Lead, follow, or get out of the way." -- Thomas Paine
>> >> "Make everything as simple as possible, but not simpler." -- Albert
>> >> Einstein
>> >> "The definition of insanity is doing the same thing over and over and
>> >> expecting a different result." -- Benjamin Franklin
>> >> "We can't solve problems by using the same kind of thinking we used
>> >> when
>> >> we created them." -- Albert Einstein
>> >>
>> >>
>> >> --
>> >>
>> >> CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging
>> >> to Purdy Good Engineering LLC and its affiliates may be contained in
>> >> this message. If you are not a recipient indicated or intended in this
>> >> message (or responsible for delivery of this message to such person),
>> >> or
>> >> you think for any reason that this message may have been addressed to
>> >> you in error, you may not use or copy or deliver this message to anyone
>> >> else. In such case, you should destroy this message and are asked to
>> >> notify the sender by reply e-mail.
>> >>
>> >

Re: accumulo enhancement suggestions

[Keith Turner]

One norm we set in Fluo is expecting the user to add a path to the
zookeeper connection string.  For example a connection string like the
following :

  zserver1:1234,zserver2:1234,zserver3:1234/foo9

Would chroot the zookeeper connection to /foo9.  So if you listed
children a / using that connection, it would actually list children of
/foo9.

I wonder if these types of connection strings would work with
Accumulo?  If not it may be worthwhile to fix the issues that prevent
it from working.  Then you could use this zookeeper feature to
accomplish your use case.

On Sat, Jan 28, 2017 at 4:16 PM, Josh Elser  wrote:
> Please use the user@accumulo.apache.org mailing list in the future for these
> kinds of questions/request. It is not in good taste to send private emails
> to those involved in any Apache projects.
>
> --
>
> The functionality you're suggesting already exists for all versions you
> mentioned.
>
> http://accumulo.apache.org/1.8/accumulo_user_manual.html#_instance_volumes
>
> Information on contacting those who work on or with Apache Accumulo is
> documented at http://accumulo.apache.org/get_involved/.
>
> Matthew Purdy wrote:
>>
>> Josh,
>>
>> I have a simple feature enhancement for accumulo that would help
>> developers; is there a place to put in suggestions/requests?
>>
>>
>> =
>>
>> i develop on linux (normally a VM). i would like to install a few
>> accumulo versions (1.6.6, 1.7.2, 1.8.0) on the same VM running on the
>> same hadoop pseudo cluster all with different zookeeper instances and
>> different root directories in hdfs. this way i can turn on the accumulo
>> version i want and run my code on that version.
>>
>> i can configure multiple zookeeper instances; however, accumulo puts all
>> its tables in hdfs under /accumulo. it would be nice to configure that
>> root path; then i could create /accumulo166, /accumulo172, accumulo180,
>> etc and only have to turn on/off different versions of accumulo (once
>> they were initially configured in zookeeper)
>>
>> i don’t think this would be a big change; should only need a new
>> property in accumulo-site.xml and expose the root directory using that
>> property
>>
>> =
>>
>> Thank You,
>> Matthew Purdy
>>
>>
>> --
>> Matthew Purdy
>> Principle Software Engineer
>> Purdy Good Engineering
>> matthew.pu...@purdygoodengineering.com
>> 
>> 443.848.1595
>> --
>> "Lead, follow, or get out of the way." -- Thomas Paine
>> "Make everything as simple as possible, but not simpler." -- Albert
>> Einstein
>> "The definition of insanity is doing the same thing over and over and
>> expecting a different result." -- Benjamin Franklin
>> "We can't solve problems by using the same kind of thinking we used when
>> we created them." -- Albert Einstein
>>
>> --
>>
>> CONFIDENTIALITY NOTICE: Proprietary/Confidential Information belonging
>> to Purdy Good Engineering LLC and its affiliates may be contained in
>> this message. If you are not a recipient indicated or intended in this
>> message (or responsible for delivery of this message to such person), or
>> you think for any reason that this message may have been addressed to
>> you in error, you may not use or copy or deliver this message to anyone
>> else. In such case, you should destroy this message and are asked to
>> notify the sender by reply e-mail.
>>
>

Re: Tablet count per table [SEC=UNOFFICIAL]

[Keith Turner]

There are two operations you can use in the public API to get these
counts.  The following code shows a sketch of what I am thinking,
however I have not tested it.

Connector conn = 
long sum = 0;
for(String table : conn.tableOperations().tableIdMap().keys()){
   // the number of tablets in a table is the number of splits plus one
   sum += conn.tableOperations().listSplits(table, Integer.MAX_INT).size() + 1;
}

The nice thing about this approach is that it uses public API, so you
are not impacted by changes in the metadata table schema.  However, a
possible downside is that it reads all splits into memory.  So if a
tables splits would not fit into memory, thats not good.   The
scanning approaches suggested by others will not have this memory
problem, but you may have to change your code if the metadata schema
changes.

The Accumulo shell has analogues for listing tables and listing a tables splits.

[1]: 
http://accumulo.apache.org/1.8/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#locate%28java.lang.String,%20java.util.Collection%29

On Mon, Jan 23, 2017 at 11:30 PM, Dickson, Matt MR
 wrote:
> UNOFFICIAL
>
> Is there a way to query to the metadata table to quickly get a list of the
> number of tablets for all tables in an Accumulo instance?
>
> I'd like to do this to integrate into existing monitoring tools rather than
> go to the Accumulo gui.
>
> Thanks in advance

Re: Orphaned table in metadata table [SEC=UNOFFICIAL]

[Keith Turner]

On Mon, Jan 16, 2017 at 7:07 PM, Dickson, Matt MR
 wrote:
> UNOFFICIAL
>
> When I run "accumulo org.apache.accumulo.server.util.FindOfflineTablets" it
> finishes with a warning that "No prev-row for key extent {1de; }"  then
> throws a nullPointerException.
>
> Looking in the metadata table there are references to 1de but when I list
> all tables it does not exist.  The master log is also listining a lot of
> warnings regarding no prev-row for this table.
>
> Should I delete all references to 1de in the metadata table?

So when you list all tables ids using the shell, you do not see 1de?
If so, it seems like deleting those entries from the metadata table
would be a good thing to do.Can you grep the master log for 1de?
Would be looking for evidence of a failed delete table operation.

>
>

Re: Teardown and deepCopy

[Keith Turner]

On Wed, Jan 4, 2017 at 11:57 AM, Roshan Punnoose <rosh...@gmail.com> wrote:
> Keith, just would like to ignore it. Basically just doing a distinct
> operation on the column qualifiers.
>
> Dylan, the hard part is that we are trying not to constitute the results on
> the client side completely to do the distinct on the client side. This piece
> is just a smaller piece of a larger query.
>
> Thanks guys for the help. I feel like I'm trying to do something way out of
> bounds of what Accumulo is really built to do. Just testing the bounds :)

Good luck.   As Dylan suggested, partial deduplication in the tserver
for batches and complete deduplication on the client side is a good
option.  Too bad that does not work for you.  You can configure the
scanner to fetch larger batches, which could result in more
deduplication on the server side.

Could possibly do something like the following.

  Scan X keys on client side keeping track of seen IDs
  while(true){
Start a new scan of X keys with last key from prev scan AND pass
seen IDs as config to a scan iterator... keep track of seen IDs
  }

>
> Roshan
>
> On Wed, Jan 4, 2017 at 11:54 AM Keith Turner <ke...@deenlo.com> wrote:
>>
>> On Wed, Jan 4, 2017 at 11:42 AM, Roshan Punnoose <rosh...@gmail.com>
>> wrote:
>> > I have a tablet with an unsorted list of IDs in the Column Qualifier,
>> > these
>> > IDs can repeat sporadically. So I was hoping to keep a set of these IDs
>> > around in memory to check if I have seen an ID or not. There is some
>> > other
>>
>> When you see an ID again, what action do you want to take?
>>
>> > logic to ensure that the set does not grow unbounded, but just trying to
>> > figure out if I can keep this ID set around. With the teardown, even
>> > though
>> > I know which was the last Key to return from the new seek Range, I don't
>> > know if I have seen the upcoming IDs. Not sure if that makes sense...
>> >
>> > Was thinking that on teardown, we could use either the deepCopy or init
>> > method to rollover state from the torn down iterator to the new
>> > iterator.
>> >
>> > On Wed, Jan 4, 2017 at 11:14 AM Keith Turner <ke...@deenlo.com> wrote:
>> >>
>> >> On Wed, Jan 4, 2017 at 10:44 AM, Roshan Punnoose <rosh...@gmail.com>
>> >> wrote:
>> >> > Keith,
>> >> >
>> >> > If an iterator has state that it is maintaining, what is the best way
>> >> > to
>> >> > transfer that state to the new iterator after a tear down?  For
>> >> > example,
>> >> > MyIterator might have a Boolean flag of some sort. After tear down,
>> >> > is
>> >> > there
>> >> > a way to copy that state to the new iterator before it starts seeking
>> >> > again?
>> >>
>> >> There is nothing currently built in to help with this.
>> >>
>> >> What are you trying to accomplish?  Are you interested in maintaining
>> >> this state for a scan or batch scan?
>> >>
>> >>
>> >> >
>> >> > Roshan
>> >> >
>> >> > On Wed, Jan 4, 2017 at 10:33 AM Keith Turner <ke...@deenlo.com>
>> >> > wrote:
>> >> >>
>> >> >> Josh,
>> >> >>
>> >> >> Deepcopy is not called when an iterator is torn down.  It has an
>> >> >> entirely different use. Deepcopy allows cloning of an iterator
>> >> >> during
>> >> >> init().  The clones allow you to have multiple pointers into a
>> >> >> tablets
>> >> >> data which allows things like server side joins.
>> >> >>
>> >> >> Keith
>> >> >>
>> >> >> On Wed, Dec 28, 2016 at 12:50 PM, Josh Clum <joshc...@gmail.com>
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > I have a question about iterator teardown. It seems from
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > https://github.com/apache/accumulo/blob/master/docs/src/main/asciidoc/chapters/iterator_design.txt#L383-L390
>> >> >> > that deepCopy should be called when an iterator is torn down. I'm
>> >> >> > not
>> >> >> > seeing
>> >> >> > that behavior. Below is a test that sets table.scan.max.memory to
>> >> 

Re: Teardown and deepCopy

[Keith Turner]

On Wed, Jan 4, 2017 at 11:42 AM, Roshan Punnoose <rosh...@gmail.com> wrote:
> I have a tablet with an unsorted list of IDs in the Column Qualifier, these
> IDs can repeat sporadically. So I was hoping to keep a set of these IDs
> around in memory to check if I have seen an ID or not. There is some other

When you see an ID again, what action do you want to take?

> logic to ensure that the set does not grow unbounded, but just trying to
> figure out if I can keep this ID set around. With the teardown, even though
> I know which was the last Key to return from the new seek Range, I don't
> know if I have seen the upcoming IDs. Not sure if that makes sense...
>
> Was thinking that on teardown, we could use either the deepCopy or init
> method to rollover state from the torn down iterator to the new iterator.
>
> On Wed, Jan 4, 2017 at 11:14 AM Keith Turner <ke...@deenlo.com> wrote:
>>
>> On Wed, Jan 4, 2017 at 10:44 AM, Roshan Punnoose <rosh...@gmail.com>
>> wrote:
>> > Keith,
>> >
>> > If an iterator has state that it is maintaining, what is the best way to
>> > transfer that state to the new iterator after a tear down?  For example,
>> > MyIterator might have a Boolean flag of some sort. After tear down, is
>> > there
>> > a way to copy that state to the new iterator before it starts seeking
>> > again?
>>
>> There is nothing currently built in to help with this.
>>
>> What are you trying to accomplish?  Are you interested in maintaining
>> this state for a scan or batch scan?
>>
>>
>> >
>> > Roshan
>> >
>> > On Wed, Jan 4, 2017 at 10:33 AM Keith Turner <ke...@deenlo.com> wrote:
>> >>
>> >> Josh,
>> >>
>> >> Deepcopy is not called when an iterator is torn down.  It has an
>> >> entirely different use. Deepcopy allows cloning of an iterator during
>> >> init().  The clones allow you to have multiple pointers into a tablets
>> >> data which allows things like server side joins.
>> >>
>> >> Keith
>> >>
>> >> On Wed, Dec 28, 2016 at 12:50 PM, Josh Clum <joshc...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I have a question about iterator teardown. It seems from
>> >> >
>> >> >
>> >> > https://github.com/apache/accumulo/blob/master/docs/src/main/asciidoc/chapters/iterator_design.txt#L383-L390
>> >> > that deepCopy should be called when an iterator is torn down. I'm not
>> >> > seeing
>> >> > that behavior. Below is a test that sets table.scan.max.memory to 1
>> >> > which
>> >> > should force a tear down for each kv returned. I should see deepCopy
>> >> > being
>> >> > called 3 times but when I tail the Tserver logs I'm not seeing it
>> >> > being
>> >> > called. Below is the test and the Tserver output.
>> >> >
>> >> > What am I missing here?
>> >> >
>> >> > Josh
>> >> >
>> >> > ➜  tail -f -n200 ./accumulo/logs/TabletServer_*.out | grep
>> >> > MyIterator
>> >> > MyIterator: init
>> >> > MyIterator: seek
>> >> > MyIterator: hasTop
>> >> > MyIterator: getTopKey
>> >> > MyIterator: getTopValue
>> >> > MyIterator: init
>> >> > MyIterator: seek
>> >> > MyIterator: hasTop
>> >> > MyIterator: getTopKey
>> >> > MyIterator: getTopValue
>> >> > MyIterator: init
>> >> > MyIterator: seek
>> >> > MyIterator: hasTop
>> >> > MyIterator: getTopKey
>> >> > MyIterator: getTopValue
>> >> > MyIterator: init
>> >> > MyIterator: seek
>> >> > MyIterator: hasTop
>> >> >
>> >> > public static class MyIterator implements SortedKeyValueIterator<Key,
>> >> > Value>
>> >> > {
>> >> >
>> >> > private SortedKeyValueIterator<Key, Value> source;
>> >> >
>> >> > public MyIterator() { }
>> >> >
>> >> > @Override
>> >> > public void init(SortedKeyValueIterator<Key, Value> source,
>> >> >  Map<String, String> options,
>> >> >  IteratorEnvironment env) throws IOException {
>> >> > Sys

Re: Teardown and deepCopy

[Keith Turner]

Josh,

Deepcopy is not called when an iterator is torn down.  It has an
entirely different use. Deepcopy allows cloning of an iterator during
init().  The clones allow you to have multiple pointers into a tablets
data which allows things like server side joins.

Keith

On Wed, Dec 28, 2016 at 12:50 PM, Josh Clum  wrote:
> Hi,
>
> I have a question about iterator teardown. It seems from
> https://github.com/apache/accumulo/blob/master/docs/src/main/asciidoc/chapters/iterator_design.txt#L383-L390
> that deepCopy should be called when an iterator is torn down. I'm not seeing
> that behavior. Below is a test that sets table.scan.max.memory to 1 which
> should force a tear down for each kv returned. I should see deepCopy being
> called 3 times but when I tail the Tserver logs I'm not seeing it being
> called. Below is the test and the Tserver output.
>
> What am I missing here?
>
> Josh
>
> ➜  tail -f -n200 ./accumulo/logs/TabletServer_*.out | grep MyIterator
> MyIterator: init
> MyIterator: seek
> MyIterator: hasTop
> MyIterator: getTopKey
> MyIterator: getTopValue
> MyIterator: init
> MyIterator: seek
> MyIterator: hasTop
> MyIterator: getTopKey
> MyIterator: getTopValue
> MyIterator: init
> MyIterator: seek
> MyIterator: hasTop
> MyIterator: getTopKey
> MyIterator: getTopValue
> MyIterator: init
> MyIterator: seek
> MyIterator: hasTop
>
> public static class MyIterator implements SortedKeyValueIterator
> {
>
> private SortedKeyValueIterator source;
>
> public MyIterator() { }
>
> @Override
> public void init(SortedKeyValueIterator source,
>  Map options,
>  IteratorEnvironment env) throws IOException {
> System.out.println("MyIterator: init");
> this.source = source;
> }
>
> @Override
> public boolean hasTop() {
> System.out.println("MyIterator: hasTop");
> return source.hasTop();
> }
>
> @Override
> public void next() throws IOException {
> System.out.println("MyIterator: next");
> source.next();
> }
>
> @Override
> public void seek(Range range, Collection columnFamilies,
> boolean inclusive) throws IOException {
> System.out.println("MyIterator: seek");
> source.seek(range, columnFamilies, inclusive);
> }
>
> @Override
> public Key getTopKey() {
> System.out.println("MyIterator: getTopKey");
> return source.getTopKey();
> }
>
> @Override
> public Value getTopValue() {
> System.out.println("MyIterator: getTopValue");
> return source.getTopValue();
> }
>
> @Override
> public SortedKeyValueIterator deepCopy(IteratorEnvironment
> env) {
> System.out.println("MyIterator: deepCopy");
> return source.deepCopy(env);
> }
> }
>
> @Test
> public void testTearDown() throws Exception {
> String table = "test";
> Connector conn = cluster.getConnector("root", "secret");
> conn.tableOperations().create(table);
> conn.tableOperations().attachIterator(table, new IteratorSetting(25,
> MyIterator.class));
> conn.tableOperations().setProperty(table, "table.scan.max.memory", "1");
>
> BatchWriter writer = conn.createBatchWriter(table, new
> BatchWriterConfig());
>
> Mutation m1 = new Mutation("row");
> m1.put("f1", "q1", 1, "val1");
> writer.addMutation(m1);
>
> Mutation m2 = new Mutation("row");
> m2.put("f2", "q2", 1, "val2");
> writer.addMutation(m2);
>
> Mutation m3 = new Mutation("row");
> m3.put("f3", "q3", 1, "val3");
> writer.addMutation(m3);
>
> writer.flush();
> writer.close();
>
> BatchScanner scanner = conn.createBatchScanner(table, new
> Authorizations(), 3);
> scanner.setRanges(Collections.singletonList(new Range()));
> for(Map.Entry entry : scanner) {
> System.out.println(entry.getKey() + " : " + entry.getValue());
> }
> System.out.println("Results complete!");
> }

Re: Importtable command not working from Java API

[Keith Turner]

On Thu, Dec 15, 2016 at 6:06 AM, vaibhav thapliyal
 wrote:
> Hi,
>
> I am trying to import an accumulo table from an HDFS directory. However when
> i use the importtable function in the java API I get the following error:
>
> org.apache.accumulo.core.client.AccumuloException: Table import  directory
> /backup/Data_records does not exist!
>
> I have verified that this directory exists in hdfs. Even then I am not able
> to import data through this directory because of this error.
>
>
> Here is the call to the importtable function:
>
> destConnector.tableOperations().importTable(prop.getProperty("tablename"),
> prop.getProperty("targetClusterDirectory"));
>
> I would also like to add that the importtable command works using the
> accumulo shell.

Do you know if you are using the same HDFS configuration in both cases?

>
> Please help.
> Thanks
> Vaibhav
>
>

Re: HDFS vs Accumulo Performance

[Keith Turner]

There is also the set batch size method[1] on the scanner.  I think
that defaults to 1000.   You could try adjusting that.

[1]: 
http://accumulo.apache.org/1.8/apidocs/org/apache/accumulo/core/client/Scanner.html#setBatchSize%28int%29

On Mon, Dec 5, 2016 at 12:10 PM, Mario Pastorelli
 wrote:
> table.scan.max.memory doesn't affect the number of seeks in our case. We
> tried with 1MB and 2MB.
>
> On Mon, Dec 5, 2016 at 5:59 PM, Josh Elser  wrote:
>>
>> If you're only ever doing sequential scans, IMO, it's expected that HDFS
>> would be faster. Remember that, architecturally, Accumulo is designed for
>> *random-read/write* workloads. This is where it would shine in comparison to
>> HDFS. Accumulo is always going to have a hit in sequential read/write
>> workloads over HDFS.
>>
>> As to your question about the number of seeks, try playing with the value
>> of "table.scan.max.memory" [1]. You should be able to easily twiddle the
>> value in the Accumulo shell and re-run the test. Accumulo tears down these
>> active scans because it expects that your client would be taking time to
>> process the results it just sent and it would want to not hold onto those in
>> memory (as your client may not come back). Increasing that property will
>> increase the amount of data sent in one RPC which in turn will reduce the
>> number of RPCs and seeks. Aside: I think this server-side "scanner" lifetime
>> is something that'd we want to revisit sooner than later.
>>
>> 25MB/s seems like a pretty reasonable a read rate for one TabletServer
>> (since you only have one tablet). Similarly, why a BatchScanner would have
>> made no difference. BatchScanners parallelize access to multiple Tablets and
>> would have nothing but overhead when you read from a single Tablet.
>>
>> [1]
>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_table_scan_max_memory
>>
>> Mario Pastorelli wrote:
>>>
>>> We are trying to understand Accumulo performance to better plan our
>>> future products that use it and we noticed that the read speed of
>>> Accumulo tends to be way lower than what we would expect. We have a
>>> testing cluster with 4 HDFS+Accumulo nodes and we ran some tests. We
>>> wrote two programs to write to HDFS and Accumulo and two programs to
>>> read/scan from HDFS and Accumulo the same number of records containing
>>> random bytes. We run all the programs from outside the cluster, on
>>> another node of the rack that doesn’t have HDFS nor Accumulo.
>>>
>>> We also wrote all the HDFS blocks and Accumulo tablets on the same
>>> machine of the cluster.
>>>
>>>
>>> First of all, we wrote 10M entries to HDFS were each entry was 50 bytes
>>> each. This resulted in 4 blocks on HDFS. Reading this records with a
>>> FSDataInputStream takes around 5.7 seconds with an average speed of
>>> around 90MB per second.
>>>
>>>
>>> Then we wrote 10M entries to HDFS where each entry has a row of 50
>>> random bytes, no column and no value. Writing is as fast as writing to
>>> HDFS modulo the compaction that we run at the end. The generated table
>>> has 1 tablet and obviously 10M records all on the same cluster. We
>>> waited for the compaction to finish, then we opened a scanner without
>>> setting the range and we read all the records. This time, reading the
>>> data took around 20 seconds with average speed of 25MB/s and 50
>>> records/s together with ~500 seeks/s. We have two questions about this
>>> result:
>>>
>>>
>>> 1 - is this kind of performance expected?
>>>
>>> 2 - Is there any configuration that we can change to improve the scan
>>> speed?
>>>
>>> 3 - why there are 500 seeks if there is only one tablet and we read
>>> sequentially all its bytes? What are those seeks doing?
>>>
>>>
>>> We tried to use a BatchScanner with 1, 5 and 10 threads but the speed
>>> was the same or even worse in some cases.
>>>
>>>
>>> I can provide the code that we used as well as information about our
>>> cluster configuration if you want.
>>>
>>> Thanks,
>>>
>>> Mario
>>>
>>> --
>>> Mario Pastorelli| TERALYTICS
>>>
>>> *software engineer*
>>>
>>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>>> phone:+41794381682
>>> email: mario.pastore...@teralytics.ch
>>> 
>>> www.teralytics.net 
>>>
>>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>>> Zurich
>>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz,
>>> Yann de Vries
>>>
>>> This e-mail message contains confidential information which is for the
>>> sole attention and use of the intended recipient. Please notify us at
>>> once if you think that it may not be intended for you and delete it
>>> immediately.
>>>
>
>
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> 

Re: HDFS vs Accumulo Performance

[Keith Turner]

Mario,

Nice test!  One possible reason for the difference is the
de-serialization, iterator, re-serialization cost incurred in the
tserver case.  But I am not sure if this is the cause.  For the HDFS
case blobs of serialized data are read from HDFS and shipped to the
client.  When you run the test what does the CPU utilization on the
tserver and client look like?

The seeks are caused by the fact that scanner fetches batches of data.
It needs to seek for each batch.   This batch size can be adjusted
with table.scan.max.memory[1].

Keith

[1]: 
http://accumulo.apache.org/1.8/accumulo_user_manual.html#_table_scan_max_memory

On Mon, Dec 5, 2016 at 11:37 AM, Mario Pastorelli
 wrote:
> We are trying to understand Accumulo performance to better plan our future
> products that use it and we noticed that the read speed of Accumulo tends to
> be way lower than what we would expect. We have a testing cluster with 4
> HDFS+Accumulo nodes and we ran some tests. We wrote two programs to write to
> HDFS and Accumulo and two programs to read/scan from HDFS and Accumulo the
> same number of records containing random bytes. We run all the programs from
> outside the cluster, on another node of the rack that doesn’t have HDFS nor
> Accumulo.
>
> We also wrote all the HDFS blocks and Accumulo tablets on the same machine
> of the cluster.
>
>
> First of all, we wrote 10M entries to HDFS were each entry was 50 bytes
> each. This resulted in 4 blocks on HDFS. Reading this records with a
> FSDataInputStream takes around 5.7 seconds with an average speed of around
> 90MB per second.
>
>
> Then we wrote 10M entries to HDFS where each entry has a row of 50 random
> bytes, no column and no value. Writing is as fast as writing to HDFS modulo
> the compaction that we run at the end. The generated table has 1 tablet and
> obviously 10M records all on the same cluster. We waited for the compaction
> to finish, then we opened a scanner without setting the range and we read
> all the records. This time, reading the data took around 20 seconds with
> average speed of 25MB/s and 50 records/s together with ~500 seeks/s. We
> have two questions about this result:
>
>
> 1 - is this kind of performance expected?
>
> 2 - Is there any configuration that we can change to improve the scan speed?
>
> 3 - why there are 500 seeks if there is only one tablet and we read
> sequentially all its bytes? What are those seeks doing?
>
>
> We tried to use a BatchScanner with 1, 5 and 10 threads but the speed was
> the same or even worse in some cases.
>
>
> I can provide the code that we used as well as information about our cluster
> configuration if you want.
>
> Thanks,
>
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Re: openjdk, Accumulo master state doesn't change from HAVE_LOCK to NORMAL

[Keith Turner]

If you run netstat -nape, do you see the master process listening on
anything?  Maybe its listening on another interface or IP6?

On Thu, Dec 1, 2016 at 5:24 PM, Jayesh Patel  wrote:
> Accumulo 1.7.0 with HDFS 2.7.1
>
>
>
> I was experimenting with openjdk instead of Oracle JRE for Accumulo and ran
> into this issue.  It seems like because it never transitions from HAVE_LOCK
> to NORMAL, it never gets around to start the thrift server.  Changing back
> to Oracle JRE didn’t make a difference.
>
>
>
> Here’s what I get in the logs for the master:
>
> 2016-12-01 17:01:07,970 [trace.DistributedTrace] INFO : SpanReceiver
> org.apache.accumulo.tracer.ZooTraceCli
>
> ent was loaded successfully.
>
> 2016-12-01 17:01:07,971 [master.Master] INFO : trying to get master lock
>
> 2016-12-01 17:01:07,987 [master.EventCoordinator] INFO : State changed from
> INITIAL to HAVE_LOCK
>
> 2016-12-01 17:01:08,038 [master.Master] INFO : New servers:
> [instance-accumulo:9997[258ac79197e000d]]
>
>
>
> On the successful install it transitions right away to NORMAL and goes on
> the listen on port :
>
> 2015-06-07 17:50:35,393 [master.Master] INFO : trying to get master lock
>
> 2015-06-07 17:50:35,408 [master.EventCoordinator] INFO : State changed from
> INITIAL to HAVE_LOCK
>
> 2015-06-07 17:50:35,432 [master.EventCoordinator] INFO : State changed from
> HAVE_LOCK to NORMAL
>
> 2015-06-07 17:50:35,524 [balancer.TableLoadBalancer] INFO : Loaded class
> org.apache.accumulo.server.master.balancer.DefaultLoadBalancer for table !0
>
> 2015-06-07 17:50:35,631 [master.Master] INFO : Setting master lock data to
> 127.0.0.1:
>
>
>
> I found ACCUMULO-4513, but it doesn’t seem relevant as I didn’t try to stop.
>
>
>
> Any ideas as to what is going on?
>
>
>
> HDFS seems fine based on my limited tests with openjdk 1.8.  I did find some
> old posts about Accumulo issues with IBM JDK.  Should I expect Accumulo to
> work with openjdk?
>
>
>
> Thank you,
> Jayesh
>
>

Re: openjdk, Accumulo master state doesn't change from HAVE_LOCK to NORMAL

[Keith Turner]

I use openjdk8 from Ubuntu 16.04 and Centos 7 packages to run Accumulo
all the time without issue.

On Thu, Dec 1, 2016 at 5:24 PM, Jayesh Patel  wrote:
> Accumulo 1.7.0 with HDFS 2.7.1
>
>
>
> I was experimenting with openjdk instead of Oracle JRE for Accumulo and ran
> into this issue.  It seems like because it never transitions from HAVE_LOCK
> to NORMAL, it never gets around to start the thrift server.  Changing back
> to Oracle JRE didn’t make a difference.
>
>
>
> Here’s what I get in the logs for the master:
>
> 2016-12-01 17:01:07,970 [trace.DistributedTrace] INFO : SpanReceiver
> org.apache.accumulo.tracer.ZooTraceCli
>
> ent was loaded successfully.
>
> 2016-12-01 17:01:07,971 [master.Master] INFO : trying to get master lock
>
> 2016-12-01 17:01:07,987 [master.EventCoordinator] INFO : State changed from
> INITIAL to HAVE_LOCK
>
> 2016-12-01 17:01:08,038 [master.Master] INFO : New servers:
> [instance-accumulo:9997[258ac79197e000d]]
>
>
>
> On the successful install it transitions right away to NORMAL and goes on
> the listen on port :
>
> 2015-06-07 17:50:35,393 [master.Master] INFO : trying to get master lock
>
> 2015-06-07 17:50:35,408 [master.EventCoordinator] INFO : State changed from
> INITIAL to HAVE_LOCK
>
> 2015-06-07 17:50:35,432 [master.EventCoordinator] INFO : State changed from
> HAVE_LOCK to NORMAL
>
> 2015-06-07 17:50:35,524 [balancer.TableLoadBalancer] INFO : Loaded class
> org.apache.accumulo.server.master.balancer.DefaultLoadBalancer for table !0
>
> 2015-06-07 17:50:35,631 [master.Master] INFO : Setting master lock data to
> 127.0.0.1:
>
>
>
> I found ACCUMULO-4513, but it doesn’t seem relevant as I didn’t try to stop.
>
>
>
> Any ideas as to what is going on?
>
>
>
> HDFS seems fine based on my limited tests with openjdk 1.8.  I did find some
> old posts about Accumulo issues with IBM JDK.  Should I expect Accumulo to
> work with openjdk?
>
>
>
> Thank you,
> Jayesh
>
>

Re: Accumulo Tip: Batch your Mutations

[Keith Turner]

Also the native map is a Map> ... when doing
updates for a mutation, it gets the Map once and uses
that.This can be much faster than a Map, because for
Map each insert may have to traverse a deeper tree than
inserting into Map.

On Wed, Nov 30, 2016 at 11:47 PM, Dylan Hutchison
 wrote:
> Hi folks,
>
> I'd like to share a tip that ~doubled BatchWriter ingest performance in my
> application.
>
> When inserting multiple entries to the same Accumulo row, put them into the
> same Mutation object.  Add that one large Mutation to a BatchWriter rather
> than an individual Mutation for each entry. The result reduces the amount of
> data transferred.
>
> The tip seems obvious enough, but hey, I used Accumulo for a couple years
> without realizing it, so I thought y'all might benefit too.
>
> Enjoy!
> Dylan

Re: upgrade from 1.7 to 1.8 faild

[Keith Turner]

Can you go in zookeeper and list the nodes at
/accumulo/8f3d8a3e-4c96-4447-aae0-8951f806ba2d/users/ ?

I looked up some of the code related to the exception on GH.  Here are
links if anyone else wants to look.

https://github.com/apache/accumulo/blob/rel/1.8.0/server/master/src/main/java/org/apache/accumulo/master/Master.java#L405
https://github.com/apache/accumulo/blob/rel/1.8.0/server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java#L680

On Tue, Nov 22, 2016 at 9:13 AM, Lu Q  wrote:
> I have a 1.7 accumulo ,and now I upgrade it to 1.8.
>
> I run the start-here.sh,it looks well.
> ```
> Starting tserver on 172.19.106.103
> 2016-11-22 22:08:59,115 [security.SecurityUtil] INFO : Attempting to login
> with keytab as ${principal}
> 2016-11-22 22:08:59,295 [security.SecurityUtil] INFO : Succesfully logged in
> as user ${principal}
> 2016-11-22 22:08:59,800 [fs.VolumeManagerImpl] WARN :
> dfs.datanode.synconclose set to false in hdfs-site.xml: data loss is
> possible on hard system reset or power loss
> 2016-11-22 22:08:59,803 [server.Accumulo] INFO : Attempting to talk to
> zookeeper
> 2016-11-22 22:08:59,924 [server.Accumulo] INFO : ZooKeeper connected and
> initialized, attempting to talk to HDFS
> 2016-11-22 22:08:59,990 [server.Accumulo] INFO : Connected to HDFS
> Starting master on 172.19.106.103
> Starting tracer on 172.19.106.103
> ```
>
> But the master not  start success.
> The log is this.
> ```
> 2016-11-22 22:09:18,388 [master.Master] DEBUG: Converting table 9z WALog
> setting to Durability
> 2016-11-22 22:09:18,396 [master.Master] DEBUG: Upgrade creating namespace
> "accumulo" (ID: +accumulo)
> 2016-11-22 22:09:18,401 [master.Master] DEBUG: Upgrade creating namespace ""
> (ID: +default)
> 2016-11-22 22:09:18,401 [master.Master] DEBUG: Upgrade creating table
> accumulo.replication (ID: +rep)
> 2016-11-22 22:09:18,403 [tables.TableManager] DEBUG: Creating ZooKeeper
> entries for new table accumulo.replication (ID: +rep) in namespace (ID:
> +accumulo)
> 2016-11-22 22:09:18,475 [master.Master] DEBUG: Upgrade creating table
> accumulo.root (ID: +r)
> 2016-11-22 22:09:18,475 [tables.TableManager] DEBUG: Creating ZooKeeper
> entries for new table accumulo.root (ID: +r) in namespace (ID: +accumulo)
> 2016-11-22 22:09:18,542 [Configuration.deprecation] INFO :
> dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
> 2016-11-22 22:09:25,597 [handler.ZKPermHandler] ERROR: KeeperErrorCode =
> NoNode for
> /accumulo/8f3d8a3e-4c96-4447-aae0-8951f806ba2d/users/cm9vdA==/Tables/+r
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for
> /accumulo/8f3d8a3e-4c96-4447-aae0-8951f806ba2d/users/cm9vdA==/Tables/+r
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> at
> org.apache.accumulo.fate.zookeeper.ZooUtil.putData(ZooUtil.java:289)
> at
> org.apache.accumulo.fate.zookeeper.ZooUtil.putPersistentData(ZooUtil.java:268)
> at
> org.apache.accumulo.fate.zookeeper.ZooReaderWriter.putPersistentData(ZooReaderWriter.java:68)
> at
> org.apache.accumulo.server.security.handler.ZKPermHandler.grantTablePermission(ZKPermHandler.java:217)
> at
> org.apache.accumulo.server.security.handler.KerberosPermissionHandler.grantTablePermission(KerberosPermissionHandler.java:101)
> at
> org.apache.accumulo.server.security.SecurityOperation.grantTablePermission(SecurityOperation.java:680)
> at
> org.apache.accumulo.server.security.AuditedSecurityOperation.grantTablePermission(AuditedSecurityOperation.java:572)
> at
> org.apache.accumulo.master.Master.upgradeZookeeper(Master.java:405)
> at org.apache.accumulo.master.Master.setMasterState(Master.java:264)
> at org.apache.accumulo.master.Master.getMasterLock(Master.java:1417)
> at org.apache.accumulo.master.Master.run(Master.java:1135)
> at org.apache.accumulo.master.Master.main(Master.java:1434)
> at
> org.apache.accumulo.master.MasterExecutable.execute(MasterExecutable.java:33)
> at org.apache.accumulo.start.Main$1.run(Main.java:120)
> at java.lang.Thread.run(Thread.java:745)
> 2016-11-22 22:09:25,605 [master.Master] ERROR: FATAL: Error performing
> upgrade
> ThriftSecurityException(user:cm9vdA==, code:CONNECTION_ERROR)
> at
> org.apache.accumulo.core.client.AccumuloSecurityException.asThriftException(AccumuloSecurityException.java:74)
> at
> org.apache.accumulo.server.security.SecurityOperation.grantTablePermission(SecurityOperation.java:683)
> at
> org.apache.accumulo.server.security.AuditedSecurityOperation.grantTablePermission(AuditedSecurityOperation.java:572)
> at
> org.apache.accumulo.master.Master.upgradeZookeeper(Master.java:405)
> at 

Re: Comparing Schemes in Accumulo

[Keith Turner]

On Mon, Nov 7, 2016 at 6:54 AM, Oliver Swoboda  wrote:
> Hello,
>
> I've stored weather data in two tables with different schemes. Scheme1 is
> using the month and station ID for the row key (e.g. 201601_GME00102292) and
> the days of the month (1-31) in the version column. Scheme2 is using the
> year and station ID for the row key (e.g. 2016_GME00102292) and the days of
> the year (1-366) in the version column. Of course, the version iterator has
> been removed from the tables. Because I have different metrics, like minimum
> temperature and maximum temperature of one day, I'm using locality groups,
> one group for each metric. (e.g. setgroups TMIN=TMIN, TMAX=TMAX).
> Additionaly I've done a pre splitting by year (e.g. 2014, 2015, 2016, ...).
>
> Now to my question: If I do a full table scan with a batch scanner, Scheme2
> is always faster than Scheme1 (with 2.5 billion entries Scheme1's scan took
> 24 minutes and Scheme2's scan took 21 minutes). Why is that? Is it because
> there are fewer seeks made when using Scheme2? Would be nice if someone can
> help me to understand what's happening here.

One possible reason is the relative encoding used in Accumulo.   When
two consecutive keys have the same row, the second key will just point
to the previous row.   This makes row comparisons faster.  Also when
data is transferred over the network from server to client, repeated
rows are not transferred.

>
> Yours faithfully,
> Oliver Swoboda

Re: New Accumulo Blog Post

[Keith Turner]

On Wed, Nov 2, 2016 at 9:01 PM, Jeff Kubina  wrote:
> Thanks for the blog post, very interesting read. Some questions ...
>
> 1. Are the operations "Writes mutation to tablet servers’ WAL/Sync or flush
> tablet servers’ WAL" and "Adds mutations to sorted in memory map of each
> tablet." performed by threads in parallel?

Not for a batch from a client.  Each batch of mutations from a client
is processed by a single thread.   Batches from separate clients can
process in parallel.


>
> 2. Could the latency of hsync-ing the WALs be overcome by modifying Accumulo
> to write them to a separate SSD-only HDFS? To maintain data locality it
> would require two datanode processes (one for the HDDs and one for the SSD),
> running on the same node, which is not hard to do.

Christopher answered this.  I just wanted to mention that I have run
Accumulo on my laptop which has a SSD.  Hsync is much faster there,
like 10x faster.


>

New Accumulo Blog Post

[Keith Turner]

I just posted a new blog post for Accumulo about configuring
durability settings and durability performance impact.

http://accumulo.apache.org/blog/2016/11/02/durability-performance.html

[ANNOUNCE] Apache Fluo Recipes 1.0.0-incubating released

[Keith Turner]

The Apache Fluo (incubating) team is happy to announce the release of Fluo
Recipes 1.0.0-incubating:

https://fluo.apache.org/release/fluo-recipes-1.0.0-incubating/

Fluo Recipes builds on the Fluo API to offer additional functionality
to developers. They are published separately from Fluo on their own
release schedule. This allows Fluo Recipes to iterate and innovate
faster than Fluo (which will maintain a more minimal API on a slower
release cycle).  Fluo Recipes offers code to implement common patterns
on top of Fluo's API.  It also offers glue code to external libraries
like Spark and Kryo.

Thanks,
The Apache Fluo team

=

*Disclaimer*

Apache Fluo is an effort undergoing incubation at The Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is
required of all newly accepted projects until a further review indicates
that the infrastructure, communications, and decision making process have
stabilized in a manner consistent with other successful ASF projects. While
incubation status is not necessarily a reflection of the completeness or
stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.

Re: Bulk ingestion of different locality groups at different times

[Keith Turner]

On Fri, Oct 28, 2016 at 10:03 AM, Mario Pastorelli
 wrote:
> Thanks for the answers. About the huge major compaction, the question is not
> about when the major compaction will be but more about how big the major
> compaction of two bulked loaded files will be. The rfiles will be already
> sorted and they will contain two different locality groups and Accumulo
> stores locality groups  separately on disk. The compaction should not do
> much here, just reuse the created groups, right?

Accumulo stores multiple locality groups into a single file.
Compactions make a pass for each locality group.  The following is a
sketch of what compactions do.

inputIter = //an iterator over the files being compacted
outputRFile = //the tmp file compaction is writing to

for(localityGroup : localityGroups) {
inputIter.seek(new Range(), localityGroup.getFamilies(), true)
outputRFile.startLocalityGroup(localityGroup.getFamilies())

//write intputIter to outputRFile
}

//write default locality group
inputIter.seek(new Range(), localityGroups.getAllFamilies(), false)
//read all families not in a configured LG
outputRFile.startDefaultLocalityGroup()

//write intputIter to outputRFile

>
> On Fri, Oct 28, 2016 at 3:54 PM,  wrote:
>>
>> >>> Is Accumulo able to import these files, considering that they are two
>> >>> different locality groups
>>
>>  Yes.
>>
>> >>> without triggering a huge major compaction?
>>
>> Depends on your table.compaction.major.ratio and table.file.max settings.
>>
>>
>> Sorry, not a real answer, but I think the answer is "it depends"
>>
>> 
>> From: "Mario Pastorelli" 
>> To: user@accumulo.apache.org
>> Sent: Friday, October 28, 2016 9:37:13 AM
>> Subject: Bulk ingestion of different locality groups at different times
>>
>>
>> Hi,
>>
>> I have a question about using bulk ingestion for a rather special case.
>> Let's say that I have the locality groups A and B. The values of each
>> locality group are written to Accumulo in at different times, which means
>> that first we ingest all the cells of the group A and then of B. We use
>> Spark to ingest those records. Right now we write all the values with a
>> custom writer but we would like to create the rfiles directly with Spark. In
>> the case above, we would have two jobs creating the rfiles for the two
>> distinct locality groups. Is Accumulo able to import these files,
>> considering that they are two different locality groups, without triggering
>> a huge major compaction?  If not, what strategy would you suggest for the
>> above use case?
>>
>> Thanks,
>> Mario
>>
>> --
>> Mario Pastorelli | TERALYTICS
>>
>> software engineer
>>
>> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
>> phone: +41794381682
>> email: mario.pastore...@teralytics.ch
>> www.teralytics.net
>>
>> Company registration number: CH-020.3.037.709-7 | Trade register Canton
>> Zurich
>> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
>> de Vries
>>
>> This e-mail message contains confidential information which is for the
>> sole attention and use of the intended recipient. Please notify us at once
>> if you think that it may not be intended for you and delete it immediately.
>>
>>
>
>
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Re: Bulk ingestion of different locality groups at different times

[Keith Turner]

Currently, locality groups can not be configured for
AccumuloFileoutputFormat, it can only write to the default locality
group.   However things can still work out nicely, because the default
locality group will track up to a 1000 column families.  It will use
these tracked column families at scan time to determine if the file
should be used. The following is an example of this.

 * Locality Group A has families 3, 4, 5
 * Locality Group B has families x, y, z
 * Spark job 1 writes to Rfile RF1 families y,z.  This data will end
up in default LG.  RF1 is imported to tablet T1.
 * Spark job 2 writes to Rfile RF2 families 3,5.  This data will end
up in default LG.  RF2 is imported.to tablet T1.
 * A scan comes in for tablet T1 for family 3.  The scan will examine
RF1 and RF2 family metadata and only use RF2.
 * If T1 compacts RF1 and RF2 into a single file RF3, that file will
have two locality groups.  When the compaction reads LG B it will only
read data from RF1.  When it reads LG A it will only read data from
RF2.


On Fri, Oct 28, 2016 at 9:37 AM, Mario Pastorelli
 wrote:
> Hi,
>
> I have a question about using bulk ingestion for a rather special case.
> Let's say that I have the locality groups A and B. The values of each
> locality group are written to Accumulo in at different times, which means
> that first we ingest all the cells of the group A and then of B. We use
> Spark to ingest those records. Right now we write all the values with a
> custom writer but we would like to create the rfiles directly with Spark. In
> the case above, we would have two jobs creating the rfiles for the two
> distinct locality groups. Is Accumulo able to import these files,
> considering that they are two different locality groups, without triggering
> a huge major compaction?  If not, what strategy would you suggest for the
> above use case?
>
> Thanks,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Re: Orphaned FATE Locks [SEC=UNOFFICIAL]

[Keith Turner]

On Wed, Sep 7, 2016 at 8:55 PM, Dickson, Matt MR
 wrote:
> UNOFFICIAL
>
>
> In Zookeeper there don't appear to be any locks with the same txid that is 
> listed via Accumulo.  However under /accumulo//table_locks/+default/ 
> there are the same number of files as orphaned locks labelled 
> 'lock-00xx', are these the locks I can delete?

Every table lock should have an associated fate transaction.   The
fate print tool has analysis[1] to look for situations when this is
not the case.  It seems like this analysis is finding locks w/o
transactions.   Looking at the code I see there is slight possibility
of race condition.  Its reads the list of locks into memory and then
reads the transactions.  So if a transaction completed and deleted
lock between those steps the tool could report a false positive.  If
you run the tool multiple times and it reports the same thing then
that race condition is not the cause.

Yes, you should delete the locks.  These orphaned locks can hold up
future table operations.  It would be nice to find out what caused
this.  You can grep for the fate txid related to the locks in the
master log to gather info about the fate tx that created the lock.

Do you know if the tables ids its complaining about as having locks
were deleted?  Do you know if fate transactions were deleted (manually
in ZK or using Accumulo's tool)?


[1]: 
https://github.com/apache/accumulo/blob/rel/1.8.0/fate/src/main/java/org/apache/accumulo/fate/AdminUtil.java#L69

>
> I should note that while investigating this there were no other fate 
> transactions being listed by Accumulo for this table, +default, so the system 
> was in a stable state.
>
>
> -Original Message-
>
>
> From: Josh Elser [mailto:josh.el...@gmail.com]
> Sent: Thursday, 8 September 2016 01:07
> To: user@accumulo.apache.org
> Subject: Re: Orphaned FATE Locks [SEC=UNOFFICIAL]
>
> Hi Matt,
>
> What version of Accumulo are you using? Figuring out why those transactions 
> aren't automatically get removed is something else we would want to look into.
>
> It sounds like these transactions are just vestigial (not actually running), 
> so I wouldn't think that they would affect current bulk loads.
>
> I believe you could just stop the Master and remove the corresponding nodes 
> in ZooKeeper (as that's where the txns are stored and `fate print` is reading 
> from), but I would defer to Keith for confirmation first :)
>
> Dickson, Matt MR wrote:
>> *UNOFFICIAL*
>>
>> When running 'fate print -t IN_PROGRESS' to list fate transactions
>> there are approximately eight orphaned locks listed as:
>> txid: xxx locked: [R:+default] I'm looking into these
>> because bulk ingests are failing and there are a lot of CopyFailed
>> transactions in the fate lock list. Could these orphaned locks block
>> further bulk ingests and is there a way to kill them?
>> When I run 'fate fail ' it states there is no fate transaction
>> associated with the transaction id.
>> Thanks advance,
>> Matt

Re: Accumulo Seek performance

[Keith Turner]

On Mon, Sep 12, 2016 at 5:50 PM, Adam J. Shook  wrote:
> As an aside, this is actually pretty relevant to the work I've been doing
> for Presto/Accumulo integration.  It isn't uncommon to have around a million
> exact Ranges (that is, Ranges with a single row ID)  spread across the five
> Presto worker nodes we use for scanning Accumulo.  Right now, these ranges
> get packed into PrestoSplits, 10k ranges per split (an arbitrary number I
> chose), and each split is run in parallel (depending on the overall number
> of splits, they may be queued for execution).
>
> I'm curious to see the query impact of changing it to use a fixed thread
> pool of Scanners over the current BatchScanner implementation.  Maybe I'll
> play around with it sometime soon.

I added a readme to Josh's GH repo w/ the info I learned from Josh on
IRC.   So this should make it quicker for others to experiment.

>
> --Adam
>
> On Mon, Sep 12, 2016 at 2:47 PM, Dan Blum  wrote:
>>
>> I think the 450 ranges returned a total of about 7.5M entries, but the
>> ranges were in fact quite small relative to the size of the table.
>>
>> -Original Message-
>> From: Josh Elser [mailto:josh.el...@gmail.com]
>> Sent: Monday, September 12, 2016 2:43 PM
>> To: user@accumulo.apache.org
>> Subject: Re: Accumulo Seek performance
>>
>> What does a "large scan" mean here, Dan?
>>
>> Sven's original problem statement was running many small/pointed Ranges
>> (e.g. point lookups). My observation was that BatchScanners were slower
>> than running each in a Scanner when using multiple BS's concurrently.
>>
>> Dan Blum wrote:
>> > I tested a large scan on a 1.6.2 cluster with 11 tablet servers - using
>> > Scanners was much slower than using a BatchScanner with 11 threads, by 
>> > about
>> > a 5:1 ratio. There were 450 ranges.
>> >
>> > -Original Message-
>> > From: Josh Elser [mailto:josh.el...@gmail.com]
>> > Sent: Monday, September 12, 2016 1:42 PM
>> > To: user@accumulo.apache.org
>> > Subject: Re: Accumulo Seek performance
>> >
>> > I had increased the readahead threed pool to 32 (from 16). I had also
>> > increased the minimum thread pool size from 20 to 40. I had 10 tablets
>> > with the data block cache turned on (probably only 256M tho).
>> >
>> > Each tablet had a single file (manually compacted). Did not observe
>> > cache rates.
>> >
>> > I've been working through this with Keith on IRC this morning too. Found
>> > that a single batchscanner (one partition) is faster than the Scanner.
>> > Two partitions and things started to slow down.
>> >
>> > Two interesting points to still pursue, IMO:
>> >
>> > 1. I saw that the tserver-side logging for MultiScanSess was near
>> > identical to the BatchScanner timings
>> > 2. The minimum server threads did not seem to be taking effect. Despite
>> > having the value set to 64, I only saw a few ClientPool threads in a
>> > jstack after running the test.
>> >
>> > Adam Fuchs wrote:
>> >> Sorry, Monday morning poor reading skills, I guess. :)
>> >>
>> >> So, 3000 ranges in 40 seconds with the BatchScanner. In my past
>> >> experience HDFS seeks tend to take something like 10-100ms, and I would
>> >> expect that time to dominate here. With 60 client threads your
>> >> bottleneck should be the readahead pool, which I believe defaults to 16
>> >> threads. If you get perfect index caching then you should be seeing
>> >> something like 3000/16*50ms = 9,375ms. That's in the right ballpark,
>> >> but
>> >> it assumes no data cache hits. Do you have any idea of how many files
>> >> you had per tablet after the ingest? Do you know what your cache hit
>> >> rate was?
>> >>
>> >> Adam
>> >>
>> >>
>> >> On Mon, Sep 12, 2016 at 9:14 AM, Josh Elser> >> >  wrote:
>> >>
>> >>  5 iterations, figured that would be apparent from the log messages
>> >> :)
>> >>
>> >>  The code is already posted in my original message.
>> >>
>> >>  Adam Fuchs wrote:
>> >>
>> >>  Josh,
>> >>
>> >>  Two questions:
>> >>
>> >>  1. How many iterations did you do? I would like to see an
>> >> absolute
>> >>  number of lookups per second to compare against other
>> >> observations.
>> >>
>> >>  2. Can you post your code somewhere so I can run it?
>> >>
>> >>  Thanks,
>> >>  Adam
>> >>
>> >>
>> >>  On Sat, Sep 10, 2016 at 3:01 PM, Josh Elser
>> >>  
>> >>  >>
>> >> wrote:
>> >>
>> >>   Sven, et al:
>> >>
>> >>   So, it would appear that I have been able to reproduce
>> >> this one
>> >>   (better late than never, I guess...). tl;dr Serially
>> >> using
>> >>  Scanners
>> >>   to do point lookups instead of a BatchScanner is ~20x
>> >>  faster. This
>> >>   sounds like a pretty serious 

Re: Accumulo Seek performance

[Keith Turner]

Note  I was running a single tserver, datanode, and zookeeper on my workstation.

On Mon, Sep 12, 2016 at 2:02 PM, Keith Turner <ke...@deenlo.com> wrote:
> Josh helped me get up and running w/ YCSB and I Am seeing very
> different results.   I am going to make a pull req to Josh's GH repo
> to add a Readme w/ what I learned from Josh in IRC.
>
> The link below is the Accumulo config I used for running a local 1.8.0 
> instance.
>
> https://gist.github.com/keith-turner/4678a0aac2a2a0e240ea5d73285743ab
>
> I created splits user1~ user2~ user3~ user4~ user5~ user6~ user7~
> user8~ user9~ AND then compacted the table.
>
> Below is the performance I saw with a single batch scanner (configured
> 1 partition).  The batch scanner has 10 threads.
>
> 2016-09-12 12:36:41,079 [client.ClientConfiguration] WARN : Found no
> client.conf in default paths. Using default client configuration
> values.
> 2016-09-12 12:36:41,428 [joshelser.YcsbBatchScanner] INFO : Connected
> to Accumulo
> 2016-09-12 12:36:41,429 [joshelser.YcsbBatchScanner] INFO : Computing ranges
> 2016-09-12 12:36:48,059 [joshelser.YcsbBatchScanner] INFO : Calculated
> all rows: Found 100 rows
> 2016-09-12 12:36:48,096 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
> 2016-09-12 12:36:48,116 [joshelser.YcsbBatchScanner] INFO : All ranges
> calculated: 3000 ranges found
> 2016-09-12 12:36:48,118 [joshelser.YcsbBatchScanner] INFO : Executing
> 1 range partitions using a pool of 1 threads
> 2016-09-12 12:36:49,372 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1252 ms
> 2016-09-12 12:36:49,372 [joshelser.YcsbBatchScanner] INFO : Executing
> 1 range partitions using a pool of 1 threads
> 2016-09-12 12:36:50,561 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1188 ms
> 2016-09-12 12:36:50,561 [joshelser.YcsbBatchScanner] INFO : Executing
> 1 range partitions using a pool of 1 threads
> 2016-09-12 12:36:51,741 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1179 ms
> 2016-09-12 12:36:51,741 [joshelser.YcsbBatchScanner] INFO : Executing
> 1 range partitions using a pool of 1 threads
> 2016-09-12 12:36:52,974 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1233 ms
> 2016-09-12 12:36:52,974 [joshelser.YcsbBatchScanner] INFO : Executing
> 1 range partitions using a pool of 1 threads
> 2016-09-12 12:36:54,146 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1171 ms
>
> Below is the performance I saw with 6 batch scanners. Each batch
> scanner has 10 threads.
>
> 2016-09-12 13:58:21,061 [client.ClientConfiguration] WARN : Found no
> client.conf in default paths. Using default client configuration
> values.
> 2016-09-12 13:58:21,380 [joshelser.YcsbBatchScanner] INFO : Connected
> to Accumulo
> 2016-09-12 13:58:21,381 [joshelser.YcsbBatchScanner] INFO : Computing ranges
> 2016-09-12 13:58:28,571 [joshelser.YcsbBatchScanner] INFO : Calculated
> all rows: Found 100 rows
> 2016-09-12 13:58:28,606 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
> 2016-09-12 13:58:28,632 [joshelser.YcsbBatchScanner] INFO : All ranges
> calculated: 3000 ranges found
> 2016-09-12 13:58:28,634 [joshelser.YcsbBatchScanner] INFO : Executing
> 6 range partitions using a pool of 6 threads
> 2016-09-12 13:58:30,273 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1637 ms
> 2016-09-12 13:58:30,273 [joshelser.YcsbBatchScanner] INFO : Executing
> 6 range partitions using a pool of 6 threads
> 2016-09-12 13:58:31,883 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1609 ms
> 2016-09-12 13:58:31,883 [joshelser.YcsbBatchScanner] INFO : Executing
> 6 range partitions using a pool of 6 threads
> 2016-09-12 13:58:33,422 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1539 ms
> 2016-09-12 13:58:33,422 [joshelser.YcsbBatchScanner] INFO : Executing
> 6 range partitions using a pool of 6 threads
> 2016-09-12 13:58:34,994 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1571 ms
> 2016-09-12 13:58:34,994 [joshelser.YcsbBatchScanner] INFO : Executing
> 6 range partitions using a pool of 6 threads
> 2016-09-12 13:58:36,512 [joshelser.YcsbBatchScanner] INFO : Queries
> executed in 1517 ms
>
> Below is the performance I saw with 6 threads each using a scanner.
>
> 2016-09-12 14:01:14,972 [client.ClientConfiguration] WARN : Found no
> client.conf in default paths. Using default client configuration
> values.
> 2016-09-12 14:01:15,287 [joshelser.YcsbBatchScanner] INFO : Connected
> to Accumulo
> 2016-09-12 14:01:15,288 [joshelser.YcsbBatchScanner] INFO : Computing ranges
> 2016-09-12 14:01:22,309 [joshelser.YcsbBatchScanner] INFO : Calculated
> all rows: Found 100 rows
> 2016-09-12 14:01:22,352 [joshelser.YcsbBat

Re: Accumulo Seek performance

[Keith Turner]

Josh helped me get up and running w/ YCSB and I Am seeing very
different results.   I am going to make a pull req to Josh's GH repo
to add a Readme w/ what I learned from Josh in IRC.

The link below is the Accumulo config I used for running a local 1.8.0 instance.

https://gist.github.com/keith-turner/4678a0aac2a2a0e240ea5d73285743ab

I created splits user1~ user2~ user3~ user4~ user5~ user6~ user7~
user8~ user9~ AND then compacted the table.

Below is the performance I saw with a single batch scanner (configured
1 partition).  The batch scanner has 10 threads.

2016-09-12 12:36:41,079 [client.ClientConfiguration] WARN : Found no
client.conf in default paths. Using default client configuration
values.
2016-09-12 12:36:41,428 [joshelser.YcsbBatchScanner] INFO : Connected
to Accumulo
2016-09-12 12:36:41,429 [joshelser.YcsbBatchScanner] INFO : Computing ranges
2016-09-12 12:36:48,059 [joshelser.YcsbBatchScanner] INFO : Calculated
all rows: Found 100 rows
2016-09-12 12:36:48,096 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
2016-09-12 12:36:48,116 [joshelser.YcsbBatchScanner] INFO : All ranges
calculated: 3000 ranges found
2016-09-12 12:36:48,118 [joshelser.YcsbBatchScanner] INFO : Executing
1 range partitions using a pool of 1 threads
2016-09-12 12:36:49,372 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1252 ms
2016-09-12 12:36:49,372 [joshelser.YcsbBatchScanner] INFO : Executing
1 range partitions using a pool of 1 threads
2016-09-12 12:36:50,561 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1188 ms
2016-09-12 12:36:50,561 [joshelser.YcsbBatchScanner] INFO : Executing
1 range partitions using a pool of 1 threads
2016-09-12 12:36:51,741 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1179 ms
2016-09-12 12:36:51,741 [joshelser.YcsbBatchScanner] INFO : Executing
1 range partitions using a pool of 1 threads
2016-09-12 12:36:52,974 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1233 ms
2016-09-12 12:36:52,974 [joshelser.YcsbBatchScanner] INFO : Executing
1 range partitions using a pool of 1 threads
2016-09-12 12:36:54,146 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1171 ms

Below is the performance I saw with 6 batch scanners. Each batch
scanner has 10 threads.

2016-09-12 13:58:21,061 [client.ClientConfiguration] WARN : Found no
client.conf in default paths. Using default client configuration
values.
2016-09-12 13:58:21,380 [joshelser.YcsbBatchScanner] INFO : Connected
to Accumulo
2016-09-12 13:58:21,381 [joshelser.YcsbBatchScanner] INFO : Computing ranges
2016-09-12 13:58:28,571 [joshelser.YcsbBatchScanner] INFO : Calculated
all rows: Found 100 rows
2016-09-12 13:58:28,606 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
2016-09-12 13:58:28,632 [joshelser.YcsbBatchScanner] INFO : All ranges
calculated: 3000 ranges found
2016-09-12 13:58:28,634 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 13:58:30,273 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1637 ms
2016-09-12 13:58:30,273 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 13:58:31,883 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1609 ms
2016-09-12 13:58:31,883 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 13:58:33,422 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1539 ms
2016-09-12 13:58:33,422 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 13:58:34,994 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1571 ms
2016-09-12 13:58:34,994 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 13:58:36,512 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 1517 ms

Below is the performance I saw with 6 threads each using a scanner.

2016-09-12 14:01:14,972 [client.ClientConfiguration] WARN : Found no
client.conf in default paths. Using default client configuration
values.
2016-09-12 14:01:15,287 [joshelser.YcsbBatchScanner] INFO : Connected
to Accumulo
2016-09-12 14:01:15,288 [joshelser.YcsbBatchScanner] INFO : Computing ranges
2016-09-12 14:01:22,309 [joshelser.YcsbBatchScanner] INFO : Calculated
all rows: Found 100 rows
2016-09-12 14:01:22,352 [joshelser.YcsbBatchScanner] INFO : Shuffled all rows
2016-09-12 14:01:22,373 [joshelser.YcsbBatchScanner] INFO : All ranges
calculated: 3000 ranges found
2016-09-12 14:01:22,376 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 14:01:25,696 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 3318 ms
2016-09-12 14:01:25,696 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions using a pool of 6 threads
2016-09-12 14:01:29,001 [joshelser.YcsbBatchScanner] INFO : Queries
executed in 3305 ms
2016-09-12 14:01:29,001 [joshelser.YcsbBatchScanner] INFO : Executing
6 range partitions

Re: Accumulo Seek performance

[Keith Turner]

On Mon, Sep 12, 2016 at 10:58 AM, Josh Elser  wrote:
> Good call. I kind of forgot about BatchScanner threads and trying to factor
> those in :). I guess doing one thread in the BatchScanners would be more
> accurate.
>
> Although, I only had one TServer, so I don't *think* there would be any
> difference. I don't believe we have concurrent requests from one
> BatchScanner to one TServer.

There are, if the batch scanner sees it has extra threads and there
are multiple tablets on the tserver, then it will submit concurrent
request to a single tserver.

>
> Dylan Hutchison wrote:
>>
>> Nice setup Josh.  Thank you for putting together the tests.  A few
>> questions:
>>
>> The serial scanner implementation uses 6 threads: one for each thread in
>> the thread pool.
>> The batch scanner implementation uses 60 threads: 10 for each thread in
>> the thread pool, since the BatchScanner was configured with 10 threads
>> and there are 10 (9?) tablets.
>>
>> Isn't 60 threads of communication naturally inefficient?  I wonder if we
>> would see the same performance if we set each BatchScanner to use 1 or 2
>> threads.
>>
>> Maybe this would motivate a /MultiTableBatchScanner/, which maintains a
>> fixed number of threads across any number of concurrent scans, possibly
>> to the same table.
>>
>>
>> On Sat, Sep 10, 2016 at 3:01 PM, Josh Elser > > wrote:
>>
>> Sven, et al:
>>
>> So, it would appear that I have been able to reproduce this one
>> (better late than never, I guess...). tl;dr Serially using Scanners
>> to do point lookups instead of a BatchScanner is ~20x faster. This
>> sounds like a pretty serious performance issue to me.
>>
>> Here's a general outline for what I did.
>>
>> * Accumulo 1.8.0
>> * Created a table with 1M rows, each row with 10 columns using YCSB
>> (workloada)
>> * Split the table into 9 tablets
>> * Computed the set of all rows in the table
>>
>> For a number of iterations:
>> * Shuffle this set of rows
>> * Choose the first N rows
>> * Construct an equivalent set of Ranges from the set of Rows,
>> choosing a random column (0-9)
>> * Partition the N rows into X collections
>> * Submit X tasks to query one partition of the N rows (to a thread
>> pool with X fixed threads)
>>
>> I have two implementations of these tasks. One, where all ranges in
>> a partition are executed via one BatchWriter. A second where each
>> range is executed in serial using a Scanner. The numbers speak for
>> themselves.
>>
>> ** BatchScanners **
>> 2016-09-10 17:51:38,811 [joshelser.YcsbBatchScanner] INFO : Shuffled
>> all rows
>> 2016-09-10 17:51:38,843 [joshelser.YcsbBatchScanner] INFO : All
>> ranges calculated: 3000 ranges found
>> 2016-09-10 17:51:38,846 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 40178 ms
>> 2016-09-10 17:52:19,025 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 42296 ms
>> 2016-09-10 17:53:01,321 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:53:47,414 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 46094 ms
>> 2016-09-10 17:53:47,415 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:54:35,118 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 47704 ms
>> 2016-09-10 17:54:35,119 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:55:24,339 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 49221 ms
>>
>> ** Scanners **
>> 2016-09-10 17:57:23,867 [joshelser.YcsbBatchScanner] INFO : Shuffled
>> all rows
>> 2016-09-10 17:57:23,898 [joshelser.YcsbBatchScanner] INFO : All
>> ranges calculated: 3000 ranges found
>> 2016-09-10 17:57:23,903 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 2833 ms
>> 2016-09-10 17:57:26,738 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 2536 ms
>> 2016-09-10 17:57:29,275 [joshelser.YcsbBatchScanner] INFO :
>> Executing 6 range partitions using a pool of 6 threads
>> 2016-09-10 17:57:31,425 [joshelser.YcsbBatchScanner] INFO : Queries
>> executed in 2150 ms
>> 2016-09-10 17:57:31,425 

Re: Accumulo Seek performance

[Keith Turner]

On Wed, Aug 24, 2016 at 9:22 AM, Sven Hodapp
 wrote:
> Hi there,
>
> currently we're experimenting with a two node Accumulo cluster (two tablet 
> servers) setup for document storage.
> This documents are decomposed up to the sentence level.
>
> Now I'm using a BatchScanner to assemble the full document like this:
>
> val bscan = instance.createBatchScanner(ARTIFACTS, auths, 10) // 
> ARTIFACTS table currently hosts ~30GB data, ~200M entries on ~45 tablets
> bscan.setRanges(ranges)  // there are like 3000 Range.exact's in the 
> ranges-list
>   for (entry <- bscan.asScala) yield {
> val key = entry.getKey()
> val value = entry.getValue()
> // etc.
>   }
>
> For larger full documents (e.g. 3000 exact ranges), this operation will take 
> about 12 seconds.
> But shorter documents are assembled blazing fast...
>
> Is that to much for a BatchScanner / I'm misusing the BatchScaner?
> Is that a normal time for such a (seek) operation?
> Can I do something to get a better seek performance?

How many threads did you configure the batch scanner with and did you
try varying this?

>
> Note: I have already enabled bloom filtering on that table.
>
> Thank you for any advice!
>
> Regards,
> Sven
>
> --
> Sven Hodapp, M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hod...@scai.fraunhofer.de
> www.scai.fraunhofer.de

Re: hard reboot... can not start accumulo

[Keith Turner]

Those instructions don't mention setting dataDir in zoo.cfg.  Well
actually a comment at the end of the article does.

On Fri, Aug 5, 2016 at 1:23 PM, Kevin Cho <kcho...@gmail.com> wrote:
> I've basically copy/paste this tutorial which have all the configurations.
>
> https://www.digitalocean.com/community/tutorials/how-to-install-the-big-data-friendly-apache-accumulo-nosql-database-on-ubuntu-14-04
>
> On Fri, Aug 5, 2016 at 5:20 PM, Keith Turner <ke...@deenlo.com> wrote:
>>
>> What if any Accumulo table or system configs are you setting?
>>
>> On Fri, Aug 5, 2016 at 12:05 PM, Kevin Cho <kcho...@gmail.com> wrote:
>> > Hi Mike,
>> >
>> > Thanks for helping.
>> > - What versions are you using?
>> > Accumulo  1.7.2
>> > Hadoop 2.7.2
>> > ZooKeeper 3.4.8
>> >
>> > - By reboot, do you mean rebooted the box?  How many nodes in your
>> > cluster?
>> > Just a single box.  Single node.
>> >
>> > - How did you determine the init was successful?
>> > I just run the 'accumulo init' after removing the /accumulo directory in
>> > hdfs.  Then I can start/stop the accumulo w/o any issue.  It's only when
>> > I
>> > start again after hard VM reboot.
>> >
>> > - What is currently happening on the master?
>> > I'm not sure how to answer this. I'm very new to accumulo.
>> >
>> > - What messages are showing up on the monitor?
>> > I'm not sure how to answer this. I'm very new to accumulo.
>> >
>> > - Can you send logs and jstacks?
>> > I'm not sure how to answer this. I'm very new to accumulo.
>> >
>> > I just freshed installed those versions above on Ubuntu 14.04.  I've
>> > reinstalled it couple of times and still had the same result.  It seems
>> > unless I execute stop-all.sh before reboot, I'm not able to start
>> > accumulo
>> > again after the reboot.
>> >
>> >
>> >
>> > On Fri, Aug 5, 2016 at 3:44 PM, Michael Wall <mjw...@gmail.com> wrote:
>> >>
>> >> Kevin,
>> >>
>> >> We are going to need more info.  Here are some things I can thing of.
>> >>
>> >> - What versions are you using?
>> >> - By reboot, do you mean rebooted the box?  How many nodes in your
>> >> cluster?
>> >> - How did you determine the init was successful?
>> >> - What is currently happening on the master?
>> >> - What messages are showing up on the monitor?
>> >> - Can you send logs and jstacks?
>> >>
>> >> Thanks,
>> >>
>> >> Mike
>> >>
>> >> On Fri, Aug 5, 2016 at 11:20 AM, Kevin Cho <kcho...@gmail.com> wrote:
>> >>>
>> >>> I was able to init accumulo and able to start/stop.  However, if I run
>> >>> the command 'reboot' and tried to start again.. I keep getting this
>> >>>
>> >>> Waiting for accumulo to be initialized
>> >>>
>> >>> It just keeps going and I'm not sure how to fix that.  Of course, I
>> >>> can
>> >>> re-init but I don't want to do that.  The reason why I'm testing with
>> >>> 'reboot' is that sometimes power gets rebooted or anyone w/ access can
>> >>> reboot as well.  Thanks for the help!
>> >>
>> >>
>> >
>
>

Re: Testing Spark Job that uses the AccumuloInputFormat

[Keith Turner]

Mario

A little bit of background, miniaccumulo cluster launches external
Accumulo processes.   To launch these processes it needs a classpath
for the JVM.  It tries to obtain that from the current classloader,
but thats failing.  The following code is causing your problem.   We
should open a bug about it not working w/ sbt.

https://github.com/apache/accumulo/blob/rel/1.7.2/minicluster/src/main/java/org/apache/accumulo/minicluster/impl/MiniAccumuloClusterImpl.java#L250

One possible work around is to use the accumulo maven plugin.  This
will launch mini accumulo outside of your test, then your test just
use that launched instance. Do you think this might work for you?  If
not maybe we can come up with another workaround.

http://accumulo.apache.org/release_notes/1.6.0#maven-plugin

As for the MiniDFSCluster issue, that should be ok.   We use mini
accumulo cluster to test Accumulo itself.  Some of this ends up
bringing in a dependency on MiniDFSCluster, even thought the public
API for mini cluster does not support using it.  We need to fix this,
so that there is no dependency on MiniDFSCluster.


Keith

On Wed, Aug 3, 2016 at 6:51 AM, Mario Pastorelli
 wrote:
> I'm trying to test a spark job that uses the AccumuloInputFormat but I'm
> having many issues with both MockInstance and MiniAccumuloCluster.
>
> 1) MockInstance doesn't work with Spark jobs in my environment because it
> looks like every task has a different instance of the MockInstance in
> memory; if I add records from the driver, the executors can't find this
> data. Is there a way to fix this?
>
> 2) MiniAccumuloCluster keeps giving strange errors. Two of them I can't
> really fix:
>   a. using sbt to run the tests throws  IllegalArgumentException Unknown
> classloader type : sbt.classpath.NullLoader when MiniAccumuloClister is
> instantiated. Anybody knows how to fix this? It's basically preventing me
> from using Spark and Accumulo together.
>   b. there is a warn that MiniDFSCluster is not found and a stub is used. I
> have all the dependencies needed, included hdfs test. Is this warn ok?
>
> Thanks for the help,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Re: AccumuloInputFormat and data locality for jobs that don't need keys sorted

[Keith Turner]

If you are not aware of it, something else to consider is the
setOfflineTableScan[1] option.  This can support much faster reads of
data.  In my experience this usually only useful for map only jobs
like you are doing.  When doing map/reduce the sort can make a speedup
in map read rate irrelevant.

You still may not get locality if tablets have multiple files because
a merged read of the files is done in the mapper.   Offline map reduce
in Accumulo attempts to run mappers at the last location a tablet
compacted some of its files.  Even w/o locality you still avoid the
cost of de-serializing , re-serializing, transmission, and
de-serializing data in the tserver+client.

[1]: 
http://accumulo.apache.org/1.7/apidocs/org/apache/accumulo/core/client/mapred/InputFormatBase.html#setOfflineTableScan%28org.apache.hadoop.mapred.JobConf,%20boolean%29

On Mon, Aug 1, 2016 at 7:55 PM, Mario Pastorelli
 wrote:
> I would like to use an Accumulo table as input for a Spark job. Let me
> clarify that my job doesn't need keys sorted and Accumulo is purely used to
> filter the input data thanks to it's index on the keys. The data that I need
> to process in Spark is still a small portion of the full dataset.
> I know that Accumulo provides the AccumuloInputFormat but in my tests almost
> no task has data locality when I use this input format which leads to poor
> performance. I'm not sure why this happens but my guess is that the
> AccumuloInputFormat creates one task per range.
> I wonder if there is a way to tell to the AccumuloInputFormat to split each
> range into the sub-ranges local to each tablet server so that each task in
> Spark will will read only data from the same machines where it is running.
>
> Thanks for the help,
> Mario
>
> --
> Mario Pastorelli | TERALYTICS
>
> software engineer
>
> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland
> phone: +41794381682
> email: mario.pastore...@teralytics.ch
> www.teralytics.net
>
> Company registration number: CH-020.3.037.709-7 | Trade register Canton
> Zurich
> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann
> de Vries
>
> This e-mail message contains confidential information which is for the sole
> attention and use of the intended recipient. Please notify us at once if you
> think that it may not be intended for you and delete it immediately.

Re: Unable to import RFile produced by AccumuloFileOutputFormat

[Keith Turner]

I have run into problems with creating a shaded jar for spark job and some
old version of hadoop client libs ending up in the shaded jar.

On Fri, Jul 8, 2016 at 6:03 PM, Keith Turner <ke...@deenlo.com> wrote:

> Is Accumulo writing RFiles to the encrypted HDFS instance and are those
> ok?  If only the spark job is having issue, maybe it using a different
> hadoop client lib or different hadoop config when it writes files.
>
> On Fri, Jul 8, 2016 at 5:29 PM, Russ Weeks <rwe...@newbrightidea.com>
> wrote:
>
>> Thanks, Bill and Josh! rfile-info was the tool I was looking for.
>>
>> It definitely seems related to HDFS encryption. I put a gist here showing
>> the results:
>>
>> https://gist.github.com/anonymous/13a8ba2c5f6794dc6207f1ae09b12825
>>
>> The two files contain exactly the same key-value pairs. They differ by 2
>> bytes in the footer of the RFile. The file written to the encrypted HDFS
>> directory is consistently corrupt - I'm not confident yet that it's always
>> corrupt in the same place because I see several different errors, but in
>> this case those 2 bytes were wrong.
>>
>> -Russ
>>
>> On Fri, Jul 8, 2016 at 12:30 PM Josh Elser <josh.el...@gmail.com> wrote:
>>
>>> Yeah, I'd lean towards something corrupting the file as well. We
>>> presently have two BCFile versions: 2.0 and 1.0. Both are presently
>>> supported by the code so it should not be possible to create a bad RFile
>>> using our APIs (assuming correctness from the filesystem, anyways)
>>>
>>> I'm reminded of HADOOP-11674, but a quick check does show that is fixed
>>> in your HDP-2.3.4 version (sorry for injecting $vendor here).
>>>
>>> Some other thoughts on how you could proceed:
>>>
>>> * Can Spark write the file to local fs? Maybe you can rule out HDFS w/
>>> encryption as a contributing issue by just writing directly to local
>>> disk and then upload them to HDFS after the fact (as a test)
>>> * `accumulo rfile-info` should fail in the same way if the metadata is
>>> busted as a way to verify things
>>> * You can use rfile-info on both files in HDFS and local fs (tying into
>>> the first point)
>>> * If you can share one of these files that is invalid, we can rip it
>>> apart and see what's going on.
>>>
>>> William Slacum wrote:
>>> > I wonder if the file isn't being decrypted properly. I don't see why it
>>> > would write out incompatible file versions.
>>> >
>>> > On Fri, Jul 8, 2016 at 3:02 PM, Josh Elser <josh.el...@gmail.com
>>> > <mailto:josh.el...@gmail.com>> wrote:
>>> >
>>> > Interesting! I have not run into this one before.
>>> >
>>> > You could use `accumulo rfile-info`, but I'd guess that would net
>>> > the same exception you see below.
>>> >
>>> > Let me see if I can dig a little into the code and come up with a
>>> > plausible explanation.
>>> >
>>> >
>>> > Russ Weeks wrote:
>>> >
>>> > Hi, folks,
>>> >
>>> > Has anybody ever encountered a problem where the RFiles that
>>> are
>>> > generated by AccumuloFileOutputFormat can't be imported using
>>> > TableOperations.importDirectory?
>>> >
>>> > I'm seeing this problem very frequently for small RFiles and
>>> > occasionally for larger RFiles. The errors shown in the
>>> > monitor's log UI
>>> > suggest a corrupt file, to me. For instance, the stack trace
>>> > below shows
>>> > a case where the BCFileVersion was incorrect, but sometimes it
>>> will
>>> > complain about an invalid length, negative offset, or invalid
>>> codec.
>>> >
>>> > I'm using HDP Accumulo 1.7.0 (1.7.0.2.3.4.12-1) on an
>>> encrypted HDFS
>>> > volume, with Kerberos turned on. The RFiles are generated by
>>> > AccumuloFileOutputFormat from a Spark job.
>>> >
>>> > A very small RFile that exhibits this problem is available
>>> here:
>>> >
>>> http://firebar.newbrightidea.com/downloads/bad_rfiles/Iwaz.rf
>>> >
>>> > I'm pretty confident that the keys are being written to the
>>> RFile in
>>> > order. Are there any tools

Re: Unable to import RFile produced by AccumuloFileOutputFormat

[Keith Turner]

Is Accumulo writing RFiles to the encrypted HDFS instance and are those
ok?  If only the spark job is having issue, maybe it using a different
hadoop client lib or different hadoop config when it writes files.

On Fri, Jul 8, 2016 at 5:29 PM, Russ Weeks  wrote:

> Thanks, Bill and Josh! rfile-info was the tool I was looking for.
>
> It definitely seems related to HDFS encryption. I put a gist here showing
> the results:
>
> https://gist.github.com/anonymous/13a8ba2c5f6794dc6207f1ae09b12825
>
> The two files contain exactly the same key-value pairs. They differ by 2
> bytes in the footer of the RFile. The file written to the encrypted HDFS
> directory is consistently corrupt - I'm not confident yet that it's always
> corrupt in the same place because I see several different errors, but in
> this case those 2 bytes were wrong.
>
> -Russ
>
> On Fri, Jul 8, 2016 at 12:30 PM Josh Elser  wrote:
>
>> Yeah, I'd lean towards something corrupting the file as well. We
>> presently have two BCFile versions: 2.0 and 1.0. Both are presently
>> supported by the code so it should not be possible to create a bad RFile
>> using our APIs (assuming correctness from the filesystem, anyways)
>>
>> I'm reminded of HADOOP-11674, but a quick check does show that is fixed
>> in your HDP-2.3.4 version (sorry for injecting $vendor here).
>>
>> Some other thoughts on how you could proceed:
>>
>> * Can Spark write the file to local fs? Maybe you can rule out HDFS w/
>> encryption as a contributing issue by just writing directly to local
>> disk and then upload them to HDFS after the fact (as a test)
>> * `accumulo rfile-info` should fail in the same way if the metadata is
>> busted as a way to verify things
>> * You can use rfile-info on both files in HDFS and local fs (tying into
>> the first point)
>> * If you can share one of these files that is invalid, we can rip it
>> apart and see what's going on.
>>
>> William Slacum wrote:
>> > I wonder if the file isn't being decrypted properly. I don't see why it
>> > would write out incompatible file versions.
>> >
>> > On Fri, Jul 8, 2016 at 3:02 PM, Josh Elser > > > wrote:
>> >
>> > Interesting! I have not run into this one before.
>> >
>> > You could use `accumulo rfile-info`, but I'd guess that would net
>> > the same exception you see below.
>> >
>> > Let me see if I can dig a little into the code and come up with a
>> > plausible explanation.
>> >
>> >
>> > Russ Weeks wrote:
>> >
>> > Hi, folks,
>> >
>> > Has anybody ever encountered a problem where the RFiles that are
>> > generated by AccumuloFileOutputFormat can't be imported using
>> > TableOperations.importDirectory?
>> >
>> > I'm seeing this problem very frequently for small RFiles and
>> > occasionally for larger RFiles. The errors shown in the
>> > monitor's log UI
>> > suggest a corrupt file, to me. For instance, the stack trace
>> > below shows
>> > a case where the BCFileVersion was incorrect, but sometimes it
>> will
>> > complain about an invalid length, negative offset, or invalid
>> codec.
>> >
>> > I'm using HDP Accumulo 1.7.0 (1.7.0.2.3.4.12-1) on an encrypted
>> HDFS
>> > volume, with Kerberos turned on. The RFiles are generated by
>> > AccumuloFileOutputFormat from a Spark job.
>> >
>> > A very small RFile that exhibits this problem is available here:
>> >
>> http://firebar.newbrightidea.com/downloads/bad_rfiles/Iwaz.rf
>> >
>> > I'm pretty confident that the keys are being written to the
>> RFile in
>> > order. Are there any tools I could use to inspect the internal
>> > structure
>> > of the RFile?
>> >
>> > Thanks,
>> > -Russ
>> >
>> > Unable to find tablets that overlap file
>> > hdfs://[redacted]/accumulo/data/tables/f/b-ze9/Izeb.rf
>> > java.lang.RuntimeException: Incompatible BCFile
>> fileBCFileVersion.
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Reader.(BCFile.java:828)
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.init(CachableBlockFile.java:246)
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBCFile(CachableBlockFile.java:257)
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.access$100(CachableBlockFile.java:137)
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader$MetaBlockLoader.get(CachableBlockFile.java:209)
>> > at
>> >
>>  
>> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Reader.getBlock(CachableBlockFile.java:313)
>> > at
>> >
>>  
>> 

Re: visibility constraint performance

[Keith Turner]

On Thu, Jun 9, 2016 at 2:49 PM, Jonathan Wonders 
wrote:

> Hi All,
>
> I've been tracking down some performance issues on a few 1.6.x
> environments and noticed some interesting and potentially undesirable
> behavior in the visibility constraint.  When the associated visibility
> evaluator checks a token to see if the user has a matching authorization,
> it uses the security operations from the tablet server's constraint
> environment which ends up authenticating the user's credentials on each
> call.  This will end up flooding logs with Audit messages corresponding to
> these authentications if the Audit logging is enabled.  It also consumes a
> non-negligible amount of CPU, produces a lot of garbage (maybe 50-60% of
> that generated under a heavy streaming ingest load), and can cause some
> contention between client pool threads when accessing the ZooCache.
>
> My initial measurements indicate a 25-30% decrease in ingest rate
> (entries/s and MB/s) for my environement and workload when this constraint
> is enabled.  This is with the Audit logging disabled.
>
> Is this intended behavior?  It seems like the authentication is redundant
> with the authentication that is performed at the beginning of the update
> session.
>

No. It would be best to avoid that behavior.


>
> Thanks,
> --Jonathan
>

Completed 2nd long run of Fluo

[Keith Turner]

A 2nd multi-day test of Fluo was completed on EC2 a week or two ago.  I
just got around to finishing writing it up. The test was run using Accumulo
1.7.1.

http://fluo.io/blog/2016/05/17/webindex-long-run-2

Re: I have a problem about change HDFS address

[Keith Turner]

Do you seen any errors in the Accumulo master log?

On Wed, May 25, 2016 at 11:17 AM, Lu Qin <luq.j...@gmail.com> wrote:

>
> I only use new HDFS, I change the instance.volumes to the new ,and the
> instance.volumes.replcaements.
> When accumulo start , I exec ./bin/accumulo
> org.apache.accumulo.server.util.FindOfflineTablets ,it shows the
> accumulo.root table UNASSIGNED
>
>
> 在 2016年5月25日，22:04，Keith Turner <ke...@deenlo.com> 写道：
>
> Accumulo stores data in HDFS and Zookeeper.   Are you using new zookeeper
> servers?  If so, did you copy zookeepers data?
>
> On Wed, May 25, 2016 at 4:08 AM, Lu Qin <luq.j...@gmail.com> wrote:
>
>> I have a accumulo 1.7.1 work with a old HDFS 2.6. Now I have a new HDFS
>> 2.6,and change the accumulo volume to the new.
>>
>> I have use distcp move the data from old HDFS to the new HDFS.And start
>> the accumulo up.
>>
>> Now the ‘Accumulo Overview' shows the Tables is 0 and Tablets is 0 with
>> red background, but In  'Table Status’ I can see all tables I have.
>> I use bin/accumulo shell and tables command,it also show all tables,but I
>> can not scan anyone.
>>
>> How can I resolve it?Thanks
>>
>
>
>

Re: I have a problem about change HDFS address

[Keith Turner]

Accumulo stores data in HDFS and Zookeeper.   Are you using new zookeeper
servers?  If so, did you copy zookeepers data?

On Wed, May 25, 2016 at 4:08 AM, Lu Qin  wrote:

> I have a accumulo 1.7.1 work with a old HDFS 2.6. Now I have a new HDFS
> 2.6,and change the accumulo volume to the new.
>
> I have use distcp move the data from old HDFS to the new HDFS.And start
> the accumulo up.
>
> Now the ‘Accumulo Overview' shows the Tables is 0 and Tablets is 0 with
> red background, but In  'Table Status’ I can see all tables I have.
> I use bin/accumulo shell and tables command,it also show all tables,but I
> can not scan anyone.
>
> How can I resolve it?Thanks
>

Re: Feedback about techniques for tuning batch scanning for my problem

[Keith Turner]

Until ACCUMULO-4164[1] is released you may see a performance difference
between 1 and 2 rfile level indexes.  However this depends on your #of
seeks to scan ratio.  If you seek to a spot and read a good bit of data
from there, it probably will not matter.

Accumulo caches a certain number of open rfiles[2].  For these cached open
rfiles the root level node is kept in memory. However, levels of the index
below the root are always read from index cache.  This is where the change
in ACCUMULO-4164 to avoid a copy from index cache helps.  So you may want
to consider ensuring the caching of open rfile is large enough and that the
index depth is one (for now).

[1]: https://issues.apache.org/jira/browse/ACCUMULO-4164
[2]:
http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_scan_files_open_max

On Mon, May 23, 2016 at 11:48 AM, Mario Pastorelli <
mario.pastore...@teralytics.ch> wrote:

> This is interesting and I think that's the main parameters that I need to
> tune. We have a lot of memory to spare for Accumulo so the index should fix
> in memory, but I'll check when I change those parameters. With small
> records, I think it makes sense to have smaller blocks for faster lookups.
> Thank you!
>
> On Mon, May 23, 2016 at 5:16 PM, Keith Turner <ke...@deenlo.com> wrote:
>
>> Mario
>>
>> You could experiment with adjusting table.file.compress.blocksize[1] and
>> table.file.compress.blocksize.index[2].  Making the data block size smaller
>> will increase the files index size and may lower random seek times
>> depending on how much of the index fits in memory.  Adjusting the index
>> block size will control the depth of the index tree.   You can adjust these
>> settings, compact, and run "accumulo rfile-info" on a file to see
>> information about the index size, depth, and number of data blocks.  You
>> can also adjust the index[6] and data[5] cache size settings and turn
>> data[3] and index[4] caching on or off for your table.
>>
>> [1]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_file_compress_blocksize
>> [2]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_file_compress_blocksize_index
>> [3]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_cache_block_enable
>> [4]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_table_cache_index_enable
>> [5]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_cache_data_size
>> [6]:
>> http://accumulo.apache.org/1.7/accumulo_user_manual#_tserver_cache_index_size
>>
>> Keith
>>
>> On Thu, May 19, 2016 at 11:08 AM, Mario Pastorelli <
>> mario.pastore...@teralytics.ch> wrote:
>>
>>> Hey people,
>>> I'm trying to tune a bit the query performance to see how fast it can go
>>> and I thought it would be great to have comments from the community. The
>>> problem that I'm trying to solve in Accumulo is the following: we want to
>>> store the entities that have been in a certain location in a certain day.
>>> The location is a Long and the entity id is a Long. I want to be able to
>>> scan ~1M of rows in few seconds, possibly less than one. Right now, I'm
>>> doing the following things:
>>>
>>>1. I'm using a sharding byte at the start of the rowId to keep the
>>>data in the same range distributed in the cluster
>>>2. all the records are encoded, one single record is composed by
>>>   1. rowId: 1 shard byte + 3 bytes for the day
>>>   2. column family: 8 byte for the long corresponding to the hash
>>>   of the location
>>>   3. column qualifier: 8 byte corresponding to the identifier of
>>>   the entity
>>>   4. value: 2 bytes for some additional information
>>>3. I use a batch scanner because I don't need sorting and it's faster
>>>
>>> As expected, it takes few seconds to scan 1M rows but now I'm wondering
>>> if I can improve it. My ideas are the following:
>>>
>>>1. set table.compaction.major.ration to 1 because I don't care about
>>>the ingestion performance and this should improve the query performance
>>>2. pre-split tables to match the number of servers and then use a
>>>byte of shard as first byte of the rowId. This should improve both 
>>> writing
>>>and reading the data because both should work in parallel for what I
>>>understood
>>>3. enable bloom filter on the table
>>>
>>> Do you think those ideas make sense? Furthermore, I have two questions:
>>>
>>>1. considering that a singl

Re: Bug in either InMemoryMap or NativeMap

[Keith Turner]

On Fri, Feb 19, 2016 at 5:59 PM, Dan Blum <db...@bbn.com> wrote:

> We are already using logical time. I can definitely change to using
> multiple mutations or more likely sum up the values myself and make a
> single put call.
>

That sounds like a good way to go.  I created an issue.

https://issues.apache.org/jira/browse/ACCUMULO-4148


>
>
> *From:* Keith Turner [mailto:ke...@deenlo.com]
> *Sent:* Friday, February 19, 2016 5:57 PM
> *To:* user@accumulo.apache.org
>
> *Subject:* Re: Bug in either InMemoryMap or NativeMap
>
>
>
>
>
>
>
> On Fri, Feb 19, 2016 at 5:14 PM, Dan Blum <db...@bbn.com> wrote:
>
> Yes, please open an issue for this.
>
>
>
> In the meantime, as a workaround is it safe to assign an arbitrary
> increasing timestamp when calling Mutation.put()? That seems the simplest
> way to get the ColumnUpdates to be treated properly.
>
>
>
> Seems like that would work, but then you may have to keep track of the
> next timestamp across processes.
>
> A possible alternative is to configure the table to use logical time and
> multiple mutations.  Logical time ensures every mutation is assigned a
> unique timestamp. The following program is an example of this.
>
> String table = getUniqueNames(1)[0];
> Connector c = getConnector();
> c.tableOperations().create(table,
> new
> NewTableConfiguration().setTimeType(TimeType.LOGICAL).withoutDefaultIterators());
>
> BatchWriterConfig config = new BatchWriterConfig();
> BatchWriter writer = c.createBatchWriter(table, config);
>
> Mutation m = new Mutation("row");
> m.put("cf1", "cq1", new Value("abc".getBytes()));
> writer.addMutation(m);
> m = new Mutation("row");
> m.put("cf1", "cq1", new Value("xyz".getBytes()));
> writer.addMutation(m);
> writer.close();
>
> Scanner scanner = c.createScanner(table, Authorizations.EMPTY);
> for (Entry<Key,Value> entry : scanner) {
>   System.out.println(entry);
> }
>
>
> This program prints
>
>   row cf1:cq1 [] 2 false=xyz
>   row cf1:cq1 [] 1 false=abc
>
>
>
> Accumulo assigned the timestamps 1 and 2.In this case Accumulo will
> keep track of the next timestamp for you.
>
>
>
> If you do not use logical time, then the two mutations would likely get
> the same timestamp because they arrived in the same millisecond.
>
>
>
> *From:* Keith Turner [mailto:ke...@deenlo.com]
> *Sent:* Friday, February 19, 2016 5:11 PM
> *To:* user@accumulo.apache.org
> *Cc:* Jonathan Lasko; Maxwell Jordan; kstud...@bbn.com
> *Subject:* Re: Bug in either InMemoryMap or NativeMap
>
>
>
>
>
>
>
> On Fri, Feb 19, 2016 at 3:34 PM, Dan Blum <db...@bbn.com> wrote:
>
> (Resend: I forgot to actually subscribe before sending originally.)
>
> I noticed a difference in behavior between our cluster and our tests
> running
> on MiniCluster: when multiple put() calls are made to a Mutation with the
> same CF, CQ, and CV and no explicit timestamp, on a live cluster only the
> last one is written, whereas in Mini all of them are.
>
> Of course in most cases it wouldn't matter but if there is a Combiner set
> on
> the column (which is the case I am dealing with) then it does.
>
> I believe the difference in behavior is due to code in NativeMap._mutate
> and
> InMemoryMap.DefaultMap.mutate. In the former if there are multiple
> ColumnUpdates in a Mutation they all get written with the same
> mutationCount
> value; I haven't looked at the C++ map code but I assume that this means
> that entries with the same CF/CQ/CV/timestamp will overwrite each other. In
> contrast, in DefaultMap multiple ColumnUpdates are stored with an
> incrementing kvCount, so the keys will necessarily be distinct.
>
>
>
> You made this issue easy to track down.
>
>
>
> This seems like a bug w/ the native map.  The code allocates a unique int
> for each key/value in the mutation.
>
>
>
> https://github.com/apache/accumulo/blob/rel/1.6.5/server/tserver/src/main/java/org/apache/accumulo/tserver/InMemoryMap.java#L476
>
>
> It seems like the native map code should increment like the DefaultMap
> code does.  Specifically it seems like the following code should increment
> mutationCount (coordinating with the code that calls it)
>
>
> https://github.com/apache/accumulo/blob/rel/1.6.5/server/tserver/src/main/java/org/apache/accumulo/tserver/NativeMap.java#L532
>
>
>
> Would you like to open an issue in Jira?
>
>
>
>
> My main question is: which of these is the intended behavior? We'll
> obv

Re: Fluo - Use cases

[Keith Turner]

Tom

A summing combiner allows you to do one large join operation.  Fluo allows
you to incrementally do a series of large join operations where one join
feeds data to other join operations.   This is analagous to a series of map
reduce jobs where one jobs feeds input to the next.

Webindex[1] is an  example that illustrates this.   This example counts
incoming links to a page and then indexes those counts multiple ways.  The
overview section of this blog post[2] provides a basic summary of some the
functionality of the example.

Looking at the example in that overview, notice how it mentions computing
the incoming link count for http://B.org as 2.   Computing (row=http://B.org
val=2) is something you could do with a SummingCombiner.  However doing the
next step mentioned in the blog post would be much more difficult.
Whenever the count for a link changes, it updates it in three different
indexes.  So when (row=http://B.org val=2) changes to (row=http://B.org
val=3), this triggers multiple follow in operations.   One of these follow
on operations udpates an indexes that tracks which link has the most
inbound links.  For this index the following changes are made.

 * Delete row=t:(999-2):http://B.org
 * Insert row=t:(999-3):http://B.org

Using Fluo, the changes to the indexes are made a fault tolerant manner.
The new index entry will not be inserted without deleting the previous one.

Keith


[1]: https://github.com/fluo-io/webindex
[2]: http://fluo.io/webindex-long-run/

On Sat, Jan 23, 2016 at 7:38 AM, Tom D  wrote:

> Hi,
>
> Been reading a bit about Fluo.
>
> I'm not 100% sure I grasp the use-cases where you'd want to use Fluo.  For
> example, counting links to a URL:
>
> http://accumulosummit.com/program/talks/using-fluo-to-incrementally-process-data-in-accumulo/
>
> What are the reasons you'd want to use Fluo for this purpose rather than a
> SummingCombiner iterator?
>
> Many thanks.
>

[ANNOUNCE] Fluo 1.0.0-beta-2 is released

[Keith Turner]

The Fluo project is happy to announce a 1.0.0-beta-2[1] release which is the
third release of Fluo and likely the final release before 1.0.0. Many
improvements in this release were driven by the creation of two new Fluo
related projects:

  * Fluo recipes[2] is a collection of common development patterns designed
to
make Fluo application development easier. Creating Fluo recipes required
new Fluo functionality and updates to the Fluo API. The first release of
Fluo recipes has been made and is available in Maven Central.

  * WebIndex[3] is an example Fluo application that indexes links to web
pages
in multiple ways. Webindex enabled the testing of Fluo on real data at
scale.  It also inspired improvements to Fluo to allow it to work better
with Apache Spark.

Fluo is now at a point where its two cluster test suites, Webindex[3] and
Stress[4], are running well for long periods on Amazon EC2. We invite early
adopters to try out the beta-2 release and help flush out problems before
1.0.0.

[1]: http://fluo.io/1.0.0-beta-2-release/
[2]: https://github.com/fluo-io/fluo-recipes
[3]: https://github.com/fluo-io/webindex
[4]: https://github.com/fluo-io/fluo-stress

Three day Fluo Common Crawl test

[Keith Turner]

We just completed a three day test of Fluo using Common Crawl data that
went pretty well.

http://fluo.io/webindex-long-run/

Re: Map Lexicoder

[Keith Turner]

Do you want all maps with the same keys to sort together?  If so doing
abc123 would make that happen.

The map data below

  { a : 1 , b : 2, c : 3 }
  { a : 1 , x : 8, y : 9 }
  { a : 4 , b : 5, c : 6 }

Would sort like the following if encoding key values pairs

  a1b2c3
  a1x8y9
  a4b5c6

If encoding all key and then all values, it would sort as follows

  abc123
  abc456
  axy189



On Tue, Dec 29, 2015 at 4:53 PM, Adam J. Shook <adamjsh...@gmail.com> wrote:

> Agreed, I came to the same conclusion while implementing.  The final
> result that I have is a SortedMapLexicoder to avoid any comparisons going
> haywire.  Additionally, would it be best to encode the map as an array of
> keys followed by an array of values, or encode all key value pairs
> back-to-back:
>
> { a : 1 , b : 2, c : 3 } encoded as
>
> a1b2c3
> -or-
> abc123
>
> Feels like I should be encoding a list of keys, then the list of values,
> and then concatenating these two encoded byte arrays.  I think the end
> solution will be to support both?  I'm having a hard time reconciling which
> method is better, if any.  Hard to find some good examples of people who
> are sorting a list of maps.
>
> On Tue, Dec 29, 2015 at 2:47 PM, Keith Turner <ke...@deenlo.com> wrote:
>
>>
>>
>> On Mon, Dec 28, 2015 at 11:47 AM, Adam J. Shook <adamjsh...@gmail.com>
>> wrote:
>>
>>> Hello all,
>>>
>>> Any suggestions for using a Map Lexicoder (or implementing one)?  I am
>>> currently using a new ListLexicoder(new PairLexicoder(some lexicoder, some
>>> lexicoder), which is working for single maps.  However, when one of the
>>> lexicoders in the Pair is itself a Map (and therefore another
>>> ListLexicoder(PairLexicoder)), an exception is being thrown because
>>> ArrayList is not Comparable.
>>>
>>
>>
>> Since maps do not have a well defined order of keys and values,
>> comparison is tricky.   The purpose of Lexicoders is encode things in such
>> a way that the lexicographical comparison of the serialized data is
>> possible.  With a hashmap if I add the same data in the same order to two
>> different hash map instances, its possible that when iterating over those
>> maps I could see the data in different orders.   This could lead to two
>> maps constructed in the same way at different times (like different JVMs
>> with different implementations of HashMap) generating different data that
>> compare as different.  Ideally comparison of the two would yield equality.
>>
>> Something like LinkedHashMap does not have this problem for the same
>> insertion order.  If you want things to be comparable regardless of
>> insertion order (which I think is more intuitive), then SortedMap seems
>> like it would be a good candidate.  So maybe a SortedMapLexicoder would be
>> a better thing to offer?
>>
>>
>>> Regards,
>>> --Adam
>>>
>>
>>
>

Re: Map Lexicoder

[Keith Turner]

On Mon, Dec 28, 2015 at 11:47 AM, Adam J. Shook 
wrote:

> Hello all,
>
> Any suggestions for using a Map Lexicoder (or implementing one)?  I am
> currently using a new ListLexicoder(new PairLexicoder(some lexicoder, some
> lexicoder), which is working for single maps.  However, when one of the
> lexicoders in the Pair is itself a Map (and therefore another
> ListLexicoder(PairLexicoder)), an exception is being thrown because
> ArrayList is not Comparable.
>


Since maps do not have a well defined order of keys and values, comparison
is tricky.   The purpose of Lexicoders is encode things in such a way that
the lexicographical comparison of the serialized data is possible.  With a
hashmap if I add the same data in the same order to two different hash map
instances, its possible that when iterating over those maps I could see the
data in different orders.   This could lead to two maps constructed in the
same way at different times (like different JVMs with different
implementations of HashMap) generating different data that compare as
different.  Ideally comparison of the two would yield equality.

Something like LinkedHashMap does not have this problem for the same
insertion order.  If you want things to be comparable regardless of
insertion order (which I think is more intuitive), then SortedMap seems
like it would be a good candidate.  So maybe a SortedMapLexicoder would be
a better thing to offer?


> Regards,
> --Adam
>

Re: Slow ReadProcessor (Degrade in performance)

[Keith Turner]

Mohit,

Were you able to figure this problem out?   Do you have enough info to know
if its a hdfs, Accumulo, network, or OS problem?

Keith

On Mon, Nov 9, 2015 at 7:12 AM, mohit.kaushik 
wrote:

>
> I have a application which query/inserts data in to Accumulo continuously.
> But the ingestion performance degrades after a day and continuously
> degrades by a considerable factor. I scanned all the hadoop and Accumulo
> logs but only found a warning in my two tservers logs after some inserts
> regularly.
>
> [hdfs.DFSClient] WARN : Slow ReadProcessor read fields took 30783ms
> (threshold=3ms); ack: seqno: 427381 reply: SUCCESS reply: SUCCESS
> reply: SUCCESS downstreamAckTimeNanos: 66816204 flag: 0 flag: 0 flag: 0,
> targets: 
> [DatanodeInfoWithStorage[192.168.10.122:50010,DS-188489f9-89d3-40bd-9d20-9db358d644c9,DISK],
> DatanodeInfoWithStorage[192.168.10.123:50010,DS-804290f9-edd0-45a4-ad2d-bb1bd336a348,DISK],
> DatanodeInfoWithStorage[192.168.10.121:50010,DS-dd6d6a25-122f-4958-a20b-4ccb82f49f11,DISK]]
>
>
> After monitoring the query time, It seems to be increased(for ex from 30
> ms to 3 sec)after a day and continues. I also get the connection reset
> exception regularly in Tserver logs in both read and write operations.
> Which I read that it comes when a connection is closed before scan or write
> completes. But how does it affect?
>
> 015-11-08 11:25:23,272 [util.CustomNonBlockingServer$CustomFrameBuffer]
> WARN : Got an IOException in internalRead!
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.thrift.transport.TNonblockingSocket.read(TNonblockingSocket.java:141)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.internalRead(AbstractNonblockingServer.java:537)
> at
> org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:203)
> at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.select(CustomNonBlockingServer.java:227)
> at
> org.apache.accumulo.server.util.CustomNonBlockingServer$SelectAcceptThread.run(CustomNonBlockingServer.java:183)
>
>
>  Please provide your inputs.
>
> -Mohit Kaushik
>
>
>

Re: Transaction type query in accumulo

[Keith Turner]

On Fri, Oct 30, 2015 at 7:36 AM, shweta.agrawal 
wrote:

> Hi,
>
> Is transaction type facility available in Accumulo?
> I have read about transaction in accumulo which says " Accumulo
> guarantees these ACID properties for a single mutation (a set of changes
> for a single row) but does not provide support for atomic updates across
> multiple rows"
>
> In my case:
> If one thread is updating the fields of a document then this document
> should be locked so that other thread cannot modify that document.
>
> I am trying to achieve this by a query through conditional mutation. I am
> checking whether the particular entry exist or not  then updating. But the
> problem is I am doing this through 150 threads. If one thread finds and
> updating particular entry then other thread should not get it.
>
> So is this the case in conditional write?
>

This could be done with a conditional write and a sequence number column
per row.

Could do something like the following

  * Read row using isolated scanner or whole row iterator.
  * Assume row has a column like sync:seq.
  * Make changes to row in client.
  * Write conditional mutation with condition that sync:seq is what was
read earlier.  Also increment the sequence number

With this pattern only one client can succesfully read-modify-write a row
concurrently.  In the case where the column synch:seq does not exist, the
conditional mutation should assert it does not exist and set it to an
initial value.


Is this what you are looking for?  Or are you trying to do something else?


>
> We are achieving same thing through mongoDB by find and modify feature.
>
> If one thread get particular document to update from conditional write
> then other thread should get that particular document.
>
> Please provide your inputs
>
> Thanks
> Shweta
>

Re: pre-sorting row keys vs not pre-sorting row keys

[Keith Turner]

I think the batch writer does sort mutations to bin them by tablet.

Did you consider JIT in your testing?  If one part of the test ran after
JIT it would be much faster because of that.

Also are you measuring the sort time and adding that to the test where you
pass sorted data?

On Thu, Oct 29, 2015 at 3:30 PM, Ara Ebrahimi 
wrote:

> Hi,
>
> We just did a simple test:
>
> - insert 10k batches of columns
> - sort the same 10k batch based on row keys and insert
>
> So basically the batch writer in the first test has items in non-sorted
> order and in the second one in sorted order. We noticed 50% better
> performance in the sorted version! Why is that the case? Is this something
> we need to consider doing for live ingest scenarios?
>
> Thanks,
> Ara.
>
>
>
> 
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Thank you in
> advance for your cooperation.
>
> 
>

Re: Is there a sensible way to do this? Sequential Batch Scanner

[Keith Turner]

Will the results always fit into memory?  If so could put results from
batch scanner into ArrayList and sort it.

On Tue, Oct 27, 2015 at 6:21 PM, Rob Povey  wrote:

> What I want is something that behaves like a BatchScanner (I.e. Takes a
> collection of Ranges in a single RPC), but preserves the scan ordering.
> I understand this would greatly impact performance, but in my case I can
> manually partition my request on the client, and send one request per
> tablet.
> I can’t use scanners, because in some cases I have 10’s of thousands of
> none consecutive ranges.
> If I use a single threaded BatchScanner, and only request data from a
> single Tablet, am I guaranteed ordering?
> This appears to work correctly in my small tests (albeit slower than a
> single 1 thread Batch scanner call), but I don’t really want to have to
> rely on it if the semantic isn’t guaranteed.
> If not Is there another “efficient” way to do this.
>
> Thanks
>
> Rob Povey
>
>

Re: Scan vs Filter performance.

[Keith Turner]

I think you are missing the last one because next() calls super.next() at
the end AND your has hasTop() calls super.hasTop()

On Tue, Sep 29, 2015 at 3:45 PM, Moises Baly <moi...@spatially.com> wrote:

> Hi there,
>
> I'm writing a custom iterator, which essentially is obtaining a range of
> values using a slightly different way to compare the rows (for keeping in
> range). In one test, it should return every row in Accumulo, but it's
> missing the last one. The most important parts of the code would look like
> this:
>
> class CIterator extends WrappingIterator() {
>   private var emitKey: Key = _
>   private var emitValue: Value = _
>
>   override def deepCopy(env: IteratorEnvironment): 
> SortedKeyValueIterator[Key, Value] = {
> new CIterator(this, env)
>   }
>
>
>   def this(_this: CIterator, env: IteratorEnvironment) = {
> this()
> setSource(_this.getSource.deepCopy(env))
>   }
>
>   override def init(source: SortedKeyValueIterator[Key, Value], options: 
> util.Map[String, String], env: IteratorEnvironment) = {
> super.init(source, options, env)
>   }
>
>   override def getTopKey(): Key = {
> emitKey
>   }
>
>   override def getTopValue(): Value = {
> emitValue
>   }
>
>   override def hasTop(): Boolean = {
> super.hasTop
>   }
>
>   override def seek(range: Range, columnFamilies: 
> util.Collection[ByteSequence], inclusive: Boolean): Unit = {
>
> ...
>
> val seekRange = new Range(partialKeyStart.toString, true, 
> partialKeyEnd.toString, true)
>
> super.seek(seekRange, columnFamilies, inclusive);
>
> if (super.hasTop()) {
>   next();
> }
>   }
>
>   override def next(): Unit = {
> ...
> val lowerBoundCheck = rangeStart.compareTo(nextKey.getRow.toString)
> val upperBoundCheck = rangeEnd.compareTo(nextKey.getRow.toString)
> if (lowerBoundCheck <= 0 && upperBoundCheck >= 0){
>   emitKey = new Key(nextKey)
>   emitValue = new Value(nextValue)
>   if (super.hasTop()){
> super.next()
>   }
>
> }
>   }
> }
>
>
> So that code, if I have a range that comprises every row, returns every one 
> of them but the last one. A high level call list would look like this:
>
> Seek ->
> Next ->
> hasTop ->
> Top key ->
> Top key ->
> Top value ->
> hasTop ->
> Top key ->
> Top value ->
> Next ->
> hasTop ->
> Top key ->
> Top key ->
> Top value ->
> (print row - value 1) -> hasTop ->
> Top key ->
> Top value ->
> Next ->
> hasTop ->
> Top key ->
> Top key ->
> Top value -> (print row - value 2) -> hasTop ->
> Top key ->
> Top value ->
> Next ->
> hasTop -> (print row - value 3) ->  hasTop ->
>
> I think I'm missing something on the call tree:
>
> 1- Is it normal to have many subsequent topKey() calls after next()?
>
> 2- This is supposed to give me every row (the condition put in place for the 
> range is working), but as you can see, it stops after the last next() call, 
> for some reason (maybe something to do with the interfaces hierarchy?)
>
> 3- In general, what would be a correct approach (execution path) for building 
> a custom iterator? I'm still hesitant on how the iterator functions (next, 
> seek, getTop...) interact with each other, specially in the way we give back 
> results to clients.
>
> Thank you for your time,
>
>
> Moises
>
>
>
> On Tue, Sep 29, 2015 at 11:16 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>>
>>
>> On Tue, Sep 29, 2015 at 12:59 AM, mohit.kaushik <mohit.kaus...@orkash.com
>> > wrote:
>>
>>> Hi Keith,
>>>
>>> When we fetch a column or column family Ii seems, it does not seek and
>>> only scan by filtering the key/value pairs. But as you said if I design a
>>> custom iterator to fetch a column family, It may work faster.
>>>
>>
>> When column families are fetched, Accumulo will seek[1].  It tries to
>> read 10 cells and then seeks.
>>
>> When fetching family and qualifier, two iterators are used.  The
>> ColumnFamilySkippingIterator and ColumnQualifierFilter.  The
>> ColumnQualifierFilter does a scan of all qualifers within a family [2].
>> The system configures the qualifier filter to have the family skipping iter
>> as a source[3], so it could still seek between families.
>>
>>
>>>
>>> But I want to know what would be the scenario if I define a locality
>>> group for the column family and run the same custom iterator on it which
&

Re: Watching for Changes with Write Ahead Log?

[Keith Turner]

You can follow constrainChecker atomic ref.  Its been awhile, but I think
its only reloaded when constraint config changes.  Also multiple threads
may use a constraint.

https://github.com/apache/accumulo/blob/1.6.2/server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java#L2462

On Thu, Oct 1, 2015 at 3:03 PM, Parise, Jonathan <jonathan.par...@gd-ms.com>
wrote:

> Are the constraint threads reused? For example is there a thread pool or
> something the constraints use to run?
>
>
>
> I have the Accumulo source on my machine, but I am not sure where to look
> to figure this out. What class invokes the constraints?
>
>
>
> Thanks,
>
>
>
> Jon
>
>
>
> *From:* Keith Turner [mailto:ke...@deenlo.com]
> *Sent:* Thursday, October 01, 2015 3:02 PM
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> Could possibly use a ThreadLocal containing a SoftReference
>
> Another place you could possibly put this code instead of in a constraint
> is in a minor compaction iterator.   Although you will still have the clean
> up problem since iterators do not have a close method.  There is an open
> ticket where Billie suggested that Accumulo can call close on any iterator
> that implements Closeable
>
>
>
> On Thu, Oct 1, 2015 at 2:48 PM, Parise, Jonathan <
> jonathan.par...@gd-ms.com> wrote:
>
> I think this is one of those things where there really aren’t great
> solutions.
>
>
>
> A static connection could work, but if multiple Constraint instances can
> exist at the same time, it would probably not work. Since all of them would
> be trying to use the same connection at the same time.
>
>
>
> ThreadLocal could possibly work better. The only question is how long
> lived is the thread that calls the constraints? For example that thread
> could be torn down as soon as the constraint is done. In that case the
> performance would be no better than creating and tearing down everything
> each time check() is called.
>
>
>
> This is why I am trying to understand the Constraint’s lifecycle, so I can
> come up with the least bad way of solving this problem.
>
>
>
> Thanks for the ideas! I am just not sure I know enough about the lifecycle
> of Constraints to understand if these suggestions would be helpful.
>
>
>
> Jon Parise
>
>
>
> *From:* John Vines [mailto:vi...@apache.org]
> *Sent:* Thursday, October 01, 2015 2:40 PM
>
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> As dirty as it is, that sounds like a case for a static, or maybe thread
> local, object
>
>
>
> On Thu, Oct 1, 2015, 7:19 PM Parise, Jonathan <jonathan.par...@gd-ms.com>
> wrote:
>
> I have a few follow up questions in regard to constraints.
>
>
>
> What is the lifecycle of a constraint? What I mean by this is are the
> constraints somehow tied to Accumulo’s lifecycle or are they just
> instantiated each time a mutation occurs and then disposed?
>
>
>
> Also, are there multiple instances of the same constraint class at any
> time or do all mutation on a table go through the exact same constraint?
>
>
>
> My guess is that  when a mutation comes in a new constraint is made
> through reflection. Then check() is called, the violation codes are parsed
> and the object is disposed/finalized.
>
>
>
> The reason I ask is that what I want to do is update my ElasticSearch
> index each time I see a mutation on the table. However, I don’t want to
> have to make a connection, send the data and then tear down the connection
> each time. That’s a lot of unnecessary overhead and with all that overhead
> happening on every mutation performance could be badly impacted.
>
>
>
> Is there some way to cache something like a connection and reuse it
> between calls to the Constraint’s check() method? How would such a thing be
> cleaned up if Accumulo is shut down?
>
>
>
>
>
> Thanks again,
>
>
>
> Jon
>
> *From:* Parise, Jonathan [mailto:jonathan.par...@gd-ms.com
> <jonathan.par...@gd-ms.com>]
> *Sent:* Wednesday, September 30, 2015 9:21 AM
> *To:* user@accumulo.apache.org
> *Subject:* RE: Watching for Changes with Write Ahead Log?
>
>
>
> In this particular case, I need to update some of my application state
> when changes made by another system occur.
>
>
>
> I would need to do a few things to accomplish my goal.
>
>
>
> 1)  Be notified or see that a table had changed
>
> 2)  Checked that against changes I know my system has made
>
> 3)  If my system is not the originator of the 

Re: Watching for Changes with Write Ahead Log?

[Keith Turner]

Could possibly use a ThreadLocal containing a SoftReference

Another place you could possibly put this code instead of in a constraint
is in a minor compaction iterator.   Although you will still have the clean
up problem since iterators do not have a close method.  There is an open
ticket where Billie suggested that Accumulo can call close on any iterator
that implements Closeable

On Thu, Oct 1, 2015 at 2:48 PM, Parise, Jonathan 
wrote:

> I think this is one of those things where there really aren’t great
> solutions.
>
>
>
> A static connection could work, but if multiple Constraint instances can
> exist at the same time, it would probably not work. Since all of them would
> be trying to use the same connection at the same time.
>
>
>
> ThreadLocal could possibly work better. The only question is how long
> lived is the thread that calls the constraints? For example that thread
> could be torn down as soon as the constraint is done. In that case the
> performance would be no better than creating and tearing down everything
> each time check() is called.
>
>
>
> This is why I am trying to understand the Constraint’s lifecycle, so I can
> come up with the least bad way of solving this problem.
>
>
>
> Thanks for the ideas! I am just not sure I know enough about the lifecycle
> of Constraints to understand if these suggestions would be helpful.
>
>
>
> Jon Parise
>
>
>
> *From:* John Vines [mailto:vi...@apache.org]
> *Sent:* Thursday, October 01, 2015 2:40 PM
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> As dirty as it is, that sounds like a case for a static, or maybe thread
> local, object
>
>
>
> On Thu, Oct 1, 2015, 7:19 PM Parise, Jonathan 
> wrote:
>
> I have a few follow up questions in regard to constraints.
>
>
>
> What is the lifecycle of a constraint? What I mean by this is are the
> constraints somehow tied to Accumulo’s lifecycle or are they just
> instantiated each time a mutation occurs and then disposed?
>
>
>
> Also, are there multiple instances of the same constraint class at any
> time or do all mutation on a table go through the exact same constraint?
>
>
>
> My guess is that  when a mutation comes in a new constraint is made
> through reflection. Then check() is called, the violation codes are parsed
> and the object is disposed/finalized.
>
>
>
> The reason I ask is that what I want to do is update my ElasticSearch
> index each time I see a mutation on the table. However, I don’t want to
> have to make a connection, send the data and then tear down the connection
> each time. That’s a lot of unnecessary overhead and with all that overhead
> happening on every mutation performance could be badly impacted.
>
>
>
> Is there some way to cache something like a connection and reuse it
> between calls to the Constraint’s check() method? How would such a thing be
> cleaned up if Accumulo is shut down?
>
>
>
>
>
> Thanks again,
>
>
>
> Jon
>
> *From:* Parise, Jonathan [mailto:jonathan.par...@gd-ms.com
> ]
> *Sent:* Wednesday, September 30, 2015 9:21 AM
> *To:* user@accumulo.apache.org
> *Subject:* RE: Watching for Changes with Write Ahead Log?
>
>
>
> In this particular case, I need to update some of my application state
> when changes made by another system occur.
>
>
>
> I would need to do a few things to accomplish my goal.
>
>
>
> 1)  Be notified or see that a table had changed
>
> 2)  Checked that against changes I know my system has made
>
> 3)  If my system is not the originator of the change, update internal
> state to reflect the change.
>
>
>
> Examples of state I may need to update include an ElasticSearch index and
> also an in memory cache.
>
>
>
> I’m going to read up on constraints again and see if I can use them for
> this purpose.
>
>
>
> Thanks!
>
>
>
> Jon
>
>
>
>
>
>
>
> *From:* Adam Fuchs [mailto:afu...@apache.org ]
> *Sent:* Tuesday, September 29, 2015 5:46 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> Jon,
>
>
>
> You might think about putting a constraint on your table. I think the API
> for constraints is flexible enough for your purpose, but I'm not exactly
> sure how you would want to manage the results / side effects of your
> observations.
>
>
>
> Adam
>
>
>
>
>
> On Tue, Sep 29, 2015 at 5:41 PM, Parise, Jonathan <
> jonathan.par...@gd-ms.com> wrote:
>
> Hi,
>
>
>
> I’m working on a system where generally changes to Accumulo will come
> through that system. However, in some cases, another system may change data
> without my system being aware of it.
>
>
>
> What I would like to do is somehow listen for changes to the tables my
> system cares about. I know there is a write ahead log that I could
> potentially listen to for changes, but I don’t know how to use it. I looked
> around for some documentation about it, and I don’t see much. I get 

Re: Scan vs Filter performance.

[Keith Turner]

On Mon, Sep 28, 2015 at 12:19 PM, Moises Baly  wrote:

> Hi all:
>
> I would like to perform a range scan on a table, tweaking the definition
> of what goes into a particular key range. One way I can think of is writing
> a filter on the key, and that would work fine. But I think it would be slow
> compared to a scan / seek custom iterator. How does the underlying login
> works? Does Filter goes through all records, or since is sorted follows the
> same underlying logic as a scan? Would a custom iterator perform better?
>

Yes, filter will read all data.  Custom iterator that seeks may be faster.

Are you aware of the following?

https://issues.apache.org/jira/browse/ACCUMULO-3961
https://github.com/apache/accumulo/pull/42


>
> Thank you for your time,
>
> Moises
>

Re: Mini Accumulo Cluster reusing the directory

[Keith Turner]

On Thu, Sep 17, 2015 at 12:38 AM, Josh Elser <josh.el...@gmail.com> wrote:

> MiniAccumuloCluster by default already uses the local file system.
>
> To get the correct sync semantics that Keith mentioned you need to set
> "fs.file.impl" in the Hadoop Configuration you pass to the
> MiniAccumuloConfig.
>

Yeah thats the main thing I was thinking of.  Thanks for digging it up
Josh.   Its really important to set that hadoop config when having Accumulo
use the local FS.  Otherwise the default class hadoop uses for the local fs
does nothing for flush.

A few caveats to be aware of :

  * The default impl is LocalFileSystem and it used to do nothing for
flush().   That may no longer be true, at one point it extended something
from CheckSumFS which did nothing for flush.
  * RawLocalFileSystem may not checksum data like LocalFileSystem does
  *  RawLocalFileSystem may not survive a power outage.  It flushes the
file to the OS... but the OS may hold it in its buffers for some time
before really writing it to disk.  So this means you would survive
processes terminating, but not the OS terminating unexpectedly.   When
using HDFS Accumulo can be configured to make the hdfs output stream make
java.nio calls to flush data to disk.



> See
> https://github.com/apache/accumulo/blob/master/test/src/main/java/org/apache/accumulo/test/VolumeIT.java#L113
> for code.
>
> mohit.kaushik wrote:
>
>> Keith,
>>
>> How can I configure Accumulo to use local file system???
>>
>> On 09/17/2015 01:11 AM, Keith Turner wrote:
>>
>>> Would you be able to provide more informaiton about your use case?
>>> Was wondering if other solutions could be of use, like configuring
>>> regular Accumulo to use the local filesystem.  This can be done, but
>>> care needs to be taken to make walogs work correctly.   If interested
>>> I could provide more info about this configuration.
>>>
>>> On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp
>>> <sven.hod...@scai.fraunhofer.de
>>> <mailto:sven.hod...@scai.fraunhofer.de>> wrote:
>>>
>>> Hi there,
>>>
>>> is it possible for MiniAccumuloCluster to reuse a given directory?
>>> Sadly, I haven't found anything in the docs?
>>>
>>> I’ll fire up my instance like this:
>>>
>>>val dict = new File("/tmp/accumulo-mini-cluster")
>>>val accumulo = new MiniAccumuloCluster(dict, "test“)
>>>
>>> If I’ll restart my JVM it will raise a error like this:
>>>
>>>Exception in thread "main" java.lang.IllegalArgumentException:
>>> Directory /tmp/accumulo-mini-cluster is not empty
>>>
>>> It would be nice if the data can survive a JVM restart and the
>>> folder structure must not be constructed every time.
>>>
>>> Thanks a lot!
>>>
>>> Regards,
>>> Sven
>>>
>>> --
>>> Sven Hodapp M.Sc.,
>>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
>>> Department of Bioinformatics
>>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
>>> sven.hod...@scai.fraunhofer.de >> sven.hod...@scai.fraunhofer.de>
>>> www.scai.fraunhofer.de <http://www.scai.fraunhofer.de>
>>>
>>>
>>>
>>
>> --
>>
>> *Mohit Kaushik*
>> Software Engineer
>> A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
>> *Tel:*+91 (124) 4969352 | *Fax:*+91 (124) 4033553
>>
>> <http://politicomapper.orkash.com>interactive social intelligence at
>> work...
>>
>> <https://www.facebook.com/Orkash2012>
>> <http://www.linkedin.com/company/orkash-services-private-limited>
>> <https://twitter.com/Orkash> <http://www.orkash.com/blog/>
>> <http://www.orkash.com>
>> <http://www.orkash.com> ... ensuring Assurance in complexity and
>> uncertainty
>>
>> /This message including the attachments, if any, is a confidential
>> business communication. If you are not the intended recipient it may be
>> unlawful for you to read, copy, distribute, disclose or otherwise use
>> the information in this e-mail. If you have received it in error or are
>> not the intended recipient, please destroy it and notify the sender
>> immediately. Thank you /
>>
>>

Re: Mini Accumulo Cluster reusing the directory

[Keith Turner]

Would you be able to provide more informaiton about your use case?  Was
wondering if other solutions could be of use, like configuring regular
Accumulo to use the local filesystem.  This can be done, but care needs to
be taken to make walogs work correctly.   If interested I could provide
more info about this configuration.

On Wed, Sep 16, 2015 at 9:20 AM, Sven Hodapp  wrote:

> Hi there,
>
> is it possible for MiniAccumuloCluster to reuse a given directory?
> Sadly, I haven't found anything in the docs?
>
> I’ll fire up my instance like this:
>
>val dict = new File("/tmp/accumulo-mini-cluster")
>val accumulo = new MiniAccumuloCluster(dict, "test“)
>
> If I’ll restart my JVM it will raise a error like this:
>
>Exception in thread "main" java.lang.IllegalArgumentException:
> Directory /tmp/accumulo-mini-cluster is not empty
>
> It would be nice if the data can survive a JVM restart and the folder
> structure must not be constructed every time.
>
> Thanks a lot!
>
> Regards,
> Sven
>
> --
> Sven Hodapp M.Sc.,
> Fraunhofer Institute for Algorithms and Scientific Computing SCAI,
> Department of Bioinformatics
> Schloss Birlinghoven, 53754 Sankt Augustin, Germany
> sven.hod...@scai.fraunhofer.de
> www.scai.fraunhofer.de
>

Re: Accumulo ProxyInstance available

[Keith Turner]

Thats interesting.  Curious what you do in the case of faults?  I.e. if
proxy does down for a bit and then comes back up, or if network between
an Accumulo ProxyInstance client and the client goes down for a bit.

On Thu, Sep 3, 2015 at 7:33 AM, Patrone, Dennis S. <
dennis.patr...@jhuapl.edu> wrote:

> I just wanted to let folks know about the availability of a new project
> called 'Accumulo ProxyInstance'. It is a Java Instance implementation
> that communicates with an Accumulo cluster via the Thrift proxy server.
>
>
>
> Basically, we have an isolated Accumulo cluster only accessible to the
> rest of the network through a single, dual-homed gatekeeper machine.  We
> started the Thrift proxy server on the gatekeeper.  We then created this
> Instance implementation to allow developers to access the cluster from
> their development machines through the gatekeeper/proxy but still using the
> traditional Java Instance APIs.This encapsulates a majority of the
> client code from needing to know if it is running on the cluster or
> connecting through the proxy server (with the notable exception of the
> actual Instance instantiation).
>
>
>
> Now for convenience we can test, debug, and perform smaller queries using
> the ProxyInstance on our development machines through the proxy server.
> When performance demands it, we can simply change 1 line of code to
> instantiate a ZooKeeperInstance instead, move the code to the isolated
> subnet, and communicate directly with the tablet servers using the
> traditional Java client library.
>
>
>
> You can read more about the project here:
> http://jhuapl.github.io/accumulo-proxy-instance/proxy_instance_user_manual.html
> .  The code is available on github (
> https://github.com/JHUAPL/accumulo-proxy-instance) and the JAR is on
> maven central (
> http://search.maven.org/#artifactdetails|edu.jhuapl.accumulo|proxy-instance|1.0.0|jar).
> It was developed and tested against Accumulo 1.6.2.
>
>
>
> We’ve found it useful in our development environment given our isolated
> Accumulo cluster; perhaps others who have a similar setup might also find
> utility in it as well.
>
>
>
> Thanks,
>
> Dennis
>
>
>
>
>

Re: Accumulo ProxyInstance available

[Keith Turner]

On Thu, Sep 3, 2015 at 12:43 PM, Patrone, Dennis S. <
dennis.patr...@jhuapl.edu> wrote:

> I think in almost all cases you’d just get a RuntimeException that wraps
> the TTransportException.  We could certainly do a better job at (automatic)
> fault recovery than we are currently… especially in the “simple” cases
> where, for example, the proxy goes down and come back while you are not
> using the Instance code (a simple “reconnect” should suffice).In other
> cases, we might be able to provide a more API-friendly exception (e.g.,
> throw a MutationRejectedException if a push of Mutations fails due to a
> network problem).   But our use case was for convenience, not production—so
> we didn’t worry too much about failures other than to make sure we knew
> they happened.
>

Thanks for the explanation.  I don't have an opinion on what it should do
in this case, I was just curious what the behavior was.  I agree that
behavior makes sense for your use case.


>
>
> The proxy currently maintains the TTransport reference for the life of the
> Instance (rather than reconnecting on every request).  One of the (other)
> issues doing this is we need an explicit “close()” on the Instance (not in
> the Instance API) when you are done with it.  We toyed with the idea of
> creating/closing/destroying the transport on every access which would
> remove the need for an explicit close as well as handle many of the
> intermittent connectivity issues you raise.  But that would be at the cost
> of runtime performance in what is (hopefully) the average case.   Perhaps
> that performance hit would be OK (hey, if you really need it to work fast
> just the standard library!) but we tried to make the runtime at least
> tolerable! :-) Maybe someone has a brilliant idea of how we could handle it
> better…
>

Could close the TTransport after X time of inactivity and make this
synchronized w.r.t to getting a TTransport.   A Guava cache might be an
easy way to accomplish this.


>
>
> Thanks,
>
> Dennis
>
>
>
>
>
> *From:* Keith Turner [mailto:ke...@deenlo.com]
> *Sent:* Thursday, September 03, 2015 11:18 AM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Accumulo ProxyInstance available
>
>
>
> Thats interesting.  Curious what you do in the case of faults?  I.e. if
> proxy does down for a bit and then comes back up, or if network between
> an Accumulo ProxyInstance client and the client goes down for a bit.
>
>
>
> On Thu, Sep 3, 2015 at 7:33 AM, Patrone, Dennis S. <
> dennis.patr...@jhuapl.edu> wrote:
>
> I just wanted to let folks know about the availability of a new project
> called 'Accumulo ProxyInstance'. It is a Java Instance implementation that
> communicates with an Accumulo cluster via the Thrift proxy server.
>
>
>
> Basically, we have an isolated Accumulo cluster only accessible to the
> rest of the network through a single, dual-homed gatekeeper machine.  We
> started the Thrift proxy server on the gatekeeper.  We then created this
> Instance implementation to allow developers to access the cluster from
> their development machines through the gatekeeper/proxy but still using the
> traditional Java Instance APIs.This encapsulates a majority of the
> client code from needing to know if it is running on the cluster or
> connecting through the proxy server (with the notable exception of the
> actual Instance instantiation).
>
>
>
> Now for convenience we can test, debug, and perform smaller queries using
> the ProxyInstance on our development machines through the proxy server.
> When performance demands it, we can simply change 1 line of code to
> instantiate a ZooKeeperInstance instead, move the code to the isolated
> subnet, and communicate directly with the tablet servers using the
> traditional Java client library.
>
>
>
> You can read more about the project here:
> http://jhuapl.github.io/accumulo-proxy-instance/proxy_instance_user_manual.html
> .  The code is available on github (
> https://github.com/JHUAPL/accumulo-proxy-instance) and the JAR is on
> maven central (
> http://search.maven.org/#artifactdetails|edu.jhuapl.accumulo|proxy-instance|1.0.0|jar).
> It was developed and tested against Accumulo 1.6.2.
>
>
>
> We’ve found it useful in our development environment given our isolated
> Accumulo cluster; perhaps others who have a similar setup might also find
> utility in it as well.
>
>
>
> Thanks,
>
> Dennis
>
>
>
>
>
>
>

Re: How to control Minor Compaction by programming

[Keith Turner]

How many tablets do you have?  Entire tablets are minor compacted at once.
If you have 1 tablet per tablet server, then minor compactions will have a
lot of work to do at once.  While this work is being done, the tablet
servers memory may fill up, leading to writes being held.

If you have 10 tablets per tablet server, then tablets can be compacted in
parallel w/ less work to do at any given point in time.This can avoid
memory filling up and writes being held.

In short, its possible that adding good split points to the table (and
therefore creating more tablets) may help w/ this issue.

Also, are you seeing hold times?

On Thu, Jul 30, 2015 at 11:24 PM, Hai Pham htp0...@tigermail.auburn.edu
wrote:

 Hey William, Josh and David,

 Thanks for explaining, I might not have been clear: I used the web
 interface with port 50095 to monitor the real-time charts (ingest, scan,
 load average, minor compaction, major compaction, ...).

 Nonetheless, as I witnessed, when I ingested about 100k entries - then
 minor compaction happened - ingest was stuck - the level of minor
 compaction on the charts was just about 1.0, 2.0 and max 3.0 while about
 20k entries were forced out of memory (I knew this by looking at the
 number of entries in memory w.r.t the table being ingested to) - then when
 minor compaction ended, ingest resumed, somewhat faster.

 Thus I presume the level 1.0, 2.0, 3.0 is not representative for number of
 files being minor-compacted from memory?

 Hai
 
 From: Josh Elser josh.el...@gmail.com
 Sent: Thursday, July 30, 2015 7:12 PM
 To: user@accumulo.apache.org
 Subject: Re: How to control Minor Compaction by programming

 
  Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web
  monitoring) denoting the level of Minor Compaction and Major Compaction?

 On the monitor, the number of compactions are of the form:

 active (queued)

 e.g. 4 (2), would mean that 4 are running and 2 are queued.

 
 
  Thank you!
 
  Hai Pham
 
 
 
 

Re: Question regarding java being the choice for accumulo

[Keith Turner]

We have a small amount of C++ code used to manage all data written to
Accumulo.  Using C++ for this very small amount of high performance, memory
intensive code has worked out well.   Coding task that do not need to be
high performance in Java, like tablet assignment and management of cluster
state is nice.

On Wed, Jul 29, 2015 at 1:51 PM, vaibhav thapliyal 
vaibhav.thapliyal...@gmail.com wrote:

 Hello everyone,

 I was wondering why did the developers chose java for writing accumulo.
 What advantage it has over using any other language say C++(in which
 another popular nosql database MongoDB is written) in context of accumulo?

 Thanks
 Vaibhav

Re: How to get TabletId

[Keith Turner]

A tablet is uniquely identified by : (table id, end row, and previous end
row).  In the Accumulo internal code KeyExtent is used to hold this info.
Other internal code can extract KeyExtents from the the metadata table.
Are you interested in a pointer to this code? Its not public API, but I can
help you find the code if thats what you are looking for.

Inoder to avoid leaking sensitive on the monitor page, KeyExtent has a
method that returns hash((table id, end row, and previous end row).   This
hash could serve as an ID.  Or you could hash, encrypt, encode etc info
that yourself with a different  function.



On Fri, Jul 17, 2015 at 9:57 AM, Rukshan Chathuranga rcruksha...@gmail.com
wrote:

 I need to get tablet id. Just for visualizing id.

 Thanks and regards.
 On Jul 17, 2015 7:18 PM, Keith Turner ke...@deenlo.com wrote:

 What do you meant by table id?  What are you trying to accomplish?

 On Thu, Jul 16, 2015 at 11:19 AM, Rukshan Chathuranga 
 rcruksha...@gmail.com wrote:

 Hi,

 how can i get the id of each Tablets?

 Thanks and Regards.

 *Rukshan Chathuranga.*

 *Department Of Computer Science  Engineering,*

 *Faculty Of Engineering,*
 *University Of Moratuwa. **Sri Lanka.*

 *WEB: http://www.rukspot.com/ http://rukspot.com/*

Re: How to get TabletId

[Keith Turner]

What do you meant by table id?  What are you trying to accomplish?

On Thu, Jul 16, 2015 at 11:19 AM, Rukshan Chathuranga rcruksha...@gmail.com
 wrote:

 Hi,

 how can i get the id of each Tablets?

 Thanks and Regards.

 *Rukshan Chathuranga.*

 *Department Of Computer Science  Engineering,*

 *Faculty Of Engineering,*
 *University Of Moratuwa. **Sri Lanka.*

 *WEB: http://www.rukspot.com/ http://rukspot.com/*

Re: Accumulo starting up but NPE in logs and can't access monitor in browser

[Keith Turner]

On Mon, Jul 6, 2015 at 5:32 PM, suave ultraback...@gmail.com wrote:

 links http://localhost:50095

 on the EC2 instance works but

 links http://EC2InstanceIP:50095

 on my laptop does not.
 I strongly think this is because of some security setting of my EC2
 instance
 though I did allow incoming connections to my EC2 instance on port 50095
 coming from any IP actually,



Have you tried using -D options with ssh along with a browser proxy plugin?


 and in accumulo-env.sh I do have

 export ACCUMULO_MONITOR_BIND_ALL=true

 However, maybe it's related to an Exception in the log because that's what
 I
 want to find out too.
 The logs are not yet clean, there's a

 [net.SocketNode] INFO :  Caught java.io.EOFException closing conneciton.

 in monitor.internal.debug.log and a

 org.apache.thrift.transport.TTransportException: java.net.ConnectException:
 Connection refused

 in tracer.internal.debug.log.
 Maybe it's related to that but I would like to get rid of these exceptions
 anyway.

 Thanks.



 --
 View this message in context:
 http://apache-accumulo.1065345.n5.nabble.com/Accumulo-starting-up-but-NPE-in-logs-and-can-t-access-monitor-in-browser-tp14624p14647.html
 Sent from the Users mailing list archive at Nabble.com.

Re: How to generate UUID in real time environment for Accumulo

[Keith Turner]

Could look into using Lexicoders.  The following program prints out 19.
However this will vary depending on how many leading 0 bytes the longs
have, because those are dropped.

long time = System.currentTimeMillis();

ListLexicoderLong ll =  new ListLexicoderLong(new ULongLexicoder());

ListLong list = Arrays.asList(new Long[3]);
list.set(0, time);
list.set(1, 123456l);
list.set(2, 987654l);

byte[] b = ll.encode(list);

System.out.println(b.length);

On Wed, Jun 24, 2015 at 2:32 AM, mohit.kaushik mohit.kaus...@orkash.com
wrote:

 On 06/23/2015 06:44 PM, Keith Turner wrote:

 row=time_client id_client counter

 this will definitely generate a UUID but if I use 14 digits for time +
 12 digits for client_id + say 6 digits for client_counter which makes
 the UUID 32 bit long. I want UUID to be 16 digits long.

 Can you suggest some encoding technique which can encode it to 16 digits
 and also maintains the time order?

 -Mohit kaushik

Re: How to generate UUID in real time environment for Accumulo

[Keith Turner]

Would something like the following work?

row=time_client id_client counter

Where the client id is a unique id per client instance, it would be
allocated once using Zookeeper or an Accumulo Conditional writer when the
client starts.   The client counter would be an AtomicLong in the client.

On Tue, Jun 23, 2015 at 8:08 AM, mohit.kaushik mohit.kaus...@orkash.com
wrote:

  Hi All,

 I have an application which can index data at very high rate from multiple
 clients. I need to generate a unique id to store documents.
 It Should
 (1) use the current system time in millies.
 (2) it should be designed to sort lexicographically on the bases of time.
 (3) if I just store the currentTimeInMillies than i can just index 1000
 unique docs per sec. It should be able to generate millions of UUID's per
 sec.

 I am searching for the best possible approach to implement, any help?
 Regards

 * Mohit Kaushik*
 Software Engineer
 A Square,Plot No. 278, Udyog Vihar, Phase 2, Gurgaon 122016, India
 *Tel:* +91 (124) 4969352 | *Fax:* +91 (124) 4033553

  http://politicomapper.orkash.cominteractive social intelligence at
 work...

  https://www.facebook.com/Orkash2012
 http://www.linkedin.com/company/orkash-services-private-limited
 https://twitter.com/Orkash  http://www.orkash.com/blog/
 http://www.orkash.com
  http://www.orkash.com ... ensuring Assurance in complexity and
 uncertainty

 *This message including the attachments, if any, is a confidential
 business communication. If you are not the intended recipient it may be
 unlawful for you to read, copy, distribute, disclose or otherwise use the
 information in this e-mail. If you have received it in error or are not the
 intended recipient, please destroy it and notify the sender immediately.
 Thank you *

Re: RowDeletingIterator returns deleted records when cq is specified

[Keith Turner]

On Wed, Jun 17, 2015 at 12:28 PM, Christopher ctubb...@apache.org wrote:

 Can you provide sample data which corresponds to these scan commands
 to help reproduce it? Also, are you using any other iterators which
 might be hiding the delete markers from this iterator at scan time?


I think an Accumulo system iterator is hiding the delete markers.  I made a
comment on ACCUMULO-3905 about this.



 --
 Christopher L Tubbs II
 http://gravatar.com/ctubbsii


 On Tue, Jun 16, 2015 at 6:26 PM, Konstantin Pelykh kpel...@gmail.com
 wrote:
  It seem that there might be a bug in RowDeletingIterator:
 
  after using RowDeletingIterator I get expected results when querying by
  rowId and CF, e.g.
 
  scan -b myrowid  -c field/abc -t table  doesn't return deleted rows
 as
  expected
 
  however if I add column qualified to the query, I see deleted items.
 
  scan -b myrowid  -c field/abc:sample_qualifier -t table -- returns
 deleted
  rows
 
  After major compaction the problem goes away.
 
  Is it expected behavior?
 
  Konstantin

Re: How to get the first key of the Tablet servers

[Keith Turner]

You can use TableOperations.getMaxRow()[1] to find the last key in a
range.   This is not a constant time operation, its ~ O(log R) where R is
the amount of bits in your row.   There is a shell command to do this also.

[1]:
http://accumulo.apache.org/1.6/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#getMaxRow%28java.lang.String,%20org.apache.accumulo.core.security.Authorizations,%20org.apache.hadoop.io.Text,%20boolean,%20org.apache.hadoop.io.Text,%20boolean%29

On Tue, Jun 16, 2015 at 12:11 AM, Rukshan Chathuranga rcruksha...@gmail.com
 wrote:

 hi,

 Does anyone know how to get the first or last key of the tablet servers?

 Thanks and Regards.

 *Rukshan Chathuranga.*

 *Department Of Computer Science  Engineering,*

 *Faculty Of Engineering,*
 *University Of Moratuwa. **Sri Lanka.*

 *WEB: http://www.rukspot.com/ http://rukspot.com/*