RE: Issue with TableOperations

[dlmarion]

 

  My guess is that the error is happening on the server side and being raised 
at 
https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/Base64.java#L713.
 If there is a stack trace in the server log, then that would help pinpoint the 
issue. I’m guessing here, but it could be the following call:

(client) TableOperationsImpl.doFateOperation ->
(client) TableOperationsImpl.beginFateOperation ->

(manager) FateServiceHandler.beginFateOperation ->
(manager) FateServiceHandler.authenticate ->

(manager) SecurityOperation.authenticateUser ->
(manager) SecurityOperations.authenticate ->
(manager) Credentials.fromThrift -> …

 

  In your code, how is client created? Is it cached, shared, etc? Does the 
framework that you are using create a new instance of your class and maybe the 
AccumuloClient is not fully initialized?

 

From: Christopher  
Sent: Sunday, April 21, 2024 4:56 PM
To: accumulo-user 
Subject: Re: Issue with TableOperations

 

I don't see this error message anywhere in Accumulo code. It looks like it 
might be coming from d4m?

 

On Sat, Apr 20, 2024, 17:07 Geoffry Roberts mailto:geoffry.robe...@gmail.com> > wrote:

v2.1.2

 

On Sat, 20 Apr 2024 at 13:11, Dave Marion mailto:dlmar...@comcast.net> > wrote:

 

 

  What version of Accumulo are you using?

 

 

 

On Apr 20, 2024 12:16 PM, Geoffry Roberts mailto:geoffry.robe...@gmail.com> > wrote:

I am setting up a microservice that accesses accumulo.  It's a specialized 
thing of limited scope.  I do need to get it to create tables (Code is below.). 
 

 

*   Java 17
*   SpringBoot running Jetty
*   Postman

 

In the below code bit, the tableName is a string "patient״.  When I run it, I 
get an error message: 

 

org.apache.accumulo.core.client.AccumuloException: Input byte[] should at least 
have 2 bytes for base64 bytes

at 
org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:413)

at 
org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:365)

at 
org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1803)

at 
org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:254)

at 
org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:226)

at 
d4m.acc.microservice.D4MACCController.createTable(D4MACCController.java:58)

 

Q: Why it seems all is quite correct?  The log message is correct.  The return 
value is correct.




@PostMapping("/create")

public String createTable(@RequestBody String tableName) {

TableOperations ops = client.tableOperations();

try {

LOG.info("create=" + tableName);

ops.create(tableName);

} catch (AccumuloException | AccumuloSecurityException | TableExistsException 
e) {

e.printStackTrace();

}

return tableName;

}--

 

Thanks

 

My software always runs perfectly in the end. If it is not perfect, it is not 
yet the end.

 

Geoffry Roberts

 




 

-- 

My software always runs perfectly in the end. If it is not perfect, it is not 
yet the end.

 

Geoffry Roberts

RE: Scan thread resource management [SEC=UNOFFICIAL]

[dlmarion]

Bruce,

 

  There are no future planned releases for the 1.x line. In 2.1 Scan
Executors[1] offer a way for the user to configure the resource pools used
for scanning a table. You could use a prioritizer to influence the execution
order of scans in the default scan executor, or setup different scan
executors for different tables, or use both in some manner.

 

[1] https://accumulo.apache.org/docs/2.x/administration/scan-executors

 

Dave

 

 

From: McClure, Bruce MR 2 via user  
Sent: Monday, January 22, 2024 11:44 PM
To: user@accumulo.apache.org
Subject: Scan thread resource management [SEC=UNOFFICIAL]

 

UNOFFICIAL

Hi,

 

I have seen that there was a fix in 1.10.3 for #3226   which also fixed an
issue by again using the "fair" semaphore for scan threads.

 

However, is there any other current or future planned ways to improve or
manage contention between scan threads such as the tendency for a scans on a
larger table to monopolise the threads preventing scans on smaller tables
from making progress?

 

Thanks,

 

Bruce.

 

RE: Impact from Enabling TLS

[dlmarion]

Does this apply? https://bugs.openjdk.org/browse/JDK-8190917

 

From: Sanjay Deshmukh  
Sent: Monday, June 5, 2023 3:28 PM
To: user@accumulo.apache.org
Subject: Re: Impact from Enabling TLS

 

Sure thing. We're running Accumulo 1.10.1. Our configuration has 15 tablet 
servers and a client pool of 4 threads that could be doing scans or batch scans 
at any given time. I wish I could copy over the exact jstack but unfortunately 
it's coming from an offline network, but I do see the same tablet servers in 
the thread names repeatedly (e.g. "batch scanner X- 4 looking up 1 ranges at 
tabletservername:9997", implying it it's not an infrequent use problem. In some 
cases I even see multiple threads trying to handshake with the same tablet 
server. The jstack here seems to go from 
TabletServerbatchReaderIterator.doLookup down through TFramedTransport.flush 
and into SSLSocketImpl.performInitialHandshake.

 

The lifecycle here is that each of these client threads is using a single 
shared instance of Connector, and instantiating a new Scanner or BatchScanner 
per use, and closing it when that use is done. There are many of these uses 
over time.

 

On Mon, Jun 5, 2023 at 3:14 PM Christopher mailto:ctubb...@apache.org> > wrote:

In Accumulo code, we do try to reuse thrift connections, but I'm not sure if 
the thrift code itself is reusing TLS sessions or not.

 

Keep in mind that Accumulo is a distributed system. Many of these handshakes 
might be going to different servers in a cluster. Are you able to tell if that 
is the case? It might be possible to increase session timeouts or the number of 
cached sessions, but the best case is that you'll still need to do a handshake 
with each server your client talks to.

 

It might also help if you told us the version of Accumulo you are using, and 
how you are using the client code (like, are you reusing client objects). If 
you can share a bit of the jstack with us, that might also be helpful if we 
need to troubleshoot further.

On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh mailto:sanj...@gmail.com> > wrote:

We've recently been required to enable TLS between our Accumulo clients and the 
tablet servers. We've got it working, but we're experiencing a significant 
performance impact. I'm running jstack on our client processes and consistently 
seeing a number of Accumulo client threads in 
sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me that 
the Accumulo client is not reusing TLS sessions, and instead establishing a new 
session for each connection. Am I reading this correctly? Is there a way to 
configure the Accumulo client to reuse TLS sessions?




 

-- 

Sanjay Deshmukh
sanj...@gmail.com  

RE: Problem creating tables in Accumulo

[dlmarion]

It looks like the issue is that `maxTime` is not being set at [0], so `time`
is not being set at [1]. If I had to guess, Accumulo is having an issue
talking with ZooKeeper. Were there errors in the log when initializing
Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods
and confirm connectivity with Accumulo?

[0]
https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or
g/apache/accumulo/tserver/tablet/TabletData.java#L159
[1]
https://github.com/apache/accumulo/blob/1.10/server/tserver/src/main/java/or
g/apache/accumulo/tserver/tablet/TabletData.java#L166

 

From: Samudrala, Ranganath [USA] via user  
Sent: Tuesday, February 21, 2023 1:43 PM
To: Samudrala, Ranganath [USA] via user 
Subject: Problem creating tables in Accumulo

 

Hello,

 

Accumulo is running in Kubernetes cluster. Metdata data is stored in HDFS
and table data is stored in Amazon S3. No attempt has been made yet to
store/insert table data. But, when I attempt to create a table in Accumulo
via Java API, I see the following error.

 

 

WARN : org.apache.accumulo.core.clientImpl.ThriftTransportPool - Thread
"main" stuck on IO to accumulo-masters-0.accumulo-masters.acc
umulo-k8s.svc.cluster.local: (0) for at least 120033 ms

 

I see the below error in T-SERVER logs:

2023-02-21 18:04:09,373 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: exception trying to assign
tablet +r<<
hdfs://accumulo-hdfs-namenode-0.accumulo-hdfs-namenodes:8020/accumulo/tables
/+r/root_tablet
java.lang.NullPointerException
at
org.apache.accumulo.server.tablets.TabletTime.getInstance(TabletTime.java:60
)
at org.apache.accumulo.tserver.tablet.Tablet.(Tablet.java:341)
at
org.apache.accumulo.tserver.TabletServer$AssignmentHandler.run(TabletServer.
java:2504)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:750)
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: failed to open tablet +r<<
reporting failure to master
2023-02-21 18:04:09,374 WARN [Root tablet assignment retry]
[org.apache.accumulo.tserver.TabletServer]: rescheduling tablet load in
600.00 seconds
2023-02-21 18:07:03,210 WARN [SimpleTimer-0]
[org.apache.accumulo.server.ServerUtil]: System swappiness setting is
greater than ten (30) which can cause time-sensitive operations to be
delayed. Accumulo is time sensitive because it needs to maintain distributed
lock agreement.

 

 

The installed configuration is made up of 2 instances of HDFS data nodes, 2
instances of T-SERVERs, 1 instance each of HDFS name node and Accumulo
master.

 

> config -t accumulo.root

.

default| table.file.replication  | 0 

site   |@override .. | 2

table  |@override .. | 5

.

.

> config -t accumulo.metadata

.

default| table.file.replication  | 0 

site   |@override .. | 2

table  |@override .. | 5

.

 

How do I fix this problem?


Thanks in advance.

Ranga

 

RE: Init accumulo with custom properties file

[dlmarion]

Take a look at https://github.com/dlmarion/accumulo-k8s. You can put the
Accumulo config in a ConfigMap (accumulo-config.yaml), then mount it at the
correct location in another container (see the server process yaml files). I
used a job to do the init phase, see accumulo-init.yaml.

 

 

From: Samudrala, Ranganath [USA] via user  
Sent: Thursday, February 2, 2023 4:11 PM
To: Samudrala, Ranganath [USA] via user 
Subject: Init accumulo with custom properties file

 

Hello

 

I see there is an option to start an Accumulo component with a different
properties file using -p, -props or --props option. Any reason same feature
does not exist while performing init?

 

The problem I am trying to solve is as below: 

I am trying to configure Accumulo with S3 using instructions here:
https://accumulo.apache.org/blog/2019/09/10/accumulo-S3-notes.html . This
instructions calls for following steps:

1.  Run "init" or "init --upload-accumulo-props" 
2.  Run "init --add-volumes" option
3.  Start all Accumulo components. 

I am trying to configure the setup in K8S and would like to merge all these
steps into one deployment invocation task. In K8S, we can attempt to do the
first two steps using "initContainers" but the "accumulo.properties" need to
be different for each step. So, let us say, I create 2 properties files,
accumulo-init.properties and accumulo.properties. In the first step we
should be able to specify accumulo-init.properties and for remaining steps,
we should be able to use accumulo.properties. 

 

Problem is there is no option to specify a property file during the "init"
process.

 

thanks

Ranga

 

RE: Accumulo On S3

[dlmarion]

I believe that S3 Guard is OBE, but you still need to put the WAL on HDFS as S3 
does not support sync. If you put your WAL in S3, and you have a tserver 
failure, then it’s possible that you will lose data.

 

From: Christopher  
Sent: Wednesday, October 12, 2022 4:12 PM
To: user@accumulo.apache.org
Subject: Re: Accumulo On S3

 

Since S3 became strongly consistent, I think it would probably just work. But, 
obviously, we can't make any guarantees, especially about the behavior of 
software outside of our control. So, your experience may vary.

 

On Wed, Oct 12, 2022 at 12:28 PM Josh Clum mailto:joshc...@gmail.com> > wrote:

Hi,

 

Question on this post: 
https://accumulo.apache.org/blog/2019/09/10/accumulo-S3-notes.html 

 

It's been a long time and it looks like several of the merge requests are 
merged into master. 

 

The point about S3 Guard being needed seems OBE since S3 is strongly consistent 
and S3 Guard is deprecated at this point: 
https://issues.apache.org/jira/browse/HADOOP-17480

 

Should Accumulo on S3 with the metadata table and WAL logs in S3 work or are 
there still commits somewhere that won't be available until 2.1.0?

 

Thanks,

Josh

RE: Tablet Server Session Id Out of Range

[dlmarion]

Please see: https://lists.apache.org/thread/p7mwtkfpbyb551pw5k7yg61jopf50m8s

 

From: Josef Roehrl - PHEMI  
Sent: Monday, June 20, 2022 7:02 PM
To: user@accumulo.apache.org
Subject: Re: Tablet Server Session Id Out of Range

 

Hi Jonathan,

 

We too have exactly this issue as of a couple of days ago.

 

This is in 1.7.2, zookeeper 3.4.5 and from a cluster that has been shut down 
for a long time.

 

Accumulo writes the root_tablet/lastlocation node with the name of the tserver, 
concatenated with the ephemeral owner form zookeeper. For us, this hex value is 
now a negative 64-bit long which parseLong throws an exception on. Note that 
parseUnsignedLong would have work. Also, note that, at least, in Accumulo 2.0, 
the code to do the same was changed to parse strings, not longs, avoiding the 
whole issue.

 

Is there not some way/hack to reset the session id to something reasonable 
somewhere in zookeeper?

 

Regards,

 

On Fri, Apr 22, 2022 at 12:30 PM Wonders, Jonathan (Serco NA) 
mailto:jonathan.wond...@serco-na.com> > wrote:

Serco Business

 

Greetings,

 

The team I work with is encountering an issue when starting an Accumulo 1.7.x 
cluster and when running troubleshooting commands such as bin/accumulo admin 
checkTablets. The primary symptom is a NumberFormatException thrown within 
ZookeeperLockChecker that occurs when parsing the tablet server session id 
(Long.parseLong) for an input string “ff804d767efe0004” (which is out of range 
when interpreting as a positive signed long).

 

>From what I can gather, our zookeeper cluster has been running for such a long 
>time that the epoch component of the session id has grown to the point where 
>interpreting the session id as a signed long would be a negative value. Within 
>the ZooKeeper code, the session id is treated as an unsigned long (e.g., 
>Long.toHexString) which leads me to think that the Accumulo code is not 
>parsing the value correctly. This discrepancy is present in all versions since 
>the introduction of the ZookeeperLockChecker class.

 

There does not appear to be an easy way to work around this problem. Currently, 
our best idea of how to recover the data from this cluster is to set up a 
separate zookeeper cluster, migrate the data we have in zookeeper to the new 
cluster, and then swap over configuration to point to the new zookeeper 
cluster. I would appreciate any ideas or suggestions from the community.

 

Thanks,

Jonathan

 

 

 

 




 

-- 

Josef Roehrl

Professional Services
Solutions Architect
I recognize that my working hours may not be the same as yours.
Please feel free to respond only during your working hours.
Has it really been 45+ years programming?
So little time. So much to learn and do.

PHEMI Systems

777 Hornby Street, Suite 600 
Vancouver, BC
V6Z 1S4
604-336-1119

Website   Twitter   
Linkedin 

  

RE: example thrift client to get master metrics

[dlmarion]

The Monitor[1] gets a MasterMonitorInfo[2] object from the Master that contains 
the information you need. The tableMap member contains a TableInfo[3] object 
for each table that holds the information you are looking for.

 

[1] 
https://github.com/apache/accumulo/blob/1.9/server/monitor/src/main/java/org/apache/accumulo/monitor/Monitor.java#L297

[2] 
https://github.com/apache/accumulo/blob/1.9/core/src/main/java/org/apache/accumulo/core/master/thrift/MasterMonitorInfo.java

[3] 
https://github.com/apache/accumulo/blob/1.9/core/src/main/java/org/apache/accumulo/core/master/thrift/TableInfo.java

 

From: Bulldog20630405  
Sent: Wednesday, July 22, 2020 1:11 PM
To: accumulo-user 
Cc: Bulldog20630405 
Subject: example thrift client to get master metrics

 

 

we are trying to grab master metrics (e.g. table x current major compaction ... 
something you can see on the monitor page) programmatically.  does anyone have 
an example client to do that?

 

RE: Uniquely identifying metrics with multiple tservers per node.

[dlmarion]

IIRC the 'service.name' property in the metrics properties file is used to
note from which service the metric was emitted. I would suggest testing this
by setting the service.name property to tserver1 and see if your metrics
show up with that name. If they do, then you might be able to modify the
start scripts to have the different tserver instances use different metrics
properties files.

 

From: Arvind Shyamsundar  
Sent: Monday, August 26, 2019 8:39 PM
To: user@accumulo.apache.org
Subject: Uniquely identifying metrics with multiple tservers per node.

 

Hello!

We are testing Apache Accumulo 1.9.3 on Microsoft Azure. For centrally
gathering metrics we are sending Accumulo and Hadoop metrics to collectd's
statsd plugin (which acts as a statsd sink), which in turn forwards those
metrics to an Azure monitoring service. The
hadoop-metrics2-accumulo.properties looks like this:

 

*.sink.statsd.class=org.apache.hadoop.metrics2.sink.StatsDSink

accumulo.sink.statsd.server.host=127.0.0.1

accumulo.sink.statsd.server.port=8125

accumulo.sink.statsd.skip.hostname=false # we have tried setting
this to true and also to false, but that still does not include the service
instance.

accumulo.sink.statsd.service.name=master 

 

Everything works great when we have the default of 1 tserver per node.
However, when running 2 tservers per node (Accumulo 1.9.3), we find that the
metrics being reported (TabletServer.tserver.general.ingestRate for example)
are not qualified / distinguished with the service instance ID (1_ or 2_) in
the name. It appears as though only the values for one instance are being
sent. As a result, we are unable to accurately measure aggregate metrics (as
an example, the value of ingestRate recorded through this route is ~ 50% of
what is shown on the Monitor UI).

 

If someone has an example of how to configure
hadoop-metrics2-accumulo.properties and multiple tservers per node in a way
that the counter names would be distinguished by their service instance ID,
that would be really helpful.

 

Thank you!

 

- Arvind.

RE: Accumulo performance on various hardware configurations

[dlmarion]

This may suggest an issue with client, either getting the data to the client or 
the client itself (although I think there are other performance related changes 
you could make). I’m curious what the end goal is here. Is this a real world 
use case? If you are using this type of benchmark to evaluate the speed of 
Accumulo, then you will likely not get the same performance when you apply your 
data and your real use cases.

 

From: guy sharon  
Sent: Wednesday, August 29, 2018 3:13 PM
To: user@accumulo.apache.org
Subject: Re: Accumulo performance on various hardware configurations

 

hi Mike,

 

As per Mike Miller's suggestion I started using 
org.apache.accumulo.examples.simple.helloworld.ReadData from Accumulo with 
debugging turned off and a BatchScanner with 10 threads. I redid all the 
measurements and although this was 20% faster than using the shell there was no 
difference once I started playing with the hardware configurations.

 

Guy.

 

On Wed, Aug 29, 2018 at 10:06 PM Michael Wall mailto:mjw...@gmail.com> > wrote:

Guy,

 

Can you go into specifics about how you are measuring this?  Are you still 
using "bin/accumulo shell -u root -p secret -e "scan -t hellotable -np" | wc 
-l" as you mentioned earlier in the thread?  As Mike Miller suggested, 
serializing that back to the display and then counting 6M entries is going to 
take some time.  Try using a Batch Scanner directly.

 

Mike

 

On Wed, Aug 29, 2018 at 2:56 PM guy sharon mailto:guy.sharon.1...@gmail.com> > wrote:

Yes, I tried the high performance configuration which translates to 4G heap 
size, but that didn't affect performance. Neither did setting 
table.scan.max.memory to 4096k (default is 512k). Even if I accept that the 
read performance here is reasonable I don't understand why none of the hardware 
configuration changes (except going to 48 cores, which made things worse) made 
any difference.

 

On Wed, Aug 29, 2018 at 8:33 PM Mike Walch mailto:mwa...@apache.org> > wrote:

Muchos does not automatically change its Accumulo configuration to take 
advantage of better hardware. However, it does have a performance profile 
setting in its configuration (see link below) where you can select a profile 
(or create your own) based on your the hardware you are using.

 

https://github.com/apache/fluo-muchos/blob/master/conf/muchos.props.example#L94

On Wed, Aug 29, 2018 at 11:35 AM Josh Elser mailto:els...@apache.org> > wrote:

Does Muchos actually change the Accumulo configuration when you are 
changing the underlying hardware?

On 8/29/18 8:04 AM, guy sharon wrote:
> hi,
> 
> Continuing my performance benchmarks, I'm still trying to figure out if 
> the results I'm getting are reasonable and why throwing more hardware at 
> the problem doesn't help. What I'm doing is a full table scan on a table 
> with 6M entries. This is Accumulo 1.7.4 with Zookeeper 3.4.12 and Hadoop 
> 2.8.4. The table is populated by 
> org.apache.accumulo.examples.simple.helloworld.InsertWithBatchWriter 
> modified to write 6M entries instead of 50k. Reads are performed by 
> "bin/accumulo org.apache.accumulo.examples.simple.helloworld.ReadData -i 
> muchos -z localhost:2181 -u root -t hellotable -p secret". Here are the 
> results I got:
> 
> 1. 5 tserver cluster as configured by Muchos 
> (https://github.com/apache/fluo-muchos), running on m5d.large AWS 
> machines (2vCPU, 8GB RAM) running CentOS 7. Master is on a separate 
> server. Scan took 12 seconds.
> 2. As above except with m5d.xlarge (4vCPU, 16GB RAM). Same results.
> 3. Splitting the table to 4 tablets causes the runtime to increase to 16 
> seconds.
> 4. 7 tserver cluster running m5d.xlarge servers. 12 seconds.
> 5. Single node cluster on m5d.12xlarge (48 cores, 192GB RAM), running 
> Amazon Linux. Configuration as provided by Uno 
> (https://github.com/apache/fluo-uno). Total time was 26 seconds.
> 
> Offhand I would say this is very slow. I'm guessing I'm making some sort 
> of newbie (possibly configuration) mistake but I can't figure out what 
> it is. Can anyone point me to something that might help me find out what 
> it is?
> 
> thanks,
> Guy.
> 
> 

RE: Problems with accumulo replication

[dlmarion]

You can also use the tserver.walog.max.age property to ensure that the walogs 
roll if there is no activity. The default is 24h and was backported to 1.7.2. 
See ACCUMULO-4004 for more info.

-Original Message-
From: Josh Elser [mailto:els...@apache.org] 
Sent: Friday, December 29, 2017 10:08 AM
To: user@accumulo.apache.org
Subject: Re: Problems with accumulo replication

If the system is reporting files that need to be replicated, it's probably one 
of two problems:

* The WALs are still in use by the TabletServers. In its current 
implementation, the WALs are not replicated until the TabletServers don't 
referenced those WALs. This happens either by writing enough data or when the 
tabletserver is restarted. You can try to investigate either for this.
* The replication is trying to happen but fails. You can look at the 
TabletServer logs on the primary instance to see if there are any reported 
exceptions around sending the data to the peer.

On 12/29/17 8:24 AM, vLex Systems wrote:
> Hi,
> 
> I've configured replication between two instances of accumulo: one is 
> the primary accumulo and the other is a peer created from a restore of 
> the backup of the primary.
> 
> I've followed the instructions in the manual
> (https://accumulo.apache.org/1.7/accumulo_user_manual#_replication)
> and I can see the 4 tables I've configured to replicate in the 
> Accumulo Monitor but they do not replicate. They have 1 or 2 "Files 
> needing replication" and this number never decreases.
> 
> I've also tried inserting data in one of the tables and the data does 
> not replicate to the accumulo peer instance.
> 
> In the master log I see many entries like this one:
> 2017-12-29 13:22:25,490 [replication.RemoveCompleteReplicationRecords]
> INFO : Removed 0 complete replication entries from the table 
> accumulo.replication
> 
> Does anyone know what could be happening?
> 
> Thanks.
> 

RE: accumulo.metadata table online but scans hang

[dlmarion]

Re #2: Does your Accumulo processes run as the hdfs user on the O/S, or as the 
accumulo user? Make sure you are checking the correct users trash folder. Also, 
check the Accumulo garbage collector log to see if the GC process deleted the 
WAL files. Take a look at [1] to see if you are hitting this case.

You can create empty rfiles and copy them into place. I believe the procedure 
to do this is to create an empty table and run a compaction on the table. Then 
you should be able to copy the resulting file into the desired locations (devs 
- please correct me here if this is not correct).

Finally, I would not do anything destructive yet. Let's see if we can get some 
other devs to chime in with some ideas.

[1] https://issues.apache.org/jira/browse/ACCUMULO-4157


-Original Message-
From: Nick Wise [mailto:nicholas.w...@sa.catapult.org.uk] 
Sent: Wednesday, August 30, 2017 5:36 PM
To: user@accumulo.apache.org
Subject: RE: accumulo.metadata table online but scans hang


Thank you very much for the pointers Dave.  Looking at those:

1. That seems reasonable, I’m not sure how to check after the fact but makes 
sense.
2. Ah.  Looks like we don’t have trash enabled, there’s no /user/hdfs/.Trash 
folder that I can see.  I’m getting a sinking feeling… 3. I had to allocate 4G 
but that worked and now I have a folder listing of 758k files.  I’ve cross 
referenced with the 1101 WAL files referred to in our logs and not a single one 
exists. Sinking some more.

So, it sounds like (speaking from a position of ignorance) that we have a 
system where accumulo.metadata has outstanding WAL files to recover, but the 
files don't exist, but the only way to restore the system is to convince the 
metadata table that it doesn't need the WAL files, but to edit the metadata 
table we have to resolve the outstanding WAL files, etc.

What would happen if we created an empty file in place of the missing WAL 
files?  Would they be considered to be an invalid format and break things more 
(I'm not sure that's possible), or might they be accepted as needing no further 
resolution?

Any other thoughts (anyone) on how we might save ourselves, besides starting 
from scratch?  (When we first loaded our 16TB of data it took 6 weeks using the 
map/reduce method!)

Thank you again!

Nick



From: Dave Marion [mailto:dlmar...@comcast.net]
Sent: 30 August 2017 20:13
To: user@accumulo.apache.org; Nick Wise 
Subject: Re: accumulo.metadata table online but scans hang

Some immediate thoughts:

1. Regarding node08 having so many files, maybe it was the last DN that had 
free space?
2. Look in the trash folder for the missing referenced WAL files 3. For you 
OOME using the HDFS CLI, I think you can increase the amount of memory that the 
client will use with: export HADOOP_CLIENT_OPTS="-Xmx1G" (or something like 
that).

Still digesting the rest


On August 30, 2017 at 2:45 PM Nick Wise 
 wrote:

Disclaimer: I don’t have much experience with Accumulo or Hadoop, I’m standing 
in because our resident expert is away on honeymoon!  We’ve done a great deal 
of reading and do not know if our situation is recoverable, so any and all 
advice would be very welcome.

Background:
We are running:
(a) Accumulo version: 1.7.0
(b) Hadoop version: 2.7.1
(c) Geomesa version: 1.2.1
We have 31 nodes, 2 masters and 3 zookeepers (obviously named in the log 
excerpts below).  Nodes are both data nodes and tablet servers, masters are 
also name nodes.  Nodes have 16GB RAM, Intel Core i5 dual core CPUs, and 500GB 
or 1TB SSD each.
This is a production deployment where we are analysing 16TB (and growing) 
geospatial data, with the outcomes being used daily.  We have customers relying 
on our results.

Initial Issue:
The non-DFS storage used in our HDFS system was falsely reporting that it was 
using all of the free space we had available, resulting in HDFS rejecting 
writes from a variety of places across our cluster.  After research it appeared 
that this may be as a result of a bug, and that restarting HDFS services would 
resolve it.  After restarting the HDFS services the non-DFS storage used 
immediately returned to expected levels, but accumulo didn’t seem to be 
responding to queries so we ran stop-all.sh and start-all.sh.  When running 
stop-all.sh it timed out trying to contact the master, and did a forced 
shutdown.

After restarting, Accumulo listed all the tables as being online (except for 
accumulo.replication which is offline) but none of the tables have their 
tablets associated except for:
(a) accumulo.metadata
(b) accumulo.root
All Geomesa tables are showing as online though the tablets, table sizes and 
record counts are not showing in the web UI.

In the logs (which are very large) there are a range of issues showing, the 
following seeming important from our Googling.

Log excerpts:
2017-08-30 14:45:06,195 [master.EventCoordinator] INFO : Marked 1 tablets as 

RE: maximize usage of cluster resources during ingestion

[dlmarion]

Regarding the referenced paper, pre-splitting the tables, using an optimized 
zookeeper deployment, and increasing concurrent minor / major compactions are 
good things. I'm not sure that we want to recommend turning off the write ahead 
logs and replication for production deployments.

-Original Message-
From: Jeremy Kepner [mailto:kep...@ll.mit.edu] 
Sent: Thursday, July 13, 2017 10:05 AM
To: user@accumulo.apache.org
Subject: Re: maximize usage of cluster resources during ingestion

https://arxiv.org/abs/1406.4923  contains a number of tricks for maximizing 
ingest performance.

On Thu, Jul 13, 2017 at 08:13:40AM -0400, Jonathan Wonders wrote:
> Keep in mind that Accumulo puts a much different kind of load on HDFS 
> than the DFSIO benchmark.  It might be more appropriate to use a tool 
> like dstat to monitor HDD utilization and queue depth.  HDD throughput 
> benchmarks usually will involve high queue depths as disks are much 
> more effective when they can pipeline and batch updates. Accumulo's 
> WAL workload will typically call hflush or hsync periodically which 
> interrupts the IO pipeline much like memory barriers can interrupt CPU 
> pipelining except more severe.  This is necessary to provide 
> durability guarantees, but definitely comes at a cost to throughput.  
> Any database that has these durability guarantees will suffer 
> similarly to an extent.  For Accumulo, it is probably worse than for 
> non-distributed databases because the flush or sync must happen at 
> each replica prior to the mutation being added into the in-memory map.
> 
> I think one of the reasons the recommendation was made to add more 
> tablet servers is because each tablet server only writes to one WAL at 
> a time and each block will live on N disk based on replication factor.  
> If you have a replication factor of 3, there will be 10x3 blocks being 
> appended to at any given time (excluding compactions).  Since you have 
> 120 disks, not all will be participating in write-ahead-logging, so 
> you should not count the IO capacity of these extra disks towards 
> expected ingest throughput.  10 tablet servers per node is probably 
> too many because there would likely be a lot of contention 
> flushing/syncing WALs.  I'm not sure how smart HDFS is about how it 
> distributes the WAL load.  You might see more benefit with 2-4 
> tservers per node.  This would mostly likely require more batch writer 
> threads in the client as well.
> 
> I'm not too surprised that snappy did not help because the WALs are 
> not compressed and are likely a bigger bottleneck than compaction 
> since you have many disks not participating in WAL.
> 
> 
> On Wed, Jul 12, 2017 at 11:16 AM, Josh Elser  wrote:
> 
> > You probably want to split the table further than just 4 tablets per 
> > tablet server. Try 10's of tablets per server.
> >
> > Also, merging the content from (who I assume is) your coworker on 
> > this stackoverflow post[1], I don't believe the suggestion[2] to 
> > verify WAL max size, minc threshold, and native maps size was brought up 
> > yet.
> >
> > Also, did you look at the JVM GC logs for the TabletServers like was 
> > previously suggested to you?
> >
> > [1] https://stackoverflow.com/questions/44928354/accumulo-tablet
> > -server-doesnt-utilize-all-available-resources-on-host-machine/
> > [2] https://accumulo.apache.org/1.8/accumulo_user_manual.html#_n
> > ative_maps_configuration
> >
> > On 7/12/17 10:12 AM, Massimilian Mattetti wrote:
> >
> >> Hi all,
> >>
> >> I ran a few experiments in the last days trying to identify what is 
> >> the bottleneck for the ingestion process.
> >> - Running 10 tservers per node instead of only one gave me a very 
> >> neglectable performance improvement of about 15%.
> >> - Running the ingestor processes from the two masters give the same 
> >> performance as running one ingestor process in each tablet server 
> >> (10
> >> ingestors)
> >> - neither the network limit (10 Gb network) nor the disk throughput 
> >> limit has been reached (1GB/s per node reached while running the 
> >> TestDFSIO benchmark on HDFS)
> >> - CPU is always around 20% on each tserver
> >> - changing compression from GZ to snappy did not provide any 
> >> benefit
> >> - increasing the tserver.total.mutation.queue.maxto 200MB actually 
> >> decreased the performance I am going to run some ingestion 
> >> experiment with Kudu over the next few days, but any other 
> >> suggestion on how improve the performance on Accumulo is very 
> >> welcome.
> >> Thanks.
> >>
> >> Best Regards,
> >> Massimiliano
> >>
> >>
> >>
> >> From: Jonathan Wonders 
> >> To: user@accumulo.apache.org, Dave Marion 
> >> Date: 07/07/2017 04:02
> >> Subject: Re: maximize usage of cluster resources during ingestion
> >> ---
> >> -
> >>
> >>
> >>
> >> I've personally never seen full CPU utilization during 

RE: Detecting database changes

[dlmarion]

What kind of changes are you looking to alert off? Is it data changes or
configuration changes?

> -Original Message-
> From: vaibhav thapliyal [mailto:vaibhav.thapliyal...@gmail.com]
> Sent: Tuesday, November 22, 2016 7:20 AM
> To: user@accumulo.apache.org
> Subject: Detecting database changes
> 
> Hi,
> 
> I have a use case where I need to send out notifications based on changes
in
> a table. Are there any kind of listeners which can be used to listen to a
> change in table event in accumulo?
> 
> How do I go about this use case?
> 
> Thanks
> Vaibhav

Re: List of Metrics2 Metrics

[dlmarion]

It used[1] to be in the documentation when it was hosted on the monitor. I did 
not see it looking at the current documentation. Looks like it was lost (and 
[1] is likely now out of date). 

[1] https://github.com/apache/accumulo/blob/1.4.0/docs/metrics.html 

- Original Message -

From: "Noe Detore"  
To: user@accumulo.apache.org 
Sent: Thursday, November 10, 2016 11:06:21 AM 
Subject: List of Metrics2 Metrics 

hello, 

Is there a documented list of produced metrics of metrics2 for accumulo? Any 
documentation explaining what the metrics are? 

RE: New Accumulo Blog Post

[dlmarion]

Regarding #2 – I think there are two options here:

 

1. Modify Accumulo to take advantage of HDFS Heterogeneous Storage

2. Modify Accumulo WAL code to support volumes

 

From: Jeff Kubina [mailto:jeff.kub...@gmail.com] 
Sent: Wednesday, November 02, 2016 9:02 PM
To: user@accumulo.apache.org
Subject: Re: New Accumulo Blog Post

 

Thanks for the blog post, very interesting read. Some questions ...

 

1. Are the operations "Writes mutation to tablet servers’ WAL/Sync or flush 
tablet servers’ WAL" and "Adds mutations to sorted in memory map of each 
tablet." performed by threads in parallel?

 

2. Could the latency of hsync-ing the WALs be overcome by modifying Accumulo to 
write them to a separate SSD-only HDFS? To maintain data locality it would 
require two datanode processes (one for the HDDs and one for the SSD), running 
on the same node, which is not hard to do.

 

Re: Bulk ingestion of different locality groups at different times

[dlmarion]

>>> Is Accumulo able to import these files, considering that they are two 
>>> different locality groups 

Yes. 

>>> without triggering a huge major compaction? 

Depends on your table.compaction.major.ratio and table.file.max settings. 


Sorry, not a real answer, but I think the answer is "it depends" 

- Original Message -

From: "Mario Pastorelli"  
To: user@accumulo.apache.org 
Sent: Friday, October 28, 2016 9:37:13 AM 
Subject: Bulk ingestion of different locality groups at different times 

Hi, 

I have a question about using bulk ingestion for a rather special case. Let's 
say that I have the locality groups A and B. The values of each locality group 
are written to Accumulo in at different times, which means that first we ingest 
all the cells of the group A and then of B. We use Spark to ingest those 
records. Right now we write all the values with a custom writer but we would 
like to create the rfiles directly with Spark. In the case above, we would have 
two jobs creating the rfiles for the two distinct locality groups. Is Accumulo 
able to import these files, considering that they are two different locality 
groups, without triggering a huge major compaction? If not, what strategy would 
you suggest for the above use case? 

Thanks, 
Mario 

-- 
Mario Pastorelli | TERA LYTICS 


software engineer 

Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland 
phone: +41794381682 
email: mario.pastore...@teralytics.ch 
www.teralytics.net 


Company registration number: CH-020.3.037.709-7 | Trade register Canton Zurich 
Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann de 
Vries 

This e-mail message contains confidential information which is for the sole 
attention and use of the intended recipient. Please notify us at once if you 
think that it may not be intended for you and delete it immediately. 

RE: 1.7 native library performance boost

[dlmarion]

I don't remember seeing anything about native library performance improvement, 
but I do know that there has been talk of a performance regression in Accumulo 
1.7.0 using the default configuration. I don't remember the specific details 
off-hand, but it had something to do with the default value of a property 
changing.

> -Original Message-
> From: Brown, Scott M (IS) [mailto:scott.bro...@ngc.com]
> Sent: Wednesday, October 19, 2016 6:49 PM
> To: user@accumulo.apache.org
> Subject: 1.7 native library performance boost
> 
> So with 1.7, we noticed the statement that utilizing the native library can 
> add
> up to 40% performance increase.  With two identical systems, one with 1.6
> one with 1.7, we're not seeing any notable difference at all with either
> queries or large ingests.
> 
> Any idea what operations can take advantage, or what circumstances the
> performance boost comes into play?

Re: [ANNOUNCE] Apache Fluo 1.0.0-incubating released

[dlmarion]

Congrats! 

- Original Message -

From: "Mike Walch"  
To: user@accumulo.apache.org 
Sent: Friday, October 14, 2016 3:21:00 PM 
Subject: [ANNOUNCE] Apache Fluo 1.0.0-incubating released 

The Apache Fluo (incubating) team is happy to announce the release of Fluo 
1.0.0-incubating: 

https://fluo.apache.org/release/fluo-1.0.0-incubating/ 

Apache Fluo adds distributed transactions to Apache Accumulo, and provides an 
observer/notification framework to allow users to incrementally update large 
data sets stored in Accumulo. This is the first release of Fluo as an Apache 
project. Starting with this release, Apache Fluo will follow semantic 
versioning ( http://semver.org ) for all future API changes. 

Thanks, 
The Apache Fluo team 

= 

*Disclaimer* 

Apache Fluo is an effort undergoing incubation at The Apache Software 
Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required 
of all newly accepted projects until a further review indicates that the 
infrastructure, communications, and decision making process have stabilized in 
a manner consistent with other successful ASF projects. While incubation status 
is not necessarily a reflection of the completeness or stability of the code, 
it does indicate that the project has yet to be fully endorsed by the ASF. 

RE: how do I list user permissions per table

[dlmarion]

You can also use the admin dumpConfig command to dump all of the configuration 
information, then look at the user permission files.

 

From: Christopher [mailto:ctubb...@apache.org] 
Sent: Friday, September 23, 2016 5:39 PM
To: user@accumulo.apache.org
Subject: Re: how do I list user permissions per table

 

Currently, there's no single command to run to list permissions for a 
particular table.
However, you can iterate through the users, and get the list of table 
permissions per user.

I did something like:

TABLENAME=trace
for x in $(bin/accumulo shell -u root -p "$PASS" -e users | grep -v "^$(date 
+%Y)"; do
  output=$(bin/accumulo shell -u root -p "$PASS" -e "userpermissions -u $x" | 
fgrep "($TABLENAME)")
  if [[ $? -eq 0 ]]; then
echo "$x has $output"
  fi
done

The "grep -v" was to filter out the timestamped log4j messages. That may or may 
not be necessary for you, depending on your client's log4j configuration.

Hope that helps.

 

On Fri, Sep 23, 2016 at 2:33 PM Jeff Kubina  wrote:

>From the accumulo shell how do I list all the users who have access to a 
>specific table?

 

Re: 1 of 20 TServers unresponsive/slow, all writes fail?

[dlmarion]

We have seen this before: a tserver that is hosting metadata tablets has issues 
and starts causing problems within the cluster. You could try using the 
HostRegexTableLoadBalancer[1,2] to segregate your metadata tablets from the 
other tables. This doesn't fully eliminate the SPOF, but it should help to 
ensure that the tablet servers hosting the metadata tablets are not busy doing 
work for other tables. 

To do this you would do the following in the shell, then restart the master: 

1) Set the 'master.tablet.balancer' property to the HostRegexTableLoadBalancer 
class name 
2) Set the property 
'table.custom.balancer.host.regex.accumulo.metadata=' 
3) Set other HostRegexTableLoadBalancer properties if desired 

[1] https://issues.apache.org/jira/browse/ACCUMULO-4173 
[2] 
https://github.com/apache/accumulo/blob/rel/1.7.2/server/base/src/main/java/org/apache/accumulo/server/master/balancer/HostRegexTableLoadBalancer.java
 

- Original Message -

From: "Michael Moss"  
To: user@accumulo.apache.org 
Cc: "Michael Moss"  
Sent: Friday, September 9, 2016 10:44:44 AM 
Subject: Re: 1 of 20 TServers unresponsive/slow, all writes fail? 

1.7.2 (client still 1.6.2). 

I think its an overall design issue, no? Serving metadata is a SPOF? 

On Fri, Sep 9, 2016 at 10:41 AM, Christopher < ctubb...@apache.org > wrote: 



What version of Accumulo? Could narrow down the search for known issue 
potentials. 

On Fri, Sep 9, 2016 at 10:36 AM Michael Moss < michael.m...@gmail.com > wrote: 



Upon further internal discussion, it looks like the metadata/root tables are 
served from the tservers (not an HA master for example) and the one in question 
was serving it. It was unable to run MajC (compaction) for many hours leading 
up to the time where it couldn't service requests any longer, but it was still 
up, hosting tablets, just very slow or unable to respond. So all writes ended 
up timing out. 

If this condition is possible and there is a SPOF here, it'd be good to see 
what's on the roadmap to address it. 

On Fri, Sep 9, 2016 at 10:24 AM, < dlmar...@comcast.net > wrote: 



What was happening on that 1 tserver? Was it in garbage collection? Was it 
having network or O/S issues? 


From: "Michael Moss (BLOOMBERG/ 731 LEX)" < mmos...@bloomberg.net > 
To: user@accumulo.apache.org 
Sent: Friday, September 9, 2016 9:40:42 AM 
Subject: 1 of 20 TServers unresponsive/slow, all writes fail? 


Hi, 

We are starting to investigate an issue where 1 tserver was up, but became 
slow/unresponsive for several hours, yet all writes to our 20+ servers began to 
fail. We could see leading up to the failure that the writes were distributed 
among all of the tablet servers, so it wasn't a hotspot. Whenever we receive a 
MutationsRejectedException, we recreate the BatchWriter (ACCUMULO-2990). I'm 
digging into the TabletServerBatchWriter code, but any ideas what could cause 
this issue? Is there some sort of initialization or healthchecking that the 
client does where 1 server could impact all? 

Thanks. 

-Mike 

Caused by: org.apache.accumulo.core.client.TimedOutException: Servers timed out 
[ pnj-bvlt-r4n03.abc.com:31113 ] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$TimeoutTracker.wroteNothing(TabletServerBatchWriter.java:177)
 ~[stormjar.jar:1.0] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$TimeoutTracker.errorOccured(TabletServerBatchWriter.java:182)
 ~[stormjar.jar:1.0] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:933)
 ~[stormjar.jar:1.0] at 

Re: 1 of 20 TServers unresponsive/slow, all writes fail?

[dlmarion]

What was happening on that 1 tserver? Was it in garbage collection? Was it 
having network or O/S issues? 

- Original Message -

From: "Michael Moss (BLOOMBERG/ 731 LEX)"  
To: user@accumulo.apache.org 
Sent: Friday, September 9, 2016 9:40:42 AM 
Subject: 1 of 20 TServers unresponsive/slow, all writes fail? 

Hi, 

We are starting to investigate an issue where 1 tserver was up, but became 
slow/unresponsive for several hours, yet all writes to our 20+ servers began to 
fail. We could see leading up to the failure that the writes were distributed 
among all of the tablet servers, so it wasn't a hotspot. Whenever we receive a 
MutationsRejectedException, we recreate the BatchWriter (ACCUMULO-2990). I'm 
digging into the TabletServerBatchWriter code, but any ideas what could cause 
this issue? Is there some sort of initialization or healthchecking that the 
client does where 1 server could impact all? 

Thanks. 

-Mike 

Caused by: org.apache.accumulo.core.client.TimedOutException: Servers timed out 
[pnj-bvlt-r4n03.abc.com:31113] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$TimeoutTracker.wroteNothing(TabletServerBatchWriter.java:177)
 ~[stormjar.jar:1.0] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$TimeoutTracker.errorOccured(TabletServerBatchWriter.java:182)
 ~[stormjar.jar:1.0] at 
org.apache.accumulo.core.client.impl.TabletServerBatchWriter$MutationWriter.sendMutationsToTabletServer(TabletServerBatchWriter.java:933)
 ~[stormjar.jar:1.0] at 

Re: Accumulo Seek performance

[dlmarion]

Calling BatchScanner.iterator() is what starts the work on the server side. You 
should do this first for all 6 batch scanners, then iterate over all of them in 
parallel. 

- Original Message -

From: "Sven Hodapp"  
To: "user"  
Sent: Thursday, August 25, 2016 4:53:41 AM 
Subject: Re: Accumulo Seek performance 

Hi, 

I've changed the code a little bit, so that it uses a thread pool (via the 
Future): 

val ranges500 = ranges.asScala.grouped(500) // this means 6 BatchScanners will 
be created 

for (ranges <- ranges500) { 
val bscan = instance.createBatchScanner(ARTIFACTS, auths, 2) 
bscan.setRanges(ranges.asJava) 
Future { 
time("mult-scanner") { 
bscan.asScala.toList // toList forces the iteration of the iterator 
} 
} 
} 

Here are the results: 

background log: info: mult-scanner time: 4807.289358 ms 
background log: info: mult-scanner time: 4930.996522 ms 
background log: info: mult-scanner time: 9510.010808 ms 
background log: info: mult-scanner time: 11394.152391 ms 
background log: info: mult-scanner time: 13297.247295 ms 
background log: info: mult-scanner time: 14032.704837 ms 

background log: info: single-scanner time: 15322.624393 ms 

Every Future completes independent, but in return every batch scanner iterator 
needs more time to complete. :( 
This means the batch scanners aren't really processed in parallel on the server 
side? 
Should I reconfigure something? Maybe the tablet servers haven't/can't allocate 
enough threads or memory? (Every of the two nodes has 8 cores and 64GB memory 
and a storage with ~300MB/s...) 

Regards, 
Sven 

-- 
Sven Hodapp, M.Sc., 
Fraunhofer Institute for Algorithms and Scientific Computing SCAI, 
Department of Bioinformatics 
Schloss Birlinghoven, 53754 Sankt Augustin, Germany 
sven.hod...@scai.fraunhofer.de 
www.scai.fraunhofer.de 

- Ursprüngliche Mail - 
> Von: "Josh Elser"  
> An: "user"  
> Gesendet: Mittwoch, 24. August 2016 18:36:42 
> Betreff: Re: Accumulo Seek performance 

> Ahh duh. Bad advice from me in the first place :) 
> 
> Throw 'em in a threadpool locally. 
> 
> dlmar...@comcast.net wrote: 
>> Doesn't this use the 6 batch scanners serially? 
>> 
>>  
>> *From: *"Sven Hodapp"  
>> *To: *"user"  
>> *Sent: *Wednesday, August 24, 2016 11:56:14 AM 
>> *Subject: *Re: Accumulo Seek performance 
>> 
>> Hi Josh, 
>> 
>> thanks for your reply! 
>> 
>> I've tested your suggestion with a implementation like that: 
>> 
>> val ranges500 = ranges.asScala.grouped(500) // this means 6 
>> BatchScanners will be created 
>> 
>> time("mult-scanner") { 
>> for (ranges <- ranges500) { 
>> val bscan = instance.createBatchScanner(ARTIFACTS, auths, 1) 
>> bscan.setRanges(ranges.asJava) 
>> for (entry <- bscan.asScala) yield { 
>> entry.getKey() 
>> } 
>> } 
>> } 
>> 
>> And the result is a bit disappointing: 
>> 
>> background log: info: mult-scanner time: 18064.969281 ms 
>> background log: info: single-scanner time: 6527.482383 ms 
>> 
>> I'm doing something wrong here? 
>> 
>> 
>> Regards, 
>> Sven 
>> 
>> -- 
>> Sven Hodapp, M.Sc., 
>> Fraunhofer Institute for Algorithms and Scientific Computing SCAI, 
>> Department of Bioinformatics 
>> Schloss Birlinghoven, 53754 Sankt Augustin, Germany 
>> sven.hod...@scai.fraunhofer.de 
>> www.scai.fraunhofer.de 
>> 
>> - Ursprüngliche Mail - 
>> > Von: "Josh Elser"  
>> > An: "user"  
>> > Gesendet: Mittwoch, 24. August 2016 16:33:37 
>> > Betreff: Re: Accumulo Seek performance 
>> 
>> > This reminded me of https://issues.apache.org/jira/browse/ACCUMULO-3710 
>> > 
>> > I don't feel like 3000 ranges is too many, but this isn't quantitative. 
>> > 
>> > IIRC, the BatchScanner will take each Range you provide, bin each Range 
>> > to the TabletServer(s) currently hosting the corresponding data, clip 
>> > (truncate) each Range to match the Tablet boundaries, and then does an 
>> > RPC to each TabletServer with just the Ranges hosted there. 
>> > 
>> > Inside the TabletServer, it will then have many Ranges, binned by Tablet 
>> > (KeyExtent, to be precise). This will spawn a 
>> > org.apache.accumulo.tserver.scan.LookupTask will will start collecting 
>> > results to send back to the client. 
>> > 
>> > The caveat here is that those ranges are processed serially on a 
>> > TabletServer. Maybe, you're swamping one TabletServer with lots of 
>> > Ranges that it could be processing in parallel. 
>> > 
>> > Could you experiment with using multiple BatchScanners and something 
>> > like Guava's Iterables.concat to make it appear like one Iterator? 
>> > 
>> > I'm curious if we should put an optimization into the BatchScanner 
>> > itself to limit the number of ranges we send in one RPC to a 
>> > TabletServer (e.g. one 

Re: Accumulo Seek performance

[dlmarion]

Doesn't this use the 6 batch scanners serially? 

- Original Message -

From: "Sven Hodapp"  
To: "user"  
Sent: Wednesday, August 24, 2016 11:56:14 AM 
Subject: Re: Accumulo Seek performance 

Hi Josh, 

thanks for your reply! 

I've tested your suggestion with a implementation like that: 

val ranges500 = ranges.asScala.grouped(500) // this means 6 BatchScanners will 
be created 

time("mult-scanner") { 
for (ranges <- ranges500) { 
val bscan = instance.createBatchScanner(ARTIFACTS, auths, 1) 
bscan.setRanges(ranges.asJava) 
for (entry <- bscan.asScala) yield { 
entry.getKey() 
} 
} 
} 

And the result is a bit disappointing: 

background log: info: mult-scanner time: 18064.969281 ms 
background log: info: single-scanner time: 6527.482383 ms 

I'm doing something wrong here? 


Regards, 
Sven 

-- 
Sven Hodapp, M.Sc., 
Fraunhofer Institute for Algorithms and Scientific Computing SCAI, 
Department of Bioinformatics 
Schloss Birlinghoven, 53754 Sankt Augustin, Germany 
sven.hod...@scai.fraunhofer.de 
www.scai.fraunhofer.de 

- Ursprüngliche Mail - 
> Von: "Josh Elser"  
> An: "user"  
> Gesendet: Mittwoch, 24. August 2016 16:33:37 
> Betreff: Re: Accumulo Seek performance 

> This reminded me of https://issues.apache.org/jira/browse/ACCUMULO-3710 
> 
> I don't feel like 3000 ranges is too many, but this isn't quantitative. 
> 
> IIRC, the BatchScanner will take each Range you provide, bin each Range 
> to the TabletServer(s) currently hosting the corresponding data, clip 
> (truncate) each Range to match the Tablet boundaries, and then does an 
> RPC to each TabletServer with just the Ranges hosted there. 
> 
> Inside the TabletServer, it will then have many Ranges, binned by Tablet 
> (KeyExtent, to be precise). This will spawn a 
> org.apache.accumulo.tserver.scan.LookupTask will will start collecting 
> results to send back to the client. 
> 
> The caveat here is that those ranges are processed serially on a 
> TabletServer. Maybe, you're swamping one TabletServer with lots of 
> Ranges that it could be processing in parallel. 
> 
> Could you experiment with using multiple BatchScanners and something 
> like Guava's Iterables.concat to make it appear like one Iterator? 
> 
> I'm curious if we should put an optimization into the BatchScanner 
> itself to limit the number of ranges we send in one RPC to a 
> TabletServer (e.g. one BatchScanner might open multiple 
> MultiScanSessions to a TabletServer). 
> 
> Sven Hodapp wrote: 
>> Hi there, 
>> 
>> currently we're experimenting with a two node Accumulo cluster (two tablet 
>> servers) setup for document storage. 
>> This documents are decomposed up to the sentence level. 
>> 
>> Now I'm using a BatchScanner to assemble the full document like this: 
>> 
>> val bscan = instance.createBatchScanner(ARTIFACTS, auths, 10) // ARTIFACTS 
>> table 
>> currently hosts ~30GB data, ~200M entries on ~45 tablets 
>> bscan.setRanges(ranges) // there are like 3000 Range.exact's in the 
>> ranges-list 
>> for (entry<- bscan.asScala) yield { 
>> val key = entry.getKey() 
>> val value = entry.getValue() 
>> // etc. 
>> } 
>> 
>> For larger full documents (e.g. 3000 exact ranges), this operation will take 
>> about 12 seconds. 
>> But shorter documents are assembled blazing fast... 
>> 
>> Is that to much for a BatchScanner / I'm misusing the BatchScaner? 
>> Is that a normal time for such a (seek) operation? 
>> Can I do something to get a better seek performance? 
>> 
>> Note: I have already enabled bloom filtering on that table. 
>> 
>> Thank you for any advice! 
>> 
>> Regards, 
>> Sven 

Re: Configuring batch writers

[dlmarion]

The batch writer has several knobs (latency time, memory buffer, etc) that you 
can tune to meet your requirements. The values for those settings will depend 
on a lot of variables, to include: 

- number of tablet servers 
- size of mutations 
- desired latency 
- memory buffer 
- configuration settings on the table(s) and tablet servers. 

Suggest picking a starting point and see how it works for you, such as 

threads - equal to the number of tablet servers (unless you have a really large 
number of tablet servers) 
buffer - 100MB 
latency - 10 seconds 

If you are hitting a wall with those settings, you could increase the buffer 
and latency and/or change some settings on the server side that have to do with 
the write ahead logs. 

- Original Message -

From: "Jamie Johnson"  
To: user@accumulo.apache.org 
Sent: Friday, July 15, 2016 2:16:40 PM 
Subject: Configuring batch writers 

Is there any documentation that outlines reasonable settings for batch writers 
given a known ingest rate? For instance if I have a source that is producing in 
the neighborhood of 15MB of mutations per second, what would a reasonable 
configuration for the batch writer be to handle an ingest at this rate? What 
are reasonable rules of thumb to follow to ensure that the writers don't block, 
etc? 

Re: java.lang.NoClassDefFoundError with fields of custom Filter

[dlmarion]

+1 good catch Josh 

- Original Message -

From: "Josh Elser"  
To: user@accumulo.apache.org 
Sent: Thursday, July 7, 2016 2:12:42 PM 
Subject: Re: java.lang.NoClassDefFoundError with fields of custom Filter 

Beware using the HDFS classloader in any Accumulo release that does not 
contain commons-vfs-2.1 as a dependency. 

Commons-vfs-2.0 has multiple known issues which prevent it from being 
usable in the more basic sense. 

Presently, there is only one release which contains the fix already: 
Accumulo 1.7.2. The upcoming 1.6.6 and 1.8.0 releases will also have the 
updated dependency. 

Massimilian Mattetti wrote: 
> Hi Jim, 
> 
> the approach of using namespace from HDFS looks promising. I need to 
> investigate a little on how it works but I guess I will take your advice. 
> Thank you. 
> 
> Cheers, 
> Massimiliano 
> 
> 
> 
> 
> From: James Hughes  
> To: user@accumulo.apache.org 
> Date: 07/07/2016 08:28 PM 
> Subject: Re: java.lang.NoClassDefFoundError with fields of custom Filter 
>  
> 
> 
> 
> Hi Massimiliano, 
> 
> I'm a fan of producing uber jars for this kind of thing; we do that for 
> GeoMesa. There is one gotcha which can come up: if you have several uber 
> jars in lib/ext, they can collide in rather unexpected ways. 
> 
> There are two options to call out: 
> 
> First, Accumulo has support for loading jars from HDFS into namespaces. 
> With that, you could have various namespaces for different versions or 
> different collections of iterator projects. If you are sharing a dev 
> cloud with other projects or co-workers working on the same project that 
> can be helpful since it would avoid restarts, etc. Big thumbs-up for 
> this approach! 
> 
> Second, rather than having an uber jar, you could build up zip files 
> with the various jars you need for your iterators and unzip them in 
> lib/ext. If you did that for multiple competing iterator projects, you'd 
> avoid duplication of code inside uber-jars. Also, you'd be able to see 
> if there are 8 versions of Log4J and Guava in lib/ext...;) It wouldn't 
> be as powerful as the namespace, but there's something nice about having 
> a low-tech approach. 
> 
> Others will likely have varied experiences; I'm not sure if there's an 
> established 'best practice' here. 
> 
> Cheers, 
> 
> Jim 
> 
> On Thu, Jul 7, 2016 at 12:56 PM, Massimilian Mattetti 
> <_massi...@il.ibm.com_ > wrote: 
> Thanks for your prompt response, you are right Jim. There is a static 
> dependency to Log4J in my lexicoder. Adding the Log4J Jar to the 
> classpath solved the problem. 
> Would you suggest to use an Uber Jar to avoid this kind of problems? 
> 
> Regards, 
> Massimiliano 
> 
> 
> 
> 
> From: James Hughes <_jnh5y@virginia.edu_ > 
> To: _user@accumulo.apache.org_  
> Date: 07/07/2016 06:25 PM 
> Subject: Re: java.lang.NoClassDefFoundError with fields of custom Filter 
>  
> 
> 
> 
> 
> Hi Massimilian, 
> 
> As a quick note, your error says that it could not initialize class 
> accumulo.lexicoders.MyLexicoder. Did you provide all the dependencies 
> for your class on Accumulo's classpath? 
> 
> That exception (or similar) can occur if there is a static block in your 
> MyLexicoder class which can't run properly. 
> 
> Cheers, 
> 
> Jim 
> 
> 
> On Thu, Jul 7, 2016 at 11:19 AM, Massimilian Mattetti 
> <_massi...@il.ibm.com_ > wrote: 
> Hi, 
> 
> I have implemented a custom filter and a custom lexicoder. Both this two 
> classes are packed in the same jar that has been deployed under the 
> directory $ACCUMULO_HOME/*lib*/*ext*of my Accumulo servers (version 
> 1.7.1). The lexicoder is used by the filter to get the real object from 
> the accumulo value and test some conditions on it. When I tried to scan 
> the table applying this filter I got the following exception: 
> 
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
> accumulo.lexicoders.MyLexicoder 
> at accumulo.filters.MyFilter.(MyFilter.java:24) 
> at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source) 
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>  
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) 
> at java.lang.Class.newInstance(Class.java:442) 
> at 
> org.apache.accumulo.core.iterators.IteratorUtil.loadIterators(IteratorUtil.java:261)
>  
> at 
> org.apache.accumulo.core.iterators.IteratorUtil.loadIterators(IteratorUtil.java:237)
>  
> at 
> org.apache.accumulo.core.iterators.IteratorUtil.loadIterators(IteratorUtil.java:218)
>  
> at 
> org.apache.accumulo.core.iterators.IteratorUtil.loadIterators(IteratorUtil.java:205)
>  
> at 
> 

[ANNOUNCE] Timely - Secure Time Series Database

[dlmarion]

Timely is a time series database application that provides secure access to 
time series data. It is designed to be used with Apache Accumulo for 
persistence and Grafana for visualization. Timely is located at 
https://github.com/NationalSecurityAgency/timely . 

Re: Class path for shell commands

[dlmarion]

Another option is to drop your jar into $ACCUMULO_HOME/lib/ext on the node 
where you will be running the shell. 


- Original Message -

From: "Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX)"  
To: user@accumulo.apache.org 
Sent: Monday, March 7, 2016 4:31:49 PM 
Subject: Re: Class path for shell commands 

Yes I am using 1.6.2 version. I am not sure if we can upgrade the version now. 
Could you please let me know if there is any other way (temporary way) to fix 
that bug? Also how to ensure that this error is coming because of that bug? 

From: user@accumulo.apache.org At: Mar 7 2016 16:28:15 
To: Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) , user@accumulo.apache.org 
Subject: Re: Class path for shell commands 



Are you using a 1.6.x or earlier version? I think we fixed a bug in 1.7.0 where 
the user-specified CLASSPATH was overridden. 


On Mon, Mar 7, 2016 at 4:23 PM Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) < 
sjav...@bloomberg.net > wrote: 



Sorry that's typo. I used CLASSPATH, still not working. 

From: user@accumulo.apache.org At: Mar 7 2016 16:23:04 
To: Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) , user@accumulo.apache.org 
Subject: Re: Class path for shell commands 



Try $CLASSPATH instead of $CLASS_PATH 

On Mon, Mar 7, 2016 at 4:11 PM Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) < 
sjav...@bloomberg.net > wrote: 



Hello Everyone, 

I built custom formatter by extending DefaultFormatter class. When I am trying 
to use DefaultFormatter, I am getting "Class not found" exception. Below are 
the steps I followed: 

>From local machine: 

$ export CLASS_PATH=$CLASS_PATH:/*.jar 

$ CONNECTED to accumulo shell server 

$ formatter -f fully_qualified_classname -t table_name 

$ scan -t table 
ERROR: Class not found 


Could someone please let me know any other way to set class path for shell? 
Also is there anyway to debug this issue like checking current classpaths that 
shell is using? 

Thanks for your time. 

- 
Regards, 
Sravan 

Re: Class path for shell commands

[dlmarion]

try: ./accumulo -add  shell 


- Original Message -

From: "Christopher"  
To: user@accumulo.apache.org, "Sravankumar Reddy Javaji" 
 
Sent: Monday, March 7, 2016 4:28:03 PM 
Subject: Re: Class path for shell commands 

Are you using a 1.6.x or earlier version? I think we fixed a bug in 1.7.0 where 
the user-specified CLASSPATH was overridden. 


On Mon, Mar 7, 2016 at 4:23 PM Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) < 
sjav...@bloomberg.net > wrote: 



Sorry that's typo. I used CLASSPATH, still not working. 

From: user@accumulo.apache.org At: Mar 7 2016 16:23:04 
To: Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) , user@accumulo.apache.org 
Subject: Re: Class path for shell commands 



Try $CLASSPATH instead of $CLASS_PATH 

On Mon, Mar 7, 2016 at 4:11 PM Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX) < 
sjav...@bloomberg.net > wrote: 



Hello Everyone, 

I built custom formatter by extending DefaultFormatter class. When I am trying 
to use DefaultFormatter, I am getting "Class not found" exception. Below are 
the steps I followed: 

>From local machine: 

$ export CLASS_PATH=$CLASS_PATH:/*.jar 

$ CONNECTED to accumulo shell server 

$ formatter -f fully_qualified_classname -t table_name 

$ scan -t table 
ERROR: Class not found 


Could someone please let me know any other way to set class path for shell? 
Also is there anyway to debug this issue like checking current classpaths that 
shell is using? 

Thanks for your time. 

- 
Regards, 
Sravan 

Re: loading iterator jars from file:// or http://

[dlmarion]

Suggest replacing the VFS 2.0 jar in the Accumulo lib directory with a VFS 
2.1-SNAPSHOT jar from [1]. There are some bugs that have been fixed, but VFS 
2.1 has not been released yet. I did this and have not had many issues loading 
from HDFS since. Loading jars from file:/// should work, I believe some of the 
Accumulo tests are written that way. 

[1] 
https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=core/target
 


- Original Message -

From: "Jonathan Lasko"  
To: "accumulo-user"  
Sent: Tuesday, November 24, 2015 3:09:20 PM 
Subject: loading iterator jars from file:// or http:// 

My project is loading custom iterator jars from HDFS and have run into 
some VFS problems (e.g. redeploying the same jar to HDFS causes 
NoClassDefFoundErrors due to NPEs in VFS). While one solution may be to 
weed out and eliminate any such "redeploys," I am curious if anyone has 
used contexts to load jars from file:// or http:// locations? 

Thanks, 

Jonathan 

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

[dlmarion]

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been 
fixed, but one of them is that if more than one monitored resource changed then 
it would miss some of them. 


- Original Message -

From: "Rob Povey"  
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 11:57:27 AM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Thanks for the help with this, 

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change. 

These are from my dev box, but the clusters look the same just with many more 
contexts 

default | general.vfs.classpaths . | 
site | general.vfs.context.classpath.rob_maanaNgram ... | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
 
site | general.vfs.context.classpath.rob_maanaSearch .. | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar
 

And then the context is set on the table like this 


default | table.classpath.context  | 
table | @override ... | rob_maanaNgram 



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the config classpaths. 

 
tserver.memory.maps.max 
2G 
 


 
tserver.memory.maps.native.enabled 
true 
 


 
tserver.cache.data.size 
4G 
 


 
tserver.cache.index.size 
3G 
 


 
trace.token.property.password 
maana 
 


 
trace.user 
root 
 


 
tserver.sort.buffer.size 
200M 
 


 
tserver.walog.max.size 
1G 
 


 
general.classpaths 
 
$ACCUMULO_HOME/lib/accumulo-server.jar, 
$ACCUMULO_HOME/lib/accumulo-core.jar, 
$ACCUMULO_HOME/lib/accumulo-start.jar, 
$ACCUMULO_HOME/lib/accumulo-fate.jar, 
$ACCUMULO_HOME/lib/accumulo-proxy.jar, 
$ACCUMULO_HOME/lib/[^.].*.jar, 
$ZOOKEEPER_HOME/zookeeper[^.].*.jar, 
$HADOOP_CONF_DIR, 
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar, 
/usr/hdp/current/hadoop-client/[^.].*.jar, 
/usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, 
/usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, 
$HADOOP_PREFIX/[^.].*.jar, 
$HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar, 
 
Classpaths that accumulo checks for updates and class 
files. 
 


 
instance.volumes 
hdfs://maana2/apps/accumulo 
 









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote: 

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the concept of classloader 
>contexts which are basically names for locations for jars. Table(s) can be 
>configured to use a classloader context allowing you to deploy server side 
>code for different applications in different locations. This new classloader 
>does "reload" jars on the classpath when they change; the same behavior with 
>the older classloader reading from lib/ext. You can read more about this 
>feature at [1]. 
> 
>We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
>fixed and you will have a better experience if you replace the VFS jar in the 
>lib directory with a snapshot of the 2.1 release[2]. 
> 
>[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader 
>[2] 
>https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target
> 
> 
> 
>> -Original Message- 
>> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net] 
>> Sent: Thursday, October 29, 2015 8:04 PM 
>> To: user@accumulo.apache.org 
>> Subject: RE: Accumulo Iterator painful development because TS don't pick up 
>> changes to Jars 
>> 
>> Can you provide the relevant classpath sections of your accumulo-site.xml 
>> file? 
>> 
>> > -Original Message- 
>> > From: Rob Povey [mailto:r...@maana.io] 
>> > Sent: Thursday, October 29, 2015 8:01 PM 
>> > To: user@accumulo.apache.org 
>> > Subject: Accumulo Iterator painful development because TS don't pick 
>> > up changes to Jars 
>> > 

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

[dlmarion]

Also, turn the logging on the tservers up to DEBUG for 
org.apache.accumulo.start.classloader.*. You should see a line in the log that 
starts with "monitoring " 


- Original Message -

From: dlmar...@comcast.net 
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 12:22:53 PM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been 
fixed, but one of them is that if more than one monitored resource changed then 
it would miss some of them. 


- Original Message -

From: "Rob Povey"  
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 11:57:27 AM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Thanks for the help with this, 

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change. 

These are from my dev box, but the clusters look the same just with many more 
contexts 

default | general.vfs.classpaths . | 
site | general.vfs.context.classpath.rob_maanaNgram ... | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
 
site | general.vfs.context.classpath.rob_maanaSearch .. | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar
 

And then the context is set on the table like this 


default | table.classpath.context  | 
table | @override ... | rob_maanaNgram 



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the config classpaths. 

 
tserver.memory.maps.max 
2G 
 


 
tserver.memory.maps.native.enabled 
true 
 


 
tserver.cache.data.size 
4G 
 


 
tserver.cache.index.size 
3G 
 


 
trace.token.property.password 
maana 
 


 
trace.user 
root 
 


 
tserver.sort.buffer.size 
200M 
 


 
tserver.walog.max.size 
1G 
 


 
general.classpaths 
 
$ACCUMULO_HOME/lib/accumulo-server.jar, 
$ACCUMULO_HOME/lib/accumulo-core.jar, 
$ACCUMULO_HOME/lib/accumulo-start.jar, 
$ACCUMULO_HOME/lib/accumulo-fate.jar, 
$ACCUMULO_HOME/lib/accumulo-proxy.jar, 
$ACCUMULO_HOME/lib/[^.].*.jar, 
$ZOOKEEPER_HOME/zookeeper[^.].*.jar, 
$HADOOP_CONF_DIR, 
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar, 
/usr/hdp/current/hadoop-client/[^.].*.jar, 
/usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, 
/usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, 
$HADOOP_PREFIX/[^.].*.jar, 
$HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar, 
 
Classpaths that accumulo checks for updates and class 
files. 
 


 
instance.volumes 
hdfs://maana2/apps/accumulo 
 









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote: 

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the concept of classloader 
>contexts which are basically names for locations for jars. Table(s) can be 
>configured to use a classloader context allowing you to deploy server side 
>code for different applications in different locations. This new classloader 
>does "reload" jars on the classpath when they change; the same behavior with 
>the older classloader reading from lib/ext. You can read more about this 
>feature at [1]. 
> 
>We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
>fixed and you will have a better experience if you replace the VFS jar in the 
>lib directory with a snapshot of the 2.1 release[2]. 
> 
>[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader 
>[2] 
>https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target
> 
> 
> 
>> -Original Message- 
>> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net] 
>> Sent: Thursday, October 29, 2015 8:04 PM 
>> To: user@accumulo.apache.org 
>> Subject: RE: Accumulo Iterator painful development because TS 

RE: Accumulo Iterator painful development because TS don't pick up changes to Jars

[dlmarion]

So, without seeing your configuration, I would suggest trying something before 
upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
different locations. At the same time, we added the concept of classloader 
contexts which are basically names for locations for jars. Table(s) can be 
configured to use a classloader context allowing you to deploy server side code 
for different applications in different locations. This new classloader does 
"reload" jars on the classpath when they change; the same behavior with the 
older classloader reading from lib/ext. You can read more about this feature at 
[1].

We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
fixed and you will have a better experience if you replace the VFS jar in the 
lib directory with a snapshot of the 2.1 release[2].

[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader
[2] 
https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target


> -Original Message-
> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net]
> Sent: Thursday, October 29, 2015 8:04 PM
> To: user@accumulo.apache.org
> Subject: RE: Accumulo Iterator painful development because TS don't pick up
> changes to Jars
> 
> Can you provide the relevant classpath sections of your accumulo-site.xml
> file?
> 
> > -Original Message-
> > From: Rob Povey [mailto:r...@maana.io]
> > Sent: Thursday, October 29, 2015 8:01 PM
> > To: user@accumulo.apache.org
> > Subject: Accumulo Iterator painful development because TS don't pick
> > up changes to Jars
> >
> > Caveat I’m still running 1.6.2 internally here, and things may have
> > changed and I could simply “be doing it wrong”, or have missed the
> > solution in the docs. It’s also probably not a typical use case.
> >
> > This is not really an issue for most day to day development, but our
> > internal testing process makes this changing iterators a nightmare.
> > Before I start I am aware of general.dynamic.classpaths, and because
> > it appears that wildcards are only respected at the file level, which
> > is insufficient for our use case as you’ll see later.
> >
> > I’ll try and explain our internal test environment to help understand
> > the issue.
> > We run daily (or more frequent) drops of our codebase against two
> > internal clusters across a variety of data sources (most of them
> > aren’t particularly large).
> > To give some idea I count 462 tables on one of of the clusters and
> > each instance of the application is using 11 or so tables of which 4
> > or so have a variety of iterators we’ve written.
> > To resolve the conflicts since our application predates namespaces we
> > prefix the tables and the table contexts and upload the iterators to
> > subdirectories with matching names.
> > To complicate matters further many of the tables are dropped and new
> > tables added at a pretty frightening rate, so having to change the
> > configuration, and restart servers to add a new path to the
> > dynamic.classpath property is something of a none starter.
> >
> > It all works fine until a build has a change in an iterator and is
> > targeted against an existing table, the app correctly identifies and
> > uploads the new jars, but accumulo obviously doesn’t pick up the
> > change. In many cases I could live with it if simply dropping the
> > tables and reingesting was sufficient, but short of ingesting into a
> > new table name even that doesn’t always pick up the new Iterators.
> > We have currently resorted to manually tracking every iterator change
> > (the rate of which has at least slowed down recently) and doing
> > rolling restarts of tablet servers on off hours, but we end up often
> > not knowing if an bug is real or an issue in a TS having an old iterator 
> > loaded.
> >
> > Is there a way to get the TS to watch an entire subtree for Jar changes?
> >
> > Assuming there isn’t, when I get a few days without a looming
> > deliverable, I was going to migrate to 1.7 and if that has the same
> > issue look at making and submitting a fix.
> >
> >
> > Rob Povey
> >
> >
> >
> >
> >
> >
> > On 10/28/15, 2:25 PM, "Josh Elser"  wrote:
> >
> > >Rob Povey wrote:
> > >> However I’m pretty reticent right now to add anymore iterators to
> > >> our project, they’ve been a test nightmare for us internally.
> > >
> > >Off-topic, I'd like to hear more about what is painful. Do you have
> > >the time to fork off a thread and let us know how it hurts?

Re: Tserver's strange state.

[dlmarion]

Are you trying to shut the whole system down, or just a couple of tablet 
servers?Is your application reading and writing from/to Accumulo during this 
time?




 Original message 
From: Denis  
Date: 10/22/2015  6:03 PM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Re: Tserver's strange state. 

Both servers has the errors in the logs like these:


2015-10-22 03:28:00,599 ERROR
org.apache.accumulo.core.client.impl.Writer: error sending update to
10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: 12 millis timeout while waiting
for channel to be ready for re
ad. ch : java.nio.channels.SocketChannel[connected
local=/10.2.142.1:36148 remote=/10.2.130.1:9997]
2015-10-22 03:28:04,283 ERROR
org.apache.accumulo.core.client.impl.Writer: error sending update to
10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: 12 millis timeout while waiting
for channel to be ready for re
ad. ch : java.nio.channels.SocketChannel[connected
local=/10.2.142.1:37047 remote=/10.2.130.1:9997]
2015-10-22 03:28:06,116 ERROR
org.apache.accumulo.core.client.impl.Writer: error sending update to
10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: 12 millis timeout while waiting
for channel to be ready for re
ad. ch : java.nio.channels.SocketChannel[connected
local=/10.2.142.1:37167 remote=/10.2.130.1:9997]


On 10/22/15, Denis  wrote:
> Hi
>
> Sometimes my Tablet Servers go into a strange state: they have some
> very old scans (see picture: http://i.imgur.com/2sOUM99.png) and being
> in this state they cannot be decomissioned gracefully using "accumulo
> stop" - number of their tablets decreases down to some fixed number
> (say from 6K tablets to 2K), not to zero.
> It is diffucult to reproduce.
> Now I have a live system with 2 tabletservers in this state.
> Any suggestions how to catch the bug?
>

RE: Tserver's strange state.

[dlmarion]

The errors in the logs regarding 10.2.130.1, was this during the time it was 
rebooted, or before? Was the tserver on 10.2.130.1 hung? I think we would 
likely need more information from the logs to determine what is occurring 
during this time. It might be best to open an issue in JIRA for this.

> -Original Message-
> From: webmas...@webmaster.ms [mailto:webmas...@webmaster.ms] On
> Behalf Of Denis
> Sent: Thursday, October 22, 2015 7:29 PM
> To: user@accumulo.apache.org
> Subject: Re: Tserver's strange state.
> 
> The server 10.2.130.1 has been rebooted.
> Yes, it is a production system with a lot of reads and writes.
> 
> On 10/22/15, dlmarion <dlmar...@comcast.net> wrote:
> >
> >
> > Are you trying to shut the whole system down, or just a couple of
> > tablet servers?Is your application reading and writing from/to
> > Accumulo during this time?
> >
> >
> >
> >
> >  Original message 
> > From: Denis <de...@camfex.cz>
> > Date: 10/22/2015  6:03 PM  (GMT-05:00)
> > To: user@accumulo.apache.org
> > Subject: Re: Tserver's strange state.
> >
> > Both servers has the errors in the logs like these:
> >
> > 
> > 2015-10-22 03:28:00,599 ERROR
> > org.apache.accumulo.core.client.impl.Writer: error sending update to
> > 10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
> > java.net.SocketTimeoutException: 12 millis timeout while waiting
> > for channel to be ready for re ad. ch :
> > java.nio.channels.SocketChannel[connected
> > local=/10.2.142.1:36148 remote=/10.2.130.1:9997]
> > 2015-10-22 03:28:04,283 ERROR
> > org.apache.accumulo.core.client.impl.Writer: error sending update to
> > 10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
> > java.net.SocketTimeoutException: 12 millis timeout while waiting
> > for channel to be ready for re ad. ch :
> > java.nio.channels.SocketChannel[connected
> > local=/10.2.142.1:37047 remote=/10.2.130.1:9997]
> > 2015-10-22 03:28:06,116 ERROR
> > org.apache.accumulo.core.client.impl.Writer: error sending update to
> > 10.2.130.1:9997: org.apache.thrift.transport.TTransportException:
> > java.net.SocketTimeoutException: 12 millis timeout while waiting
> > for channel to be ready for re ad. ch :
> > java.nio.channels.SocketChannel[connected
> > local=/10.2.142.1:37167 remote=/10.2.130.1:9997] 
> >
> > On 10/22/15, Denis <de...@camfex.cz> wrote:
> >> Hi
> >>
> >> Sometimes my Tablet Servers go into a strange state: they have some
> >> very old scans (see picture: http://i.imgur.com/2sOUM99.png) and
> >> being in this state they cannot be decomissioned gracefully using
> >> "accumulo stop" - number of their tablets decreases down to some
> >> fixed number (say from 6K tablets to 2K), not to zero.
> >> It is diffucult to reproduce.
> >> Now I have a live system with 2 tabletservers in this state.
> >> Any suggestions how to catch the bug?
> >>
> >

Re: How does Accumulo process a r-files for bulk ingesting?

[dlmarion]

I believe the RFile is assigned to each tablet that contain keys in the RFiles 
key range. So, 1 RFile is assigned to one or more tablets. The bulk import 
RFile is removed after it has been compacted by all assigned tablets. 

- Original Message -

From: "Jeff Kubina"  
To: user@accumulo.apache.org 
Sent: Wednesday, October 7, 2015 8:46:52 AM 
Subject: How does Accumulo process a r-files for bulk ingesting? 

How does Accumulo process an r-file for bulk ingesting when the key range of an 
r-file is within one tablet's key range and when the key range of an r-file 
spans two or more tablets? 

If the r-file is within one tablet's range I thought the file was "just 
renamed" and added to the tablet's list of r-files. Is that correct? 

If the key range of the r-file spans two or more files is the r-file 
partitioned into separate r-files for each appropriate tablet server or are the 
records "batch-written" to each appropriate tablet in memory? 

Re: [ADVISORY] Possible data loss during HDFS decommissioning

[dlmarion]

Known issue in the release notes on the web page? We would have to update every 
version though. Seems like we need a known issues document that lists issues in 
dependencies that transcend Accumulo versions. 

- Original Message -

From: "Josh Elser"  
To: d...@accumulo.apache.org 
Cc: user@accumulo.apache.org 
Sent: Wednesday, September 23, 2015 10:26:50 AM 
Subject: Re: [ADVISORY] Possible data loss during HDFS decommissioning 

What kind of documentation can we put in the user manual about this? 
Recommend to only decom one rack at a time until we get the issue sorted 
out in Hadoop-land? 

dlmar...@comcast.net wrote: 
> BLUF: There exists the possibility of data loss when performing DataNode 
> decommissioning with Accumulo running. This note applies to installations of 
> Accumulo 1.5.0+ and Hadoop 2.5.0+. 
> 
> DETAILS: During DataNode decommissioning it is possible for the NameNode to 
> report stale block locations (HDFS-8208). If Accumulo is running during this 
> process then it is possible that files currently being written will not close 
> properly. Accumulo is affected in two ways: 
> 
> 1. During compactions temporary rfiles are created, then closed, and renamed. 
> If a failure happens during the close, the compaction will fail. 
> 2. Write ahead log files are created, written to, and then closed. If a 
> failure happens during the close, then the NameNode will have a walog file 
> with no finalized blocks. 
> 
> If either of these cases happen, decommissioning of the DataNode could hang 
> (HDFS-3599, HDFS-5579) because the files are left in an open for write state. 
> If Accumulo needs the write ahead log for recovery it will be unable to read 
> the file and will not recover. 
> 
> RECOMMENDATION: Assuming that the replication pipeline for the write ahead 
> log is working properly, then you should not run into this issue if you only 
> decommission one rack at a time. 
> 

Re: Presplitting tables for the YCSB workloads

[dlmarion]

I don't have a script for you, but if you need to create one you could use the 
script command in the shell to do something similar to the hbase script. Some 
examples are in the comments in jira[1]. If you can figure out how you want the 
table split, and it can be scripted, I might have time this weekend to write it 
up for you. 

[1] https://issues.apache.org/jira/browse/ACCUMULO-1399 


- Original Message -

From: "Sean Busbey"  
To: "Accumulo User List"  
Sent: Thursday, September 17, 2015 10:10:29 PM 
Subject: Presplitting tables for the YCSB workloads 

YCSB is gearing up for its next monthly release, and I really want to 
add in an Accumulo specific README for running workloads. 

This is generally so that folks have an easier time running tests 
themselves. It's also because I keep testing Accumulo for the YCSB 
releases and coupled with a README file we'd get an Accumulo-specific 
convenience binary. Avoiding the bulk of dependencies that get 
included in the generic YCSB distribution artifact is a big win. 

The thing I keep getting hung up on is remembering how to properly 
split the Accumulo table for YCSB workloads. The HBase README has a 
great hbase shell snippet for doing this (because users can copy/paste 
it)[1]: 

 
3. Create a HBase table for testing 

For best results, use the pre-splitting strategy recommended in HBASE-4163: 

hbase(main):001:0> n_splits = 200 # HBase recommends (10 * number of 
regionservers) 
hbase(main):002:0> create 'usertable', 'family', {SPLITS => 
(1..n_splits).map {|i| "user#{1000+i*(-1000)/n_splits}"}} 

Failing to do so will cause all writes to initially target a single 
region server. 
 

Anyone have a work up of an equivalent for Accumulo that I can include 
under an ASLv2 license? I seem to recall madrob had something done in 
a bash script, but I can't find it anywhere. 

[1]: http://s.apache.org/CFe 

-- 
Sean 

RE: Error BAD_AUTHORIZATIONS for user root

[dlmarion]

It’s likely that you created a table, inserted data with column visibilities, 
but never gave the user the authorization tokens to see the data. Try setting 
the authorization tokens for the user with the setauths command. Since you are 
using the ‘root’ user, it would be something like: setauths –u root –s 
token1,token2,…

 

From: sateesh kumar [mailto:sk_ac...@yahoo.com] 
Sent: Sunday, September 13, 2015 10:33 AM
To: user@accumulo.apache.org
Subject: Error BAD_AUTHORIZATIONS for user root

 

Hi All, 

  I have created few tables in Accumulo as a User "root" and when i tried to 
get the data from those tables i am getting the following Error: 

[ERROR] 
java.lang.RuntimeException: 
org.apache.accumulo.core.client.AccumuloSecurityException: Error 
BAD_AUTHORIZATIONS for user root on table entities_v(ID:8) - The user does not 
have the specified authorizations 
at 
org.apache.accumulo.core.client.impl.ScannerIterator.hasNext(ScannerIterator.java:187)
 
at 
org.apache.accumulo.core.util.PeekingIterator.(PeekingIterator.java:29) 
at 
org.apache.accumulo.core.client.RowIterator.(RowIterator.java:110) 
at 
org.securegraph.accumulo.AccumuloGraph$6.createIterator(AccumuloGraph.java:906) 
at 
org.securegraph.util.LookAheadIterable.iterator(LookAheadIterable.java:10) 
at 
org.securegraph.accumulo.AccumuloGraph.getVertex(AccumuloGraph.java:811) 
at org.securegraph.GraphBase.getVertex(GraphBase.java:67) 
[/ERROR] 

When i log on to accumulo CLI and check for permissions for the user 

root@localhost> userpermissions 
System permissions: System.GRANT, System.CREATE_TABLE, System.DROP_TABLE, 
System.ALTER_TABLE, System.CREATE_USER, System.DROP_USER, System.ALTER_USER, 
System.SYSTEM, System.CREATE_NAMESPACE, System.DROP_NAMESPACE, 
System.ALTER_NAMESPACE 

Namespace permissions (accumulo): Namespace.READ, Namespace.ALTER_TABLE 

Table permissions (accumulo.metadata): Table.READ, Table.ALTER_TABLE 
Table permissions (accumulo.root): Table.READ, Table.ALTER_TABLE 
Table permissions (my_table): Table.READ, Table.WRITE, Table.BULK_IMPORT, 
Table.ALTER_TABLE, Table.GRANT, Table.DROP_TABLE 

I see the permissions for my_table. 

Not sure why i am getting an Exception when i tried to retrieve the data. 

Thanks 

RE: imbalance in number of zookeeper clients

[dlmarion]

Hey Jeff,
 Take a look at [1] and see if the zookeeper balance issue mentioned is 
applicable.
Dave
[1] https://accumulo.apache.org/release_notes/1.6.2.html




 Original message 
From: Jeff Turner  
Date: 09/10/2015  7:42 PM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: imbalance in number of zookeeper clients 

sorry if this is a faq.  i can't come up with a good google query to 
find the answer.

how bad is it that four of our five zookeepers have 600-700 clients, and one
has about 250?

i assumed that zookeeper or accumulo has some sort of natural 
rebalancing property,
so it will all work itself out.

i've been resisting a full accumulo/zk restart.
and restarting the one zookeeper to see what happens has a big 
unpleasant wake, too.

so
   - will they eventually rebalance
   - if not, how bad is it that four of them are working harder

thanks,
jeff

Re: Accumulo: "BigTable" vs. "Document Model"

[dlmarion]

Both will work, but I think the answer depends on the amount of data that you 
will be querying over and your query latency requirements. I would include the 
wikisearch[1] storage scheme into your list as well (k/v table + indices). 
Then, personally, I would rate them in the following order as database size 
increases and query latency requirements decrease: 

1. Document Model 
2. K/V Model 
3. K/V Model with indices (wikisearch) 

[1] https://accumulo.apache.org/example/wikisearch.html 

- Original Message -

From: "Michael Moss"  
To: user@accumulo.apache.org 
Sent: Friday, September 4, 2015 11:42:20 AM 
Subject: Accumulo: "BigTable" vs. "Document Model" 

Hello, everyone. 

I'd love to hear folks' input on using the "natural" data model of Accumulo 
("BigTable" style) vs more of a Document Model. I'll try to succinctly describe 
with a contrived example. 

Let's say I have one domain object I'd like to model, "SensorReadings". A 
single entry might look something like the following with 4 distinct CF, CQ 
pairs. 

RowKey: DeviceID-YYYMMDD-ReadingID (i.e. - 1-20150101-1234) 
CF: "Meta", CQ: "Timestamp", Value:  
CF: "Sensor", CQ: "Temperature", Value: 80.4 
CF: "Sensor", CQ: "Humidity", Value: 40.2 
CF: "Sensor", CQ: "Barometer", Value: 29.1 

I might do queries like "get me all SensorReadings for 2015 for DeviceID = 1" 
and if I wanted to operate on each SensorReading as a single unit (and not as 
the 4 'rows' it returns for each one), I'd either have to aggregate the 4 CF, 
CQ pairs for each RowKey client side, or use something like the 
WholeRowIterator. 

In addition, if I wanted to write a query like, "for DeviceID = 1 in 2015, 
return me SensorReadings where Temperature > 90, Humidity < 40, Barometer > 
31", I'd again have to either use the WholeRowIterator to 'see' each entire 
SensorReading in memory on the server for the compound query, or I could take 
the intersection of the results of 3 parallel, independent queries on the 
client side. 

Where I am going with this is, what are the thoughts around creating a Java, 
Protobuf, Avro (etc) object with these 4 CF, CQ pairs as fields and storing 
each SensorReading as a single 'Document'? 

RowKey: DeviceID-YYYMMDD 
CF: ReadingID Value: Protobuf(Timestamp=123, Temperature=80.4, Humidity=40.2, 
Barometer = 29.1) 

This way you avoid having to use the WholeRowIterator and unless you often have 
queries that only look at a tiny subset of your fields (let's say just 
"Temperature"), the serialization costs seem similar since Value is just bytes 
anyway. 

Appreciate folks' experience and wisdom here. Hope this makes sense, happy to 
clarify. 

Best. 

-Mike 

RE: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library

[dlmarion]

Dylan,
  I am a little confused about whether you want to place this in the contrib 
area or whether you want to create a sub-project as both are mentioned in your 
proposal. Also, if you intend for this to be a sub-project, have you looked at 
the incubator process? From what I understand given that this is a code 
contribution,it will have to go through that process.


 Original message 
From: Dylan Hutchison dhutc...@uw.edu 
Date: 08/28/2015  2:43 AM  (GMT-05:00) 
To: Accumulo Dev List d...@accumulo.apache.org 
Cc: Accumulo User List user@accumulo.apache.org 
Subject: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library 

Dear Accumulo community,
I humbly ask your consideration of Graphulo as a new contrib project to 
Accumulo.  Let's use this thread to discuss what Graphulo is, how it fits into 
the Accumulo community, where we can take it together as a new community, and 
how you can use it right now.  Please see the README at Graphulo's Github, and 
for a more in-depth look see the docs/ folder or the examples.
https://github.com/Accla/graphulo
Graphulo is a Java library for the Apache Accumulo database delivering 
server-side sparse matrix math primitives that enable higher-level graph 
algorithms and analytics.
Pitch: Organizations use Accumulo for high performance indexed and distributed 
data storage.  What do they do after their data is stored?  Many use cases 
perform analytics and algorithms on data in Accumulo, which aside from simple 
iterators uses, require scanning data out from Accumulo to a computation 
engine, only to write computation results back to Accumulo.  Graphulo enables a 
class of algorithms to run inside the Accumulo server like a stored procedure, 
especially (but not restricted to) those written in the language of graphs and 
linear algebra.  Take breadth first search as a simple use case and PageRank as 
one more complex.  As a stretch goal, imagine analysts and mathematicians 
executing PageRank and other high level algorithms on top of the Graphulo 
library on top of Accumulo at high performance.
I have developed Graphulo at the MIT Lincoln Laboratory with support from the 
NSF since last March.  I owe thanks to Jeremy Kepner, Vijay Gadepally, and Adam 
Fuchs for high level comments during design and performance testing phases.  I 
proposed a now-obsolete design document last Spring to the Accumulo community 
too which received good feedback.
The time is ripe for Graphulo to graduate my personal development into larger 
participation.  Beyond myself and beyond the Lincoln Laboratory, Graphulo is 
for the Accumulo community.  Users need a place where they can interact, 
developers need a place where they can look, comment, and debate designs and 
diffs, and both users and developers need a place where they can interact and 
see Graphulo alongside its Accumulo base.  
The following outlines a few reasons why I see contrib to Accumulo as 
Graphulo's proper place:Establishing Graphulo as an Apache (sub-)project is a 
first step toward building a community.  The spirit of Apache--its mailing list 
discussions, low barrier to interactions between users and developers new and 
old, open meritocracy and more--is a surefire way to bring Graphulo to the 
people it will help and the people who want to help it in turn.

Committing to core Accumulo doesn't seem appropriate for all of Graphulo, 
because Graphulo uses Accumulo in a specific way (server-side computation) in 
support of algorithms and applications.  Parts of Graphulo that are useful for 
all Accumulo users (not just matrix math for algorithms) could be transferred 
from Graphulo to Accumulo, such as ApplyIterator or SmallLargeRowFilter or 
DynamicIterator.  

Leaving Graphulo as an external project leaves Graphulo too decoupled from 
Accumulo.  Graphulo has potential to drive features in core Accumulo such as 
ACCUMULO-3978, ACCUMULO-3710, and ACCUMULO-3751.  By making Graphulo a contrib 
sub-project, the two can maintain a tight relationship while still maintaining 
independent versions.  
Historically, contrib projects have gone into Accumulo contrib and become 
stale.  I assure you I do not intend Graphulo this fate.  The Lincoln 
Laboratory has interests in Graphulo, and I will continue developing Graphulo 
at the very least to help transition Graphulo to greater community involvement. 
 However, since I will start a PhD program at UW next month, I cannot make 
Graphulo a full time job as I have in recent history.  I do have ideas for 
using Graphulo as part of my PhD database research.  
Thus, in the case of large community support, I can transition to a support 
role while others in the community step up.  If smaller community support, I 
can continue working on Graphulo as before at my own pace and perhaps more 
publicly.  There are only a few steps left before Graphulo could be considered 
finished software: Developing a new interface to Graphulo's core functions 
using 

Re: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library

[dlmarion]

[adding dev@ back in, I forgot to reply-all last time] 

I think the process and exit criteria would be different for a code contrbution 
vs a sub-project. [1] talks about projects and sub-projects, [2] talks about 
contributions. I don't know if the exit criteria for a sub-project is the same 
as a top level project; will you be required to show community growth, 
understanding the process, setting up the infrastructure, etc. If so, who is 
going to shepherd Graphulo through this process? I'm not an expert in this 
area. I just wanted to point out that they are likely hurdles of different 
height. 

[1] incubator.apache.org/incubation/Incubation_Policy.html 
[2] http://incubator.apache.org/ip-clearance/ 

- Original Message -

From: Dylan Hutchison dhutc...@uw.edu 
To: Accumulo User List user@accumulo.apache.org, Accumulo Dev List 
d...@accumulo.apache.org 
Sent: Friday, August 28, 2015 11:25:11 AM 
Subject: Re: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math 
library 



place this in the contrib area or create a sub-project? 



Ah ha, I indeed had the two avenues mistakenly equated in my head since both 
involve Incubator approval and the same proposal and IP template. 

I intend Graphulo as a sub-project of Accumulo. There are enough use cases 
unrelated to Accumulo's core development (algorithms, Graphulo client, 
Graphulo-specific iterators) that it makes sense to form a dedicated project 
for Graphulo. That said, Graphulo is coupled to Accumulo by design and purpose, 
and there is large opportunity for synergy in that Graphulo development may 
help Accumulo development and vice versa. We're in that happy middle spot where 
a sub-project makes sense. That said, this is a community decision, and so I'm 
open to other opinions. 

Regards, Dylan 

On Fri, Aug 28, 2015 at 8:08 AM, dlmarion  dlmar...@comcast.net  wrote: 

blockquote

Dylan, 

I am a little confused about whether you want to place this in the contrib area 
or whether you want to create a sub-project as both are mentioned in your 
proposal. Also, if you intend for this to be a sub-project, have you looked at 
the incubator process? From what I understand given that this is a code 
contribution,it will have to go through that process. 




 Original message  
From: Dylan Hutchison  dhutc...@uw.edu  
Date: 08/28/2015 2:43 AM (GMT-05:00) 
To: Accumulo Dev List  d...@accumulo.apache.org  
Cc: Accumulo User List  user@accumulo.apache.org  
Subject: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library 

Dear Accumulo community, 

I humbly ask your consideration of Graphulo as a new contrib project to 
Accumulo. Let's use this thread to discuss what Graphulo is, how it fits into 
the Accumulo community, where we can take it together as a new community, and 
how you can use it right now. Please see the README at Graphulo's Github, and 
for a more in-depth look see the docs/ folder or the examples. 


blockquote

https://github.com/Accla/graphulo 

/blockquote


blockquote

Graphulo is a Java library for the Apache Accumulo database delivering 
server-side sparse matrix math primitives that enable higher-level graph 
algorithms and analytics. 


/blockquote

Pitch: Organizations use Accumulo for high performance indexed and distributed 
data storage. What do they do after their data is stored? Many use cases 
perform analytics and algorithms on data in Accumulo, which aside from simple 
iterators uses, require scanning data out from Accumulo to a computation 
engine, only to write computation results back to Accumulo. Graphulo enables a 
class of algorithms to run inside the Accumulo server like a stored procedure, 
especially (but not restricted to) those written in the language of graphs and 
linear algebra. Take breadth first search as a simple use case and PageRank as 
one more complex. As a stretch goal, imagine analysts and mathematicians 
executing PageRank and other high level algorithms on top of the Graphulo 
library on top of Accumulo at high performance. 

I have developed Graphulo at the MIT Lincoln Laboratory with support from the 
NSF since last March. I owe thanks to Jeremy Kepner, Vijay Gadepally, and Adam 
Fuchs for high level comments during design and performance testing phases. I 
proposed a now-obsolete design document last Spring to the Accumulo community 
too which received good feedback. 

The time is ripe for Graphulo to graduate my personal development into larger 
participation. Beyond myself and beyond the Lincoln Laboratory, Graphulo is for 
the Accumulo community. Users need a place where they can interact, developers 
need a place where they can look, comment, and debate designs and diffs, and 
both users and developers need a place where they can interact and see Graphulo 
alongside its Accumulo base. 

The following outlines a few reasons why I see contrib to Accumulo as 
Graphulo's proper place: 


1. Establishing Graphulo as an Apache (sub-)project

RE: Accumulo Monitor NullPointerException

[dlmarion]

Anything in the master log?




 Original message 
From: Scott sct...@gmail.com 
Date: 08/21/2015  6:15 PM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Accumulo Monitor NullPointerException 

Hello,
  We have Accumulo 1.6.3 set up on the HortonWorks install with a basic cluster 
of one master and one tablet server. Initially everything seemed good but now 
I'm getting a NullPointerException when I go to the monitor web page. The logs 
also keep adding a message to that effect as well. I searched and couldn't find 
any posts with the same problem. I went back through all of the config files 
and made sure they conform to the configuration instructions. I'm hoping 
someone can point me to the right place to look for troubleshooting. 
http://machine-name:50095 responds with a page that has the following:
prejava.lang.NullPointerException at 
org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:252)  at 
org.apache.accumulo.monitor.servlets.BasicServlet.doGet(BasicServlet.java:59)   
 at 
org.apache.accumulo.monitor.servlets.DefaultServlet.doGet(DefaultServlet.java:101)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at 
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)   at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)   at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)   
 at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)  at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
   at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) 
   at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)   
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 
 at org.eclipse.jetty.server.Server.handle(Server.java:370)  at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
   at 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
  at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) 
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at 
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
 at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at 
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) 
 at java.lang.Thread.run(Thread.java:745)/pre

The logs are recording the following error:
2015-08-21 16:37:21,152 [monitor.Monitor] WARN :java.lang.NullPointerException  
      at org.apache.accumulo.monitor.Monitor.fetchData(Monitor.java:252)        
at org.apache.accumulo.monitor.Monitor$2.run(Monitor.java:508)        at 
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)      
  at java.lang.Thread.run(Thread.java:745)
Thanks,
 Scott

Re: thrift versions in accumulo

[dlmarion]

+1 to Netty. I have used it and like it, but I think you will have to write all 
of the code yourself. Nifty (Facebook) could be an option to use Thrift and 
Netty together. Finagle (Twitter) would be another alternative. 

- Original Message -

From: Christopher ctubb...@apache.org 
To: Accumulo User List user@accumulo.apache.org 
Sent: Wednesday, August 19, 2015 3:50:45 PM 
Subject: Re: thrift versions in accumulo 

Well, Thrift serves two purposes for us. It manages the RPC API and 
handles the data serialization for the RPC. 

Long term, I'd like to look at Netty for handling the API (because it 
seems stable, widely used, and feature-rich). I'm not particularly 
concerned about the serialization, as long as the library is stable. 
Avro makes as much sense to me as any other, but I haven't actually 
used it yet, so I'll reserve judgment. 

Even if we move away from Thrift in the long-term for the 
client/server and server/server RPCs, we'll probably not get rid of it 
entirely. It's still pretty useful for the accumulo-proxy. However, 
since that's a small, optional add-on, it'd be pretty easy to isolate 
thrift to just be a dependency of that... and presumably it'd be easy 
to rebuild that add-on component for different thrift versions or to 
support multiple versions. 

-- 
Christopher L Tubbs II 
http://gravatar.com/ctubbsii 


On Wed, Aug 19, 2015 at 3:41 PM, Max Thomas max.tho...@jhu.edu wrote: 
 I'm not opposed to the idea... 
 
 What about the long term future (e.g., Christopher's comment) for Thrift? 
 Are there particularly attractive alternatives (Avro) that you have in mind 
 for the project? Asking both as a interested user and general technologist. 
 
 
 On 8/19/15 3:20 PM, Christopher wrote: 
 
 +1 to that. 
 
 -- 
 Christopher L Tubbs II 
 http://gravatar.com/ctubbsii 
 
 
 On Wed, Aug 19, 2015 at 3:14 PM, Josh Elser josh.el...@gmail.com wrote: 
 
 I guess it's also worth being explicit: if you have the cycles to work on 
 this, we can help give direction on trying to switch 1.8.0 to 0.9.2, Max. 
 
 Necessary work doesn't imply that it has to be a committer. Likely 
 vetted 
 by one of us in the end, but you can definitely help push this along if 
 you 
 have the time/cycles. 
 
 
 Christopher wrote: 
 
 
 ACCUMULO-756 has been put on the back-burner for now (at least from 
 me). I'd love to revisit it, but unfortunately, I've had other 
 priorities. 
 
 As for 0.9.2, I've had a lot of concerns over our continued use of 
 Thrift. It has had serious regressions, and behavior changes in bug 
 fixes that have not been expected. I'd be in favor of switching to 
 0.9.2 for Accumulo 1.8.0, if we can be reasonably assured it's not 
 going to wreak havoc on us like 0.9.0-0.9.1 did. That means time and 
 testing. I think we can go ahead and make a ticket for it (if it's not 
 done already), and if somebody has time to do it for 1.8.0, great. If 
 it turns out to be more trouble than it's worth, or if it causes 
 unnecessary problems, we can postpone or roll back. 
 
 For 1.7.x, we'll likely stay with 0.9.1, unless our experience working 
 with 0.9.2 on 1.8.0 leads us to believe that it'd be a safe 
 transition. It'd have to be *very* low risk, I imagine... but we'd 
 probably need to discuss further once we have more information. 
 
 -- 
 Christopher L Tubbs II 
 http://gravatar.com/ctubbsii 
 
 
 On Wed, Aug 19, 2015 at 1:56 PM, Josh Elserjosh.el...@gmail.com 
 wrote: 
 
 
 We talked about updating 1.8.0 to 0.9.2. 1.7 would very likely not be 
 updated to 0.9.2 due to worry of stability from Thrift (this has been a 
 real 
 problem in the past). 
 
 Thrift itself needs consistent libraries on both sides of the wire. An 
 0.9.1 
 client doesn't work with an 0.9.2 server. Thrift's compatibility gives 
 us 
 the ability to run Accumulo 1.6 clients against Accumulo 1.7 servers 
 (hypothetically). 
 
 Hopefully, you could bump the version, recompile the Thrift code in 1.7 
 and 
 it would work if you want to experiment. YMMV 
 
 
 Max Thomas wrote: 
 
 
 The latest major thrift release is 0.9.2; Accumulo 1.7 (and the 
 current 
 master branch on Github) depends on 0.9.1. 0.9.2 fixes a number of 
 issues that are desirable to have, such as functions in Python for 
 hash 
 codes. 
 
 An initial test with a Java project depending on thrift 0.9.2 and 
 accumulo results in an exception when the Accumulo server starts, 
 presumably due to conflicting classes. 
 
 I have a few questions: 
 
 1) Are there creative Maven things to get Accumulo and thrift 0.9.2 to 
 place nicely together without building Accumulo from scratch? 
 
 2) Is there a plan in place to upgrade the thrift dependency for 1.8.0 
 or the 1.7.x line? (Perhaps short term; see below) 
 
 3) I see this issue on JIRA: 
 
 
 
 https://issues.apache.org/jira/browse/ACCUMULO-756?jql=project%20%3D%20ACCUMULO%20AND%20text%20~%20thrift
  
 
 
 Is work still active on this issue? 

RE: Accumulo GC and Hadoop trash settings

[dlmarion]

  All of the components that you need to perform point in time recovery of an 
Accumulo instance exist already. I have been working on a tool[1] in my copious 
amounts of free time to integrate them into something usable, but it doesn’t 
actually use the files in the trash. My approach is to let you determine the 
MTTR and then schedule your backups accordingly; a backup in case you are not 
able to recover your database using the techniques in the current 
documentation.  

 

[1] https://github.com/dlmarion/raccovery

 

From: James Hughes [mailto:jn...@virginia.edu] 
Sent: Monday, August 17, 2015 4:28 PM
To: user@accumulo.apache.org
Subject: Re: Accumulo GC and Hadoop trash settings

 

Ok, I can the see the benefit of being able to recovery data.  Is this process 
documented?  And is there any kind of user-friendly tool for it?

 

On Mon, Aug 17, 2015 at 4:11 PM, dlmar...@comcast.net wrote:

 

 It's not temporary files, it's any file that has been compacted away. If you 
keep files around longer than {dfs.namenode.checkpoint.period}, then you have a 
chance to recover in case your most recent checkpoint is corrupt.

 

  _  

From: James Hughes jn...@virginia.edu
To: user@accumulo.apache.org
Sent: Monday, August 17, 2015 3:57:57 PM
Subject: Accumulo GC and Hadoop trash settings

 

 

Hi all,

 

From reading about the Accumulo GC, it sounds like temporary files are 
routinely deleted during GC cycles.  In a small testing environment, I've the 
HDFS Accumulo user's .Trash folder have 10s of gigabytes of data.


Is there any reason that the default value for gc.trash.ignore is false?  Is 
there any downside to deleting GC'ed files completely?

 

Thanks in advance,

 

Jim

 

http://accumulo.apache.org/1.6/accumulo_user_manual.html#_gc_trash_ignore

 

 

Re: Accumulo GC and Hadoop trash settings

[dlmarion]

It's not temporary files, it's any file that has been compacted away. If you 
keep files around longer than { dfs.namenode.checkpoint.period}, then you have 
a chance to recover in case your most recent checkpoint is corrupt. 

- Original Message -

From: James Hughes jn...@virginia.edu 
To: user@accumulo.apache.org 
Sent: Monday, August 17, 2015 3:57:57 PM 
Subject: Accumulo GC and Hadoop trash settings 

Hi all, 

From reading about the Accumulo GC, it sounds like temporary files are 
routinely deleted during GC cycles. In a small testing environment, I've the 
HDFS Accumulo user's .Trash folder have 10s of gigabytes of data. 

Is there any reason that the default value for gc.trash.ignore is false? Is 
there any downside to deleting GC'ed files completely? 

Thanks in advance, 

Jim 

http://accumulo.apache.org/1.6/accumulo_user_manual.html#_gc_trash_ignore 

RE: How to control Minor Compaction by programming

[dlmarion]

It sounds like you want to try and not minor compact during your ingest of your 
data. Is that correct?

 

From: William Slacum [mailto:wsla...@gmail.com] 
Sent: Thursday, July 30, 2015 8:10 PM
To: user@accumulo.apache.org
Subject: Re: How to control Minor Compaction by programming

 

See 
http://accumulo.apache.org/1.5/apidocs/org/apache/accumulo/core/client/admin/TableOperations.html#flush%28java.lang.String,%20org.apache.hadoop.io.Text,%20org.apache.hadoop.io.Text,%20boolean%29
 for minor compacting (aka flushing) a table via the API.

 

On Thu, Jul 30, 2015 at 5:52 PM, Hai Pham htp0...@tigermail.auburn.edu wrote:

Hi, 

 

Please share with me is there any way that we can init / control the Minor 
Compaction by programming (not from the shell). My situation is when I ingest a 
large data using the BatchWriter, the minor compaction is triggered 
uncontrollably. The flush() command in BatchWriter seems not for this purpose. 

I also tried to play around with parameters in documentation but seems not much 
helpful. 

 

Also, can you please explain the number 0, 1.0, 2.0, ... in charts (web 
monitoring) denoting the level of Minor Compaction and Major Compaction? 

 

Thank you! 

Hai Pham 

 

 

 

 

Re: Entry-based TableBalancer

[dlmarion]

Hotspotting was the first thing that came to my mind with the proposed 
balancer. The fservers don't keep all the K/V in memory. You are balancing 
query and live ingest across your resources.




 Original message 
From: Eric Newton eric.new...@gmail.com 
Date: 07/29/2015  8:46 PM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Re: Entry-based TableBalancer 

To my knowledge, nobody has written such a balancer.
In the history of the project, we started writing advanced, complicated 
balancers that moved tablets around much too quickly, which degraded 
performance. After that, we wrote much simpler balancers to avoid the chaos. 
We're moving back to more complex balancers, but mostly just to ensure that we 
aren't hotspoting, based on known ingest patterns (date related, for example).
If you write a new balancer, make it slow to move tablets, and very simple.  
Avoid over-optimizing tablet placement.
-Eric
On Wed, Jul 29, 2015 at 8:20 PM, Konstantin Pelykh kpel...@gmail.com wrote:
Hi, 

I'm looking for a tablet balancer which operates based on a number of entries 
per tablet as opposed to a number of tablets per tablet server. My goal is to 
get even distribution of entries across the cluster. 

As an example: 

tablet #1  15M entries
tablet #2   5M entries
tablet #3   8M entries

After balancing tablets I would want to get:

Server 1 hosts: tablet1 
Server 2 hosts: tablet2, tablet3

The idea is pretty simple and I believe such balancer has already been 
developed, so I decided to check before reinventing the wheel. 

Thanks!
Konstantin


Big Data / Lucene and Solr Consultant
LinkedIn: linkedin.com/in/kpelykh
Website: www.kpelykh.com

RE: TSDB on Accumulo row key

[dlmarion]

https://github.com/ericnewton/accumulo-opentsdb

 

 

From: Ranjan Sen [mailto:ranjan_...@hotmail.com] 
Sent: Monday, July 20, 2015 6:25 PM
To: user@accumulo.apache.org
Subject: TSDB on Accumulo row key

 

Hi All, 

Is there something like TSDB (Time series database) on Accumulo?

 

Thanks 
Ranjan

Re: Custom Iterator output

[dlmarion]

via the getTopKey() and getTopValue() methods. [1] should be a simple example. 

[1] 
https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=core/src/main/java/org/apache/accumulo/core/iterators/user/GrepIterator.java;h=043a729a778fc34d2ee87a0227056ffac81b7fe7;hb=refs/heads/master
 

- Original Message -

From: shweta.agrawal shweta.agra...@orkash.com 
To: user@accumulo.apache.org 
Sent: Friday, April 17, 2015 8:50:26 AM 
Subject: Custom Iterator output 

Hi, 

I am working on custom iterator. I want to know, how do i get the output 
from the custom iterators? 

Thanks and Regards 
Shweta 

RE: Installing custom authorizor

[dlmarion]

Did you restart Accumulo after you copied the jar? The general.classpaths 
locations are not monitored and loaded dynamically.



div Original message /divdivFrom: Srikanth Viswanathan 
srikant...@gmail.com /divdivDate:02/18/2015  8:42 PM  (GMT-05:00) 
/divdivTo: user@accumulo.apache.org /divdivCc:  /divdivSubject: 
Installing custom authorizor /divdiv
/divWhat are the steps involved in installing a custom authorizor for use
by accumulo? I copied the jar containing my custom authorizor class
into the path contained in the general.classpaths property, but
accumulo isn't able to find it:

[conf.AccumuloConfiguration] WARN : Failed to load class
java.lang.ClassNotFoundException:
com.myenterprise.accumulo.authorizor.MyAuthorizor
at 
org.apache.commons.vfs2.impl.VFSClassLoader.findClass(VFSClassLoader.java:175)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at 
org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.loadClass(AccumuloVFSClassLoader.java:112)
at 
org.apache.accumulo.core.conf.AccumuloConfiguration.instantiateClassProperty(AccumuloConfiguration.java:239)
at 
org.apache.accumulo.server.security.SecurityOperation.getAuthorizor(SecurityOperation.java:88)
at 
org.apache.accumulo.server.security.AuditedSecurityOperation.getInstance(AuditedSecurityOperation.java:74)
at 
org.apache.accumulo.server.security.AuditedSecurityOperation.getInstance(AuditedSecurityOperation.java:69)
at org.apache.accumulo.tserver.TabletServer.config(TabletServer.java:3476)
at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:3671)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)

Am I missing a step? Thank you.

RE: Moving HDFS name node to another host in accumulo 1.6

[dlmarion]

What Accumulo version?

What Hadoop version?

 

From: Calvin Feder [mailto:calvin.fe...@argyledata.com] 
Sent: Friday, January 16, 2015 7:29 PM
To: user@accumulo.apache.org
Subject: Moving HDFS name node to another host in accumulo 1.6

 

We need to move the HDFS name node of our cluster to a new host (the old
host is experiencing hardware issues and may die at any time).  We need to
keep the current accumulo data.  We are trying to do it in a test
environment and running into numerous issues. We stopped the cluster, moved
the name node to a new host, and restarted the cluster.  We changed the
entry in zookeeper for the accumulo root tablet to the new location, and got
accumulo to start.  We see all of the tables we expected to see in accumulo,
and they show up in the monitor as online, but the monitor shows that no
tablets or entries exist in those tables, and obviously we cannot scan them.

 

This is an example of what we are seeing:

 



 

Has anyone tried to do something similar?  What are we missing?

 

Thank you,

___

Calvin Feder | QA Lead | Argyle Data, Inc. 

cid:A79ABAC4-829A-4B95-AFE2-C09282CD7AE7

 

 





  _  

This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise confidential information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited. Thank you in
advance for your cooperation. 

  _  

RE: Moving HDFS name node to another host in accumulo 1.6

[dlmarion]

In 1.6, accumulo.metadata entries are absolute URLs. You can change the 
existing URLs using the instance.volume.replacments[1] parameter.

[1] 
http://accumulo.apache.org/1.6/accumulo_user_manual.html#_instance_volumes_replacements


 -Original Message-
 From: webmas...@webmaster.ms [mailto:webmas...@webmaster.ms] On
 Behalf Of Denis
 Sent: Friday, January 16, 2015 7:53 PM
 To: user@accumulo.apache.org
 Subject: Re: Moving HDFS name node to another host in accumulo 1.6
 
 Hello
 
 Table accumulo.metadata contains absolute urls in a form of hdfs://old-
 name-node:8020/
 I do not know how to work around it, except of patching accumulo code
 adding .replace(OLD_NODE, NEW_NODE) into relevant places
 
 On 1/17/15, Calvin Feder calvin.fe...@argyledata.com wrote:
  We need to move the HDFS name node of our cluster to a new host (the
  old host is experiencing hardware issues and may die at any time).  We
  need to keep the current accumulo data.  We are trying to do it in a
  test environment and running into numerous issues. We stopped the
  cluster, moved the name node to a new host, and restarted the cluster.
  We changed the entry in zookeeper for the accumulo root tablet to the
  new location, and got accumulo to start.  We see all of the tables we
  expected to see in accumulo, and they show up in the monitor as
  online, but the monitor shows that no tablets or entries exist in
  those tables, and obviously we cannot scan them.
 
  This is an example of what we are seeing:
 
  [cid:image002.png@01D031A9.978AE280]
 
  Has anyone tried to do something similar?  What are we missing?
 
  Thank you,
  ___
  Calvin Feder | QA Lead | Argyle Data, Inc.
  [cid:A79ABAC4-829A-4B95-AFE2-C09282CD7AE7]
 
 
 
 
 
  
 
  This message is for the designated recipient only and may contain
  privileged, proprietary, or otherwise confidential information. If you
  have received it in error, please notify the sender immediately and
  delete the original. Any other use of the e-mail by you is prohibited.
  Thank you in advance for your cooperation.
 
  
 

Re: (U) I do I tell Accumulo where the jars for my custom formatters and balancers are for a specific table?

[dlmarion]

We fixed some issues recently with the classloader in 1.6.1, make sure you are 
using an up-to-date snapshot version. The classloader was changed in version 
1.5 and now supports hosting jars from various locations, including HDFS. 
Additionally, it supports creating a table level classloader that has its own 
specific classpath settings. See 
https://blogs.apache.org/accumulo/entry/the_accumulo_classloader for more 
information. There are a couple of known issues that will hopefully get 
resolved with the next release of Apache Commons VFS, but you should be good to 
go for testing. 

- Original Message -

From: Jeff Kubina jeff.kub...@gmail.com 
To: user@accumulo.apache.org 
Sent: Monday, August 25, 2014 3:30:56 PM 
Subject: Re: (U) I do I tell Accumulo where the jars for my custom formatters 
and balancers are for a specific table? 

Versions 1.4, 1.5, and 1.6.1. A solution for just 1.6.1 would help. 

As only an Accumulo user with create/read/write/ access of tables (not an 
admin) I won't have write access to the classpath directories in 
accumulo-site.xml. Do I need to have the admins add a user directory for my 
jars to accumulo-site.xml? 


-- 
Jeff Kubina 
410-988-4436 



On Mon, Aug 25, 2014 at 3:20 PM,  dlmar...@comcast.net  wrote: 



Jeff, 

Which version of Accumulo are you using? Most of the classpath settings are in 
the accumulo-site.xml file. 



From: Jeff Kubina  jeff.kub...@gmail.com  
To: user@accumulo.apache.org 
Sent: Monday, August 25, 2014 3:17:17 PM 
Subject: Re: (U) I do I tell Accumulo where the jars for my custom formatters 
and balancers are for a specific table? 


Sorry, that was suppose to read How do I tell Accumulo where the jars 
containing my custom formatters and balancers are for a specific table? 

-- 
Jeff Kubina 
410-988-4436 



On Mon, Aug 25, 2014 at 2:03 PM, Jeff Kubina  jeff.kub...@gmail.com  wrote: 

blockquote

(U) I do I tell Accumulo where the jars for my custom formatters and balancers 
are for a specific table? 







/blockquote

Re: accumulo 1.6 and HDFS non-HA conversion to HDFS HA

[dlmarion]

I believe the problem that you are running into is that because dfs.default.uri 
was not specified, then fs.defaultFS was used to write entries to the 
accumulo.root and accumulo.metadata tables. Suggest doing the following: 

Update to the latest version of Accumulo 1.6.1-SNAPSHOT 
set instance.dfs.uri to: hdfs://mycluster 
set instance.volumes to: hdfs://mycluster 
set instance.volumes.replacements to: hdfs://somehostname:9000/accumulo 
hdfs://mycluster/accumulo 

- Original Message -

From: craig w codecr...@gmail.com 
To: user@accumulo.apache.org 
Sent: Friday, August 8, 2014 9:13:02 AM 
Subject: Re: accumulo 1.6 and HDFS non-HA conversion to HDFS HA 

In $HADOOP_CONF_DIR/core-site.xml I used to have: 

property 
name fs.default.name /name 
valuehdfs://somehostname:9000/value 
/property 

When going to HDFS HA, I removed that property and have: 
property 
namefs.defaultFS/name 
valuehdfs://mycluster/value 
/property 

In the accumulo-site.xml I don't have instance.volumes or instance.dfs.uri. 


On Tue, Aug 5, 2014 at 1:41 PM, Keith Turner  ke...@deenlo.com  wrote: 



Did your HDFS URI change and are errors you are seeing connecting to the old 
HDFS URI? If so, you may need to configure instance.volumes.replacements to 
replace the old URI in Accumulo metadata. 


On Tue, Aug 5, 2014 at 1:06 PM, craig w  codecr...@gmail.com  wrote: 

blockquote

I've setup an Accumulo 1.6 cluster with Hadoop 2.4.0 (with a secondary 
namenode). I wanted to convert the secondary namenode to be a standby (hence 
HDFS HA). 

After getting HDFS HA up and making sure the hadoop configuration files were 
accessible by Accumulo, I started up Accumulo. I noticed some reports of tablet 
servers failing to connect, however, they were failing to connect to HDFS over 
port 9000. That port is not configured/used with HDFS HA so I'm unsure why they 
are still trying to talk to HDFS using the old configuration. 

Any thoughts ideas? I know Accumulo 1.6 works with HDFS HA, but I'm curious if 
the tests have ever been run against a non-HA cluster that was converted to HA 
(with data in it). 

-- 
https://github.com/mindscratch 
https://www.google.com/+CraigWickesser 
https://twitter.com/mind_scratch 
https://twitter.com/craig_links 





/blockquote




-- 
https://github.com/mindscratch 
https://www.google.com/+CraigWickesser 
https://twitter.com/mind_scratch 
https://twitter.com/craig_links 

Re: Unable to load Iterator with setscaniter and setshelliter

[dlmarion]

general.vfs.classpaths is not set?
I saw in an earlier email that you are setting the context classloader. There 
are some places where the context manager was not being used when it should. 
This could be one of those cases. I would submit a jira ticket with all the 
information and I will try to look at it in the next day or so. Be sure to 
include classpath settings, list of files in HDFS for those locations, and the 
commands that are failing.


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

div Original message /divdivFrom: David Medinets 
david.medin...@gmail.com /divdivDate:06/15/2014  10:53 AM  (GMT-05:00) 
/divdivTo: accumulo-user user@accumulo.apache.org /divdivSubject: Re: 
Unable to load Iterator with setscaniter and setshelliter /divdiv
/divThe classpath settings in accumulo-site.xml are the following (which I 
think is the default):

property
  namegeneral.classpaths/name
  value
$ACCUMULO_HOME/server/target/classes/,
$ACCUMULO_HOME/core/target/classes/,
$ACCUMULO_HOME/start/target/classes/,
$ACCUMULO_HOME/examples/target/classes/,
$ACCUMULO_HOME/lib/[^.].$ACCUMULO_VERSION.jar,
$ACCUMULO_HOME/lib/[^.].*.jar,
$ZOOKEEPER_HOME/zookeeper[^.].*.jar,
$HADOOP_HOME/conf,
$HADOOP_HOME/[^.].*.jar,
$HADOOP_HOME/lib/[^.].*.jar,
  /value
  descriptionClasspaths that accumulo checks for updates and class files.
  When using the Security Manager, please remove the .../target/classes/ 
values.
  /description
/property


On Sun, Jun 15, 2014 at 10:49 AM, dlmarion dlmar...@comcast.net wrote:
What does your classpath settings look like in accumulo-site.xml. I recently 
made some fixes in 1.6.1-Snapshot where the context classloader was not being 
used in all cases. I dont think this case was affected though.


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone


 Original message 
From: Josh Elser
Date:06/15/2014 10:31 AM (GMT-05:00)
To: user@accumulo.apache.org
Subject: Re: Unable to load Iterator with setscaniter and setshelliter

Naw, the commons-vfs loader should be loading those resources using a second 
classloader.

Maybe it's just a problem with the HDFS code? Does it work if you have the jar 
with your iterator in lib/ or lib/ext? Or, maybe something is wrong like you 
defined a private constructor which is throwing that Exception?

On Jun 15, 2014 8:50 AM, William Slacum wilhelm.von.cl...@accumulo.net 
wrote:
Wouldn't the iterator have to be on the classpath for the JVM that launches the 
shell command?


On Sun, Jun 15, 2014 at 9:02 AM, Vicky Kak vicky@gmail.com wrote:

setiter -n MyIterator -p 10 -scan -minc -majc -class 
com.codebits.d4m.iterator.MyIterator
scan

The above line fails for me with the similar kind of error i.e 
ClassNotFoundException

root@accumulo atest setiter -n MyIterator -p 10 -scan -minc -majc -class 
org.dallaybatta.MyIterator
2014-06-15 18:20:18,061 [shell.Shell] ERROR: 
org.apache.accumulo.shell.ShellCommandException: Command could not be 
initialized (Unable to load org.dallaybatta.MyIterator; class not found.)


My hdfs contains the corresponding jars but it yet fails.
After digging a code for a while I figured that the error is coming from 
org.apache.accumulo.shell.commands.SetIterCommand::execute

*
try {
  clazz = 
classloader.loadClass(className).asSubclass(SortedKeyValueIterator.class);
  untypedInstance = clazz.newInstance();
} catch (ClassNotFoundException e) {
  StringBuilder msg = new StringBuilder(Unable to load 
).append(className);
  if (className.indexOf('.')  0) {
msg.append(; did you use a fully qualified package name?);
  } else {
msg.append(; class not found.);
  }
  throw new ShellCommandException(ErrorCode.INITIALIZATION_FAILURE, 
msg.toString());
} catch (InstantiationException e) {

*

Typically the ClassNotFoundException can appear also if the dependent classes 
are not present, here SortedKeyValueIterator could be the use case, I moved the 
accumulo core to the classpath folder in the hdfs but still could not get it 
working. May be some other dependent classes are required, needs more time to 
analyse in this direction.

The document states the following so it should ideally work,
although the VFS classloader allows for classpath manipulation using a variety 
of schemes including URLs and HDFS URIs.

I find it strange that your test and my tests results differ as you are able to 
set Iterator for a single row but I am not.

Thanks,
Vicky




On Sun, Jun 15, 2014 at 8:25 AM, David Medinets david.medin...@gmail.com 
wrote:
I'm sure that I'm overlooking something simple. I can load my iterator using 
setiter but not with setscaniter or setshelliter.

Here is my do-nothing iterator:

public class MyIterator extends WrappingIterator

RE: Losing tservers - Unusually high Last Contact times

[dlmarion]

You are hitting the zookeeper timeout, default 30s I believe. You said you
are not oversubscribed for memory, but what about CPU? Are you running YARN
processes on the same nodes as the tablet servers? Is the tablet server
being pushed into swap or starved of CPU?

-Original Message-
From: thomasa [mailto:tho...@ccri.com] 
Sent: Monday, May 19, 2014 4:22 PM
To: user@accumulo.apache.org
Subject: Losing tservers - Unusually high Last Contact times

Hello all,

I am having issues with tablet servers going down due to poor contact times
(my hypothesis at least). In the past I have had stability success with
smaller clouds (20-40 nodes), but have run into issues with a larger number
of nodes (150+). Each node is a datanode, nodemanger, and tablet server.
There is a master node that is running the hadoop namenode, hadoop resource
manager and accumulo master, monitor, etc. There are three zookeeper nodes.
All nodes are vms. This same setup is used on the smaller, stable clouds as
well. 

I do not believe memory allocation is an issue as I have only given
hadoop/yarn (2.2.0) and accumulo (1.5.1) less than half of the available
memory. The FATAL errors I have seen are:

Lost tablet server lock (resaon = SESSION_EXPIRED), exiting

Lost ability to monitor tablet server lock, exiting

Other than bumping up rpc timeout (which I have done but would rather not do
that and find the root cause of the problem), I have run out of ideas on how
to solve this issue. 

Does anyone have any insight into why I would be seeing such bad response
times between nodes? Are there any configuration parameters I can play with
to fix this?

I realize this is a very general question, so let me know if there is any
information I can provide to help clarify the issue.

Thank you in advance for your time.

Thomas



--
View this message in context:
http://apache-accumulo.1065345.n5.nabble.com/Losing-tservers-Unusually-high-
Last-Contact-times-tp9950.html
Sent from the Users mailing list archive at Nabble.com.

RE: Advice on increasing ingest rate

[dlmarion]

 

How many threads are you using in the AccumuloOutputFormat? What is your
latency set to?

 

From: Adam Fuchs [mailto:afu...@apache.org] 
Sent: Tuesday, April 08, 2014 5:36 PM
To: user@accumulo.apache.org
Subject: Re: Advice on increasing ingest rate

 

MIke,

 

What version of Accumulo are you using, how many tablets do you have, and
how many threads are you using for minor and major compaction pools? Also,
how big are the keys and values that you are using?

 

Here are a few settings that may help you:

1. WAL replication factor (tserver.wal.replication). This defaults to 3
replicas (the HDFS default), but if you set it to 2 it will give you a
performance boost without a huge hit to reliability.

2. Ingest buffer size (tserver.memory.maps.max), also known as the in-memory
map size. Increasing this generally improves the efficiency of minor
compactions and reduces the number of major compactions that will be
required down the line. 4-8 GB is not unreasonable.

3. Make sure your WAL settings are such that the size of a log
(tserver.walog.max.size) multiplied by the number of active logs
(table.compaction.minor.logs.threshold) is greater than the in-memory map
size. You probably want to accomplish this by bumping up the number of
active logs.

4. Increase the buffer size on the BatchWriter that the clients use. This
can be done with the setBatchWriterOptions method on the
AccumuloOutputFormat.

 

Cheers,

Adam

 

 

On Tue, Apr 8, 2014 at 4:47 PM, Mike Hugo m...@piragua.com wrote:

Hello,

 

We have an ingest process that operates via Map Reduce, processing a large
set of XML files and  inserting mutations based on that data into a set of
tables.

 

On a 5 node cluster (each node has 64G ram, 20 cores, and ~600GB SSD) I get
400k inserts per second with 20 mapper tasks running concurrently.
Increasing the number of concurrent mapper tasks to 40 doesn't have any
effect (besides causing a little more backup in compactions).

 

I've increased the table.compaction.major.ratio and increased the number of
concurrent allowed compactions for both min and max compaction but each of
those only had negligible impact on ingest rates.

 

Any advice on other settings I can tweak to get things to move more quickly?
Or is 400k/second a reasonable ingest rate?  Are we at a point where we
should consider generating r files like the bulk ingest example?

 

Thanks in advance for any advice.

 

Mike

 

RE: Hadoop HA with Accumulo 1.5

[dlmarion]

We had an issue in our testing
(https://issues.apache.org/jira/browse/ACCUMULO-2480). The root cause was a
misconfiguration for automatic failover. The sshfence feature does not
handle network failures, so you have to configure it with the
shell(/bin/true) command also (separated by a newline, unlike other Hadoop
configuration property values). However, if you end up with a hiccup in the
failover for some reason, you could run into ACCUMULO-2480. I had to restart
the entire Accumulo database because different tservers started reporting
the error. A restart of Accumulo did work and recovered with no issues.

-Original Message-
From: Josh Elser [mailto:josh.el...@gmail.com] 
Sent: Thursday, March 20, 2014 7:06 PM
To: user@accumulo.apache.org
Subject: Re: Hadoop HA with Accumulo 1.5

Yes, removing the biggest SPOF from the entire Accumulo architecture is a
good thing :)

The usage of it with Accumulo is, given all of my testing, completely
transparent. Once you configure HDFS correctly, there should be nothing
additional you have to do with Accumulo except make sure instance.dfs.uri in
accumulo.site.xml is up to date.

On 3/20/14, 6:17 PM, Ott, Charlie H. wrote:
 So I was looking into the software configuration for HA in regard to 
 hdfs clients utilizing the class 

org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
 (http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/l
 atest/CDH4-High-Availability-Guide/cdh4hag_topic_2_3.html)

 So I was wondering, does Accumulo 1.5 benefit from the HA feature of 
 having a failover namenode?

RE: pre-split table tablet server assignment

[dlmarion]

The master runs a load balancer.


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

 Original message 
From: Kepner, Jeremy - 0553 - MITLL kep...@ll.mit.edu 
Date:03/02/2014  8:34 PM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: pre-split table tablet server assignment 

If I pre-split a table, how are those splits assigned to the tablet server?
Is their an algorithm based on the split key?

RE: best practice for deploying server side iterator jars

[dlmarion]

You will also want to set the replication on the directory/jars in HDFS to
something high. I think in my testing I could not set it above 50.

-Original Message-
From: Josh Elser [mailto:josh.el...@gmail.com] 
Sent: Thursday, February 20, 2014 12:30 PM
To: user@accumulo.apache.org
Subject: Re: best practice for deploying server side iterator jars

A word of warning... https://issues.apache.org/jira/browse/VFS-487

Be careful about redeploying jars in rapid succession as it's possible
that you'll get hit with the above issue. The classloader is currently
configured to monitor the filesystem every second -- it might be prudent to
make sure you wait at least 5s to make sure each file monitor sees the
update.

On 2/20/14, 5:53 AM, Brian Loss wrote:
 Starting with Accumulo 1.5 there is a VFS class loader that is capable of
loading jars from HDFS.  If you have a look at the accumulo-site.xml in
conf/examples/vfs-classloader, there is example configuration in the VFS
ClassLoader Settings section.

 If you're on 1.4 or earlier, I don't believe there is any alternative to
just making sure the jars are distributed to all tservers.

 On Feb 20, 2014, at 7:54 AM, Mike Hugo m...@piragua.com wrote:

 Hello,

 I'm wondering if there's an easy way to make sure a jar we have with
custom server side iterators gets properly placed into the tserver classpath
on each tserver (in ACCUMULO_HOME/lib/ext).  We are about to integrate that
into our devops scripts so that the jar is put in the right place as part of
server install and configuration, but i was wondering - is there a way to
put the jar into HDFS or something so that it's more automagically picked
up by all the tservers?

 Thanks,

 Mike

RE: Found two locations for the same extent

[dlmarion]

Have you tried killing one or both of the tservers?


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

 Original message 
From: Anthony F afc...@gmail.com 
Date:01/25/2014  10:12 AM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Found two locations for the same extent 

I have a table that appears to be in an inconsistent state.  The master keeps 
logging the following exception:

java.lang.RuntimeException: 
org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
 found two locations for the same extent

and never balances/migrates or loads any tablets.  I've tried deleting one of 
the offending rows in the !METADATA table but, not surprisingly, root doesn't 
have proper permissions.

Anyone know how Accumulo 1.5.0 can get in this state and what can be done to 
get out of it?

Re: Found two locations for the same extent

[dlmarion]

I dont recall ever seeing/hearing of this situation. Maybe someone else on the 
list has. Have you had any recent failures?


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

 Original message 
From: Anthony F afc...@gmail.com 
Date:01/25/2014  11:03 AM  (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Re: Found two locations for the same extent 

Yes, I tried killing one and waiting for the system to stabilize - nothing 
changed.  Tried killing both, same deal.  I even tried bouncing the whole 
system.  I found a previous message about how to grant write permissions on 
!METADATA to root and was able to delete one of the offending entries.  At that 
point, the system stabilized.  Any tips on avoiding getting into this situation?


On Sat, Jan 25, 2014 at 10:51 AM, dlmarion dlmar...@comcast.net wrote:
Have you tried killing one or both of the tservers?


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone


 Original message 
From: Anthony F 
Date:01/25/2014 10:12 AM (GMT-05:00) 
To: user@accumulo.apache.org 
Subject: Found two locations for the same extent 

I have a table that appears to be in an inconsistent state.  The master keeps 
logging the following exception:

java.lang.RuntimeException: 
org.apache.accumulo.server.master.state.TabletLocationState$BadLocationStateException:
 found two locations for the same extent

and never balances/migrates or loads any tablets.  I've tried deleting one of 
the offending rows in the !METADATA table but, not surprisingly, root doesn't 
have proper permissions.

Anyone know how Accumulo 1.5.0 can get in this state and what can be done to 
get out of it?

RE: utility to print Accumulo config (feedback appreciated)

[dlmarion]

There is a dumpconfig command in 1.6.0. The source should help you:
https://github.com/apache/accumulo/blob/1.6.0-SNAPSHOT/server/base/src/main/
java/org/apache/accumulo/server/util/Admin.java. Look at the printConfig
method.

 

From: Sujee Maniyam [mailto:su...@sujee.net] 
Sent: Tuesday, December 17, 2013 7:36 PM
To: user@accumulo.apache.org
Subject: utility to print Accumulo config (feedback appreciated)

 

Hi all

 

Just wrote a tiny snippet to dump Accumulo config

 

file :
https://github.com/sujee/print-hadoop-config/blob/master/src/main/java/confi
g/AccumuloConfig.java

 

project : https://github.com/sujee/print-hadoop-config

 

 

I am using 'ServerConfiguration.getSiteConfiguration();'

is this the right way to get 'current' server config  (Default  + Site) ?

 

thanks!

Sujee Maniyam (http://sujee.net)

RE: [VOTE] Deprecate mock in 1.6.0

[dlmarion]

+1

 

From: Keith Turner [mailto:ke...@deenlo.com] 
Sent: Thursday, November 14, 2013 3:42 PM
To: d...@accumulo.apache.org; user@accumulo.apache.org
Subject: [VOTE] Deprecate mock in 1.6.0

 

Should we deprecate mock accumulo for 1.6.0?  This was considered [1] for
1.5.0.  I started thinking about this because I never added conditional
writer to mock.

 

[1] : https://issues.apache.org/jira/browse/ACCUMULO-878

RE: Bulk Ingest data latency [SEC=UNOFFICIAL]

[dlmarion]

Data is available immediately. The counts on the monitor wont update until a 
compaction occurs.


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

 Original message 
From: Dickson, Matt MR matt.dick...@defence.gov.au 
Date: 10/17/2013  8:35 PM  (GMT-05:00) 
To: 'user@accumulo.apache.org' user@accumulo.apache.org 
Subject: Bulk Ingest data latency [SEC=UNOFFICIAL] 
 
UNOFFICIAL

 
When doing a bulk ingest and importing a directory of rfiles, should the new 
data be available immediately or do I need to trigger a compaction to bring the 
new data online? 
 
Thanks in advance,
Matt

Re: Straggler problem in Accumulo BatchScans

[dlmarion]

You can set it in the shell on the table. Just override the default tablet 
balancer for the table. I think the master has to use the Table load balancer 
also if it is not set by default. 

- Original Message -

From: David M. Slater david.sla...@jhuapl.edu 
To: user@accumulo.apache.org 
Sent: Wednesday, August 21, 2013 8:12:46 PM 
Subject: RE: Straggler problem in Accumulo BatchScans 



Thanks Eric, 



Just to make sure I’m going in the right direction, this would involve 
extending the TabletBalancer class, correct? How do I add it to the table after 
that (and remove the old one)? I don’t see it under the Connector’s 
TableOperations(). 



Is using a load-balancer what you would recommend if I wanted to make sure that 
two different tables stored related information (e.g. data and indexes) on the 
same tablets? 



Thanks, 
David 



From: Eric Newton [mailto:eric.new...@gmail.com] 
Sent: Wednesday, August 21, 2013 8:03 PM 
To: user@accumulo.apache.org 
Subject: Re: Straggler problem in Accumulo BatchScans 




A new balancer is a plug-in class that instructs the Master process where to 
place tablets. 





If you know you need your tablets spread out over servers based on time (row 
id), you can do that. It's pretty common, in fact. 





-Eric 





On Wed, Aug 21, 2013 at 7:54 PM, Slater, David M.  david.sla...@jhuapl.edu  
wrote: 


Hi Dave, 



The table is currently organizing netflow data with its rowID of 
timestamp_netflowRecordID, some columns corresponding to various netflow 
quantites, and one column representing the entire netflow in binary form. 



The table is about 1.2 TB, and I am scanning 5-40 GB per scan, which scans 
about 7-28 tablets. 



What do you mean by a custom load balancer? Do you mean balancing the data on 
ingest, or balancing the query load? What would you recommend for balancing the 
query load if I can only retrieve the data from a particular tablet? 



I’ve played with index/data caches, though I haven’t used readahead threads or 
max open files. Is that referring to rfiles? 



I’m noticing that most of the queries are CPU bound, and that read i/o is not 
being hit very hard. Is that a typical behavior for scans? 



Thanks, 
David 




From: Dave Marion [mailto: dlmar...@comcast.net ] 


Sent: Wednesday, August 21, 2013 7:29 PM 
To: user@accumulo.apache.org 


Subject: RE: Straggler problem in Accumulo BatchScans 




How is the table organized? 

What percent of the table are you scanning in these large operations? 

Have you considered writing a custom load balancer? 



I don’t think that a tablet can be hosted on multiple servers. But you might be 
able to play around with the index/data caches, readahead threads (concurrent 
queries), and max open files to achieve better performance. 




From: Slater, David M. [ mailto:david.sla...@jhuapl.edu ] 
Sent: Wednesday, August 21, 2013 7:09 PM 
To: user@accumulo.apache.org 
Subject: Straggler problem in Accumulo BatchScans 




Hey, I have a 7 node network running accumulo 1.4.1 and hadoop 1.0.4. 



When I run large BatchScanner operations, the number of tablets scanned per 
node is not uniform, leading to the overloaded nodes taking much longer to 
finish than the others. For queries that require all of the scans to finish 
before returning, this is a major latency issue. What are some practical means 
of load-balancing this to reduce delay? 



Is it possible for tablets to be hosted on multiple tablet servers, up to the 
replication factor of the underlying hdfs? Are there reasons this might be an 
undesirable design? 



Thanks in advance, 
David 

RE: Move high ingest tablets to alternate node? [SEC=UNOFFICIAL]

[dlmarion]

You can create a custom load balancer for your table. I think you said that 
your tables are organized by date. If the hot spot are contiguous days, then 
you could write a load balancer that does a round robin assignment.


Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

 Original message 
From: Dickson, Matt MR matt.dick...@defence.gov.au 
Date: 08/20/2013  9:50 PM  (GMT-05:00) 
To: 'user@accumulo.apache.org' user@accumulo.apache.org 
Subject: Move high ingest tablets to alternate node? [SEC=UNOFFICIAL] 
 
UNOFFICIAL

Hi,
 
While loading data we are seeing one node with 3 tablets that have higher 
ingest rates than all others on the cluster.  All 3 of the these tablets reside 
on the same node which is causing that node to struggle.  When we take this 
node out of the cluster the overall ingest rate jumps much higher. 
 
Is there a way to manually move high ingest tablets to different nodes in order 
to seperate the 3 tablets in question? 
 
I started looking at the tablet splits and tried to find the mapping of these 
splits to nodes in the !METADATA table but couldn't find the mappings.
 
Thanks in advance,
Matt Dickson

Re: Accumulo / HBase migration

[dlmarion]

I believe that Brian Loss committed code in 1.5 for a column visibility 
correction iterator or something that you could use to do this. You could use 
that and compact the table after the import. 

- Original Message -
From: Donald Miner dmi...@clearedgeit.com 
To: user@accumulo.apache.org 
Sent: Tuesday, July 9, 2013 1:36:20 PM 
Subject: Re: Accumulo / HBase migration 


I did think about this. My naive answer is just by default ignore visibilities 
(meaning make everything public or make everything the same visibility). It 
would be interesting however to be able to insert a chunk of code that inferred 
the visibility from the record itself. That is, you'd have a function you can 
pass in that returns a ColumnVisibility and takes in a value/rowkey/etc. 




On Tue, Jul 9, 2013 at 1:31 PM, Kurt Christensen  hoo...@hoodel.com  wrote: 



I don't have a response to your question, but it seems to me that the big 
capability difference is visibility field. When doing bulk translations like 
this, do you just fill visibility with some default value? 

-- Kurt 



On 7/9/13 1:26 PM, Donald Miner wrote: 

blockquote


Has anyone developed tools to migrate data from an existing HBase 
implementation to Accumulo? My team has done it manually in the past but it 
seems like it would be reasonable to write a process that handled the steps in 
a more automated fashion. 

Here are a few sample designs I've kicked around: 

HBase - mapreduce - mappers bulk write to accumulo - Accumulo 
or 
HBase - mapreduce - tfiles via AccumuloFileOutputFormat - Accumulo bulk load 
- Accumulo 
or 
HBase - bulk export - map-only mapreduce to translate hfiles into tfiles (how 
hard would this be??) - Accumulo bulk load - Accumulo 

I guess this could be extended to go the other way around (and also include 
Cassandra perhaps). 

Maybe we'll start working on this soon. I just wanted to kick the idea out 
there to see if it's been done before or if anyone has some gut reactions to 
the process. 

-Don 

This communication is the property of ClearEdge IT Solutions, LLC and may 
contain confidential and/or privileged information. Any review, 
retransmissions, dissemination or other use of or taking of any action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you receive this communication in error, please 
immediately notify the sender and destroy all copies of the communication and 
any attachments. 



-- 

Kurt Christensen 
P.O. Box 811 
Westminster, MD 21158-0811 

-- --  
I'm not really a trouble maker. I just play one on TV. 

/blockquote



-- 






Donald Miner 
Chief Technology Officer 
ClearEdge IT Solutions, LLC 
Cell: 443 799 7807 
www.clearedgeit.com 

This communication is the property of ClearEdge IT Solutions, LLC and may 
contain confidential and/or privileged information. Any review, 
retransmissions, dissemination or other use of or taking of any action in 
reliance upon this information by persons or entities other than the intended 
recipient is prohibited. If you receive this communication in error, please 
immediately notify the sender and destroy all copies of the communication and 
any attachments. 

Re: v1.6.0; What are the different levels in the shell classpath command?

[dlmarion]

I think these might be more accurate descriptions based on the changes that 
occurred to the code after the point at which that snippet was put into the 
ACCUMULO-708 comments. 



1. JVM s ystem c lasspath --- JDK jars on the local server 

2. JVM application classpath  --- Hadoop, ZK, accumulo-start and dependencies 
on the local server 

3. Accumulo system classpath   The rest of the accumulo jars and their 
dependencies available via some Commons VFS destination 

4. Accumulo context classpaths --- user jars specific to a context that they 
define and bound to one or more tables, available via some Commons VFS 
destination . 



- Original Message -


From: Christopher ctubb...@apache.org 
To: user@accumulo.apache.org 
Sent: Friday, April 12, 2013 12:37:30 PM 
Subject: Re: v1.6.0; What are the different levels in the shell classpath 
command? 

+1 for creating a ticket to bake the descriptions into the display 

-- 
Christopher L Tubbs II 
http://gravatar.com/ctubbsii 


On Fri, Apr 12, 2013 at 12:17 PM, David Medinets 
david.medin...@gmail.com wrote: 
 Thanks for the references. It seems like the following is the important bit. 
 If this hierarchy is stable, can we bake it into the classpath display? 
 
 Level 1: SystemClassLoader 
 Level 2: URLClassLoader 
 Level 3: AccumuloContextClassLoader 
 Level 4: MapString, AccumuloReloadingVFSClassLoader 

Re: Accumulo VM

[dlmarion]

John, 



  Where is this VM located? 



Dave 



- Original Message -


From: John Vines vi...@apache.org 
To: user@accumulo.apache.org 
Sent: Friday, November 30, 2012 3:17:48 PM 
Subject: Accumulo VM 

I recut an Accumulo-1.4.2 VM for people to have a quick start with Accumulo. It 
is configured to use 4GB of memory. I eventually be updating it with a new VM 
which set the appropriate footprint at startup so you can work on a smaller 
sized VM, but for now this is what's available. 

Everything is init.d, you just need to wait a few minutes for everything to get 
up and going. You can view the monitor on port 50095 after starting up the VM 
to check to see if it's all the way up. It is based on Ubuntu Server 12.04.1. 

Any questions/comments are appreciated. 
John 

Re: Number of partitions for sharded table

[dlmarion]

Krishmin, 

In the wikisearch example there is a non-sharded index table and a sharded 
document table. The index table is used to reduce the number of tablets that 
need to be searched for a given set of terms. Is your setup similar? I'm a 
little confused since you mention using a sharded index table and that all 
queries will have an infinite range. 

Dave Marion 

- Original Message -
From: Krishmin Rai kr...@missionfoc.us 
To: user@accumulo.apache.org 
Sent: Tuesday, October 30, 2012 3:28:15 PM 
Subject: Re: Number of partitions for sharded table 

I should clarify that I've been pre-splitting tables at each shard so that each 
tablet consists of a single row. 

On Oct 30, 2012, at 3:06 PM, Krishmin Rai wrote: 

 Hi All, 
 We're working with an index table whose row is a shardId (an integer, like 
 the wiki-search or IndexedDoc examples). I was just wondering what the right 
 strategy is for choosing a number of partitions, particularly given a cluster 
 that could potentially grow. 
 
 If I simply set the number of shards equal to the number of slave nodes, 
 additional nodes would not improve query performance (at least over the data 
 already ingested). But starting with more partitions than slave nodes would 
 result in multiple tablets per tablet server… I'm not really sure how that 
 would impact performance, particularly given that all queries against the 
 table will be batchscanners with an infinite range. 
 
 Just wondering how others have addressed this problem, and if there are any 
 performance rules of thumb regarding the ratio of tablets to tablet servers. 
 
 Thanks! 
 Krishmin 

Re: Running Accumulo straight from Memory

[dlmarion]

Matt, 



  Did you see Eric Newton's response yesterday? Running on a ram disk has been 
done; however minor and major compactions will still occur. 



- Dave 



- Original Message -


From: Matthew J. Moore matthew.j.mo...@saic.com 
To: user@accumulo.apache.org 
Sent: Wednesday, September 12, 2012 12:32:31 PM 
Subject: RE: Running Accumulo straight from Memory 




Adam, 

It does look like we are the first to try this.  We are trying to keep 
everything in memory and as a result there is no minor compactions, and 
probably major compactions to make tables larger.  We tried this on SSDs using 
a file system and we were not getting the processing speeds that we had wanted. 

  

Matt 

  

  


From: user-return-1330-MATTHEW.J.MOORE=saic@accumulo.apache.org 
[mailto:user-return-1330-MATTHEW.J.MOORE=saic@accumulo.apache.org] On 
Behalf Of Adam Fuchs 
Sent: Tuesday, September 11, 2012 5:30 PM 
To: user@accumulo.apache.org 
Subject: Re: Running Accumulo straight from Memory 

  

Matthew, 


  


I don't know of anyone who has done this, but I believe you could: 


1. mount a RAM disk 


2. point the hdfs core-site.xml  fs.default.name  property to file:/// 


3. point the accumulo-site.xml instance.dfs.dir property to a directory on the 
RAM disk 


4. disable the WAL for all tables by setting the accumulo-site.xml 
table.walog.enabled to false 


5. initialize and start up accumulo as you regularly would and cross your 
fingers 

Of course, the you may lose data and this is not an officially supported 
configuration caveats apply. Out of curiosity, what would you be trying to 
accomplish with this configuration? 


  


Adam 


  

  


On Tue, Sep 11, 2012 at 12:02 PM, Moore, Matthew J.  matthew.j.mo...@saic.com 
 wrote: 



Has anyone run Accumulo on a single server straight from memory?  Probably 
using something like a Fusion  IO drive.  We are trying to use it without using 
an SSD or any spinning discs. 

  

Matthew Moore 

Systems Engineer 

SAIC, ISBU 

Columbia, MD 

410-312-2542