Re: Running Accumulo straight from Memory

Matt,   Did you see Eric Newton's response yesterday? Running on a ram disk has been done; however minor and major compactions will still occur. - Dave - Original Message - From: Matthew J. Moore matthew.j.mo...@saic.com To: user@accumulo.apache.org Sent: Wednesday,

Re: Number of partitions for sharded table

Krishmin, In the wikisearch example there is a non-sharded index table and a sharded document table. The index table is used to reduce the number of tablets that need to be searched for a given set of terms. Is your setup similar? I'm a little confused since you mention using a sharded index

Re: Accumulo VM

John,   Where is this VM located? Dave - Original Message - From: John Vines vi...@apache.org To: user@accumulo.apache.org Sent: Friday, November 30, 2012 3:17:48 PM Subject: Accumulo VM I recut an Accumulo-1.4.2 VM for people to have a quick start with Accumulo. It

Re: v1.6.0; What are the different levels in the shell classpath command?

I think these might be more accurate descriptions based on the changes that occurred to the code after the point at which that snippet was put into the ACCUMULO-708 comments. 1. JVM s ystem c lasspath --- JDK jars on the local server 2. JVM application classpath  --- Hadoop, ZK,

Re: Accumulo / HBase migration

I believe that Brian Loss committed code in 1.5 for a column visibility correction iterator or something that you could use to do this. You could use that and compact the table after the import. - Original Message - From: Donald Miner dmi...@clearedgeit.com To:

RE: Move high ingest tablets to alternate node? [SEC=UNOFFICIAL]

You can create a custom load balancer for your table. I think you said that your tables are organized by date. If the hot spot are contiguous days, then you could write a load balancer that does a round robin assignment. Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone

Re: Straggler problem in Accumulo BatchScans

You can set it in the shell on the table. Just override the default tablet balancer for the table. I think the master has to use the Table load balancer also if it is not set by default. - Original Message - From: David M. Slater david.sla...@jhuapl.edu To: user@accumulo.apache.org

RE: Bulk Ingest data latency [SEC=UNOFFICIAL]

Data is available immediately. The counts on the monitor wont update until a compaction occurs. Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone Original message From: Dickson, Matt MR matt.dick...@defence.gov.au Date: 10/17/2013 8:35 PM (GMT-05:00) To:

RE: [VOTE] Deprecate mock in 1.6.0

+1 From: Keith Turner [mailto:ke...@deenlo.com] Sent: Thursday, November 14, 2013 3:42 PM To: d...@accumulo.apache.org; user@accumulo.apache.org Subject: [VOTE] Deprecate mock in 1.6.0 Should we deprecate mock accumulo for 1.6.0? This was considered [1] for 1.5.0. I started thinking

RE: utility to print Accumulo config (feedback appreciated)

There is a dumpconfig command in 1.6.0. The source should help you: https://github.com/apache/accumulo/blob/1.6.0-SNAPSHOT/server/base/src/main/ java/org/apache/accumulo/server/util/Admin.java. Look at the printConfig method. From: Sujee Maniyam [mailto:su...@sujee.net] Sent: Tuesday,

RE: Found two locations for the same extent

Have you tried killing one or both of the tservers? Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone Original message From: Anthony F afc...@gmail.com Date:01/25/2014 10:12 AM (GMT-05:00) To: user@accumulo.apache.org Subject: Found two locations for the same

Re: Found two locations for the same extent

permissions on !METADATA to root and was able to delete one of the offending entries.  At that point, the system stabilized.  Any tips on avoiding getting into this situation? On Sat, Jan 25, 2014 at 10:51 AM, dlmarion dlmar...@comcast.net wrote: Have you tried killing one or both of the tservers

RE: best practice for deploying server side iterator jars

You will also want to set the replication on the directory/jars in HDFS to something high. I think in my testing I could not set it above 50. -Original Message- From: Josh Elser [mailto:josh.el...@gmail.com] Sent: Thursday, February 20, 2014 12:30 PM To: user@accumulo.apache.org Subject:

RE: pre-split table tablet server assignment

The master runs a load balancer. Sent via the Samsung GALAXY S®4, an ATT 4G LTE smartphone Original message From: Kepner, Jeremy - 0553 - MITLL kep...@ll.mit.edu Date:03/02/2014 8:34 PM (GMT-05:00) To: user@accumulo.apache.org Subject: pre-split table tablet server

RE: Hadoop HA with Accumulo 1.5

We had an issue in our testing (https://issues.apache.org/jira/browse/ACCUMULO-2480). The root cause was a misconfiguration for automatic failover. The sshfence feature does not handle network failures, so you have to configure it with the shell(/bin/true) command also (separated by a newline,

RE: Advice on increasing ingest rate

How many threads are you using in the AccumuloOutputFormat? What is your latency set to? From: Adam Fuchs [mailto:afu...@apache.org] Sent: Tuesday, April 08, 2014 5:36 PM To: user@accumulo.apache.org Subject: Re: Advice on increasing ingest rate MIke, What version of Accumulo are

RE: Losing tservers - Unusually high Last Contact times

You are hitting the zookeeper timeout, default 30s I believe. You said you are not oversubscribed for memory, but what about CPU? Are you running YARN processes on the same nodes as the tablet servers? Is the tablet server being pushed into swap or starved of CPU? -Original Message- From:

Re: Unable to load Iterator with setscaniter and setshelliter

the .../target/classes/ values. /description /property On Sun, Jun 15, 2014 at 10:49 AM, dlmarion dlmar...@comcast.net wrote: What does your classpath settings look like in accumulo-site.xml. I recently made some fixes in 1.6.1-Snapshot where the context classloader was not being used

Re: accumulo 1.6 and HDFS non-HA conversion to HDFS HA

I believe the problem that you are running into is that because dfs.default.uri was not specified, then fs.defaultFS was used to write entries to the accumulo.root and accumulo.metadata tables. Suggest doing the following: Update to the latest version of Accumulo 1.6.1-SNAPSHOT set

Re: (U) I do I tell Accumulo where the jars for my custom formatters and balancers are for a specific table?

We fixed some issues recently with the classloader in 1.6.1, make sure you are using an up-to-date snapshot version. The classloader was changed in version 1.5 and now supports hosting jars from various locations, including HDFS. Additionally, it supports creating a table level classloader that

RE: Moving HDFS name node to another host in accumulo 1.6

What Accumulo version? What Hadoop version? From: Calvin Feder [mailto:calvin.fe...@argyledata.com] Sent: Friday, January 16, 2015 7:29 PM To: user@accumulo.apache.org Subject: Moving HDFS name node to another host in accumulo 1.6 We need to move the HDFS name node of our cluster to a

RE: Moving HDFS name node to another host in accumulo 1.6

In 1.6, accumulo.metadata entries are absolute URLs. You can change the existing URLs using the instance.volume.replacments[1] parameter. [1] http://accumulo.apache.org/1.6/accumulo_user_manual.html#_instance_volumes_replacements -Original Message- From: webmas...@webmaster.ms

RE: Installing custom authorizor

Did you restart Accumulo after you copied the jar? The general.classpaths locations are not monitored and loaded dynamically. div Original message /divdivFrom: Srikanth Viswanathan srikant...@gmail.com /divdivDate:02/18/2015 8:42 PM (GMT-05:00) /divdivTo:

Re: Custom Iterator output

via the getTopKey() and getTopValue() methods. [1] should be a simple example. [1] https://git-wip-us.apache.org/repos/asf?p=accumulo.git;a=blob;f=core/src/main/java/org/apache/accumulo/core/iterators/user/GrepIterator.java;h=043a729a778fc34d2ee87a0227056ffac81b7fe7;hb=refs/heads/master

RE: Accumulo GC and Hadoop trash settings

you determine the MTTR and then schedule your backups accordingly; a backup in case you are not able to recover your database using the techniques in the current documentation. [1] https://github.com/dlmarion/raccovery From: James Hughes [mailto:jn...@virginia.edu] Sent: Monday, August

Re: thrift versions in accumulo

+1 to Netty. I have used it and like it, but I think you will have to write all of the code yourself. Nifty (Facebook) could be an option to use Thrift and Netty together. Finagle (Twitter) would be another alternative. - Original Message - From: Christopher ctubb...@apache.org To:

RE: Accumulo Monitor NullPointerException

Anything in the master log? Original message From: Scott sct...@gmail.com Date: 08/21/2015 6:15 PM (GMT-05:00) To: user@accumulo.apache.org Subject: Accumulo Monitor NullPointerException Hello,   We have Accumulo 1.6.3 set up on the HortonWorks install with a

Re: Accumulo GC and Hadoop trash settings

It's not temporary files, it's any file that has been compacted away. If you keep files around longer than { dfs.namenode.checkpoint.period}, then you have a chance to recover in case your most recent checkpoint is corrupt. - Original Message - From: James Hughes jn...@virginia.edu

Re: Entry-based TableBalancer

Hotspotting was the first thing that came to my mind with the proposed balancer. The fservers don't keep all the K/V in memory. You are balancing query and live ingest across your resources. Original message From: Eric Newton eric.new...@gmail.com Date: 07/29/2015

RE: How to control Minor Compaction by programming

It sounds like you want to try and not minor compact during your ingest of your data. Is that correct? From: William Slacum [mailto:wsla...@gmail.com] Sent: Thursday, July 30, 2015 8:10 PM To: user@accumulo.apache.org Subject: Re: How to control Minor Compaction by programming See

RE: TSDB on Accumulo row key

https://github.com/ericnewton/accumulo-opentsdb From: Ranjan Sen [mailto:ranjan_...@hotmail.com] Sent: Monday, July 20, 2015 6:25 PM To: user@accumulo.apache.org Subject: TSDB on Accumulo row key Hi All, Is there something like TSDB (Time series database) on Accumulo? Thanks

Re: Tserver's strange state.

Are you trying to shut the whole system down, or just a couple of tablet servers?Is your application reading and writing from/to Accumulo during this time? Original message From: Denis Date: 10/22/2015 6:03 PM (GMT-05:00) To:

RE: Tserver's strange state.

gt; Yes, it is a production system with a lot of reads and writes. > > On 10/22/15, dlmarion <dlmar...@comcast.net> wrote: > > > > > > Are you trying to shut the whole system down, or just a couple of > > tablet servers?Is your applic

RE: Accumulo Iterator painful development because TS don't pick up changes to Jars

So, without seeing your configuration, I would suggest trying something before upgrading to 1.7. In 1.5 we changed the classloader so that it could load from different locations. At the same time, we added the concept of classloader contexts which are basically names for locations for jars.

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been fixed, but one of them is that if more than one monitored resource changed then it would miss some of them. - Original Message - From: "Rob Povey" To: user@accumulo.apache.org Sent:

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Also, turn the logging on the tservers up to DEBUG for org.apache.accumulo.start.classloader.*. You should see a line in the log that starts with "monitoring " - Original Message - From: dlmar...@comcast.net To: user@accumulo.apache.org Sent: Friday, October 30, 2015 12:22:53 PM

Re: How does Accumulo process a r-files for bulk ingesting?

I believe the RFile is assigned to each tablet that contain keys in the RFiles key range. So, 1 RFile is assigned to one or more tablets. The bulk import RFile is removed after it has been compacted by all assigned tablets. - Original Message - From: "Jeff Kubina"

RE: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library

Dylan,   I am a little confused about whether you want to place this in the contrib area or whether you want to create a sub-project as both are mentioned in your proposal. Also, if you intend for this to be a sub-project, have you looked at the incubator process? From what I understand

Re: [Accumulo Contrib Proposal] Graphulo: Server-side Matrix Math library

, Dylan On Fri, Aug 28, 2015 at 8:08 AM, dlmarion dlmar...@comcast.net wrote: blockquote Dylan, I am a little confused about whether you want to place this in the contrib area or whether you want to create a sub-project as both are mentioned in your proposal. Also, if you intend

Re: Accumulo: "BigTable" vs. "Document Model"

Both will work, but I think the answer depends on the amount of data that you will be querying over and your query latency requirements. I would include the wikisearch[1] storage scheme into your list as well (k/v table + indices). Then, personally, I would rate them in the following order as

RE: imbalance in number of zookeeper clients

Hey Jeff,  Take a look at [1] and see if the zookeeper balance issue mentioned is applicable. Dave [1] https://accumulo.apache.org/release_notes/1.6.2.html Original message From: Jeff Turner Date: 09/10/2015 7:42 PM (GMT-05:00) To:

RE: Error BAD_AUTHORIZATIONS for user root

It’s likely that you created a table, inserted data with column visibilities, but never gave the user the authorization tokens to see the data. Try setting the authorization tokens for the user with the setauths command. Since you are using the ‘root’ user, it would be something like: setauths

Re: [ADVISORY] Possible data loss during HDFS decommissioning

Known issue in the release notes on the web page? We would have to update every version though. Seems like we need a known issues document that lists issues in dependencies that transcend Accumulo versions. - Original Message - From: "Josh Elser" To:

Re: Presplitting tables for the YCSB workloads

I don't have a script for you, but if you need to create one you could use the script command in the shell to do something similar to the hbase script. Some examples are in the comments in jira[1]. If you can figure out how you want the table split, and it can be scripted, I might have time

Re: loading iterator jars from file:// or http://

Suggest replacing the VFS 2.0 jar in the Accumulo lib directory with a VFS 2.1-SNAPSHOT jar from [1]. There are some bugs that have been fixed, but VFS 2.1 has not been released yet. I did this and have not had many issues loading from HDFS since. Loading jars from file:/// should work, I

[ANNOUNCE] Timely - Secure Time Series Database

Timely is a time series database application that provides secure access to time series data. It is designed to be used with Apache Accumulo for persistence and Grafana for visualization. Timely is located at https://github.com/NationalSecurityAgency/timely .

Re: Class path for shell commands

try: ./accumulo -add shell - Original Message - From: "Christopher" To: user@accumulo.apache.org, "Sravankumar Reddy Javaji" Sent: Monday, March 7, 2016 4:28:03 PM Subject: Re: Class path for shell commands Are you using a 1.6.x or

Re: Class path for shell commands

Another option is to drop your jar into $ACCUMULO_HOME/lib/ext on the node where you will be running the shell. - Original Message - From: "Sravankumar Reddy Javaji (BLOOMBERG/ 731 LEX)" To: user@accumulo.apache.org Sent: Monday, March 7, 2016 4:31:49 PM

Re: Configuring batch writers

The batch writer has several knobs (latency time, memory buffer, etc) that you can tune to meet your requirements. The values for those settings will depend on a lot of variables, to include: - number of tablet servers - size of mutations - desired latency - memory buffer - configuration

Re: java.lang.NoClassDefFoundError with fields of custom Filter

+1 good catch Josh - Original Message - From: "Josh Elser" To: user@accumulo.apache.org Sent: Thursday, July 7, 2016 2:12:42 PM Subject: Re: java.lang.NoClassDefFoundError with fields of custom Filter Beware using the HDFS classloader in any Accumulo release

Re: Accumulo Seek performance

Calling BatchScanner.iterator() is what starts the work on the server side. You should do this first for all 6 batch scanners, then iterate over all of them in parallel. - Original Message - From: "Sven Hodapp" To: "user"

Re: 1 of 20 TServers unresponsive/slow, all writes fail?

What was happening on that 1 tserver? Was it in garbage collection? Was it having network or O/S issues? - Original Message - From: "Michael Moss (BLOOMBERG/ 731 LEX)" To: user@accumulo.apache.org Sent: Friday, September 9, 2016 9:40:42 AM Subject: 1 of 20

Re: 1 of 20 TServers unresponsive/slow, all writes fail?

We have seen this before: a tserver that is hosting metadata tablets has issues and starts causing problems within the cluster. You could try using the HostRegexTableLoadBalancer[1,2] to segregate your metadata tablets from the other tables. This doesn't fully eliminate the SPOF, but it should

RE: how do I list user permissions per table

You can also use the admin dumpConfig command to dump all of the configuration information, then look at the user permission files. From: Christopher [mailto:ctubb...@apache.org] Sent: Friday, September 23, 2016 5:39 PM To: user@accumulo.apache.org Subject: Re: how do I list user permissions

Re: Accumulo Seek performance

Doesn't this use the 6 batch scanners serially? - Original Message - From: "Sven Hodapp" To: "user" Sent: Wednesday, August 24, 2016 11:56:14 AM Subject: Re: Accumulo Seek performance Hi Josh, thanks for your reply!

Re: List of Metrics2 Metrics

It used[1] to be in the documentation when it was hosted on the monitor. I did not see it looking at the current documentation. Looks like it was lost (and [1] is likely now out of date). [1] https://github.com/apache/accumulo/blob/1.4.0/docs/metrics.html - Original Message - From:

RE: 1.7 native library performance boost

I don't remember seeing anything about native library performance improvement, but I do know that there has been talk of a performance regression in Accumulo 1.7.0 using the default configuration. I don't remember the specific details off-hand, but it had something to do with the default value

RE: Detecting database changes

What kind of changes are you looking to alert off? Is it data changes or configuration changes? > -Original Message- > From: vaibhav thapliyal [mailto:vaibhav.thapliyal...@gmail.com] > Sent: Tuesday, November 22, 2016 7:20 AM > To: user@accumulo.apache.org > Subject: Detecting database

Re: Bulk ingestion of different locality groups at different times

>>> Is Accumulo able to import these files, considering that they are two >>> different locality groups Yes. >>> without triggering a huge major compaction? Depends on your table.compaction.major.ratio and table.file.max settings. Sorry, not a real answer, but I think the answer is "it

RE: New Accumulo Blog Post

Regarding #2 – I think there are two options here: 1. Modify Accumulo to take advantage of HDFS Heterogeneous Storage 2. Modify Accumulo WAL code to support volumes From: Jeff Kubina [mailto:jeff.kub...@gmail.com] Sent: Wednesday, November 02, 2016 9:02 PM To: user@accumulo.apache.org

Re: [ANNOUNCE] Apache Fluo 1.0.0-incubating released

Congrats! - Original Message - From: "Mike Walch" To: user@accumulo.apache.org Sent: Friday, October 14, 2016 3:21:00 PM Subject: [ANNOUNCE] Apache Fluo 1.0.0-incubating released The Apache Fluo (incubating) team is happy to announce the release of Fluo

RE: maximize usage of cluster resources during ingestion

Regarding the referenced paper, pre-splitting the tables, using an optimized zookeeper deployment, and increasing concurrent minor / major compactions are good things. I'm not sure that we want to recommend turning off the write ahead logs and replication for production deployments.

RE: accumulo.metadata table online but scans hang

Re #2: Does your Accumulo processes run as the hdfs user on the O/S, or as the accumulo user? Make sure you are checking the correct users trash folder. Also, check the Accumulo garbage collector log to see if the GC process deleted the WAL files. Take a look at [1] to see if you are hitting

RE: Problems with accumulo replication

You can also use the tserver.walog.max.age property to ensure that the walogs roll if there is no activity. The default is 24h and was backported to 1.7.2. See ACCUMULO-4004 for more info. -Original Message- From: Josh Elser [mailto:els...@apache.org] Sent: Friday, December 29, 2017

RE: Accumulo performance on various hardware configurations

This may suggest an issue with client, either getting the data to the client or the client itself (although I think there are other performance related changes you could make). I’m curious what the end goal is here. Is this a real world use case? If you are using this type of benchmark to

RE: Uniquely identifying metrics with multiple tservers per node.

IIRC the 'service.name' property in the metrics properties file is used to note from which service the metric was emitted. I would suggest testing this by setting the service.name property to tserver1 and see if your metrics show up with that name. If they do, then you might be able to modify the

RE: example thrift client to get master metrics

The Monitor[1] gets a MasterMonitorInfo[2] object from the Master that contains the information you need. The tableMap member contains a TableInfo[3] object for each table that holds the information you are looking for. [1]

RE: Tablet Server Session Id Out of Range

Please see: https://lists.apache.org/thread/p7mwtkfpbyb551pw5k7yg61jopf50m8s From: Josef Roehrl - PHEMI Sent: Monday, June 20, 2022 7:02 PM To: user@accumulo.apache.org Subject: Re: Tablet Server Session Id Out of Range Hi Jonathan, We too have exactly this issue as of a couple of

RE: Accumulo On S3

I believe that S3 Guard is OBE, but you still need to put the WAL on HDFS as S3 does not support sync. If you put your WAL in S3, and you have a tserver failure, then it’s possible that you will lose data. From: Christopher Sent: Wednesday, October 12, 2022 4:12 PM To:

RE: Problem creating tables in Accumulo

It looks like the issue is that `maxTime` is not being set at [0], so `time` is not being set at [1]. If I had to guess, Accumulo is having an issue talking with ZooKeeper. Were there errors in the log when initializing Accumulo? Are you able to `exec` into one of the Accumulo Kubernetes pods and

RE: Init accumulo with custom properties file

Take a look at https://github.com/dlmarion/accumulo-k8s. You can put the Accumulo config in a ConfigMap (accumulo-config.yaml), then mount it at the correct location in another container (see the server process yaml files). I used a job to do the init phase, see accumulo-init.yaml. From

RE: Impact from Enabling TLS

Does this apply? https://bugs.openjdk.org/browse/JDK-8190917 From: Sanjay Deshmukh Sent: Monday, June 5, 2023 3:28 PM To: user@accumulo.apache.org Subject: Re: Impact from Enabling TLS Sure thing. We're running Accumulo 1.10.1. Our configuration has 15 tablet servers and a client pool

RE: Scan thread resource management [SEC=UNOFFICIAL]

Bruce, There are no future planned releases for the 1.x line. In 2.1 Scan Executors[1] offer a way for the user to configure the resource pools used for scanning a table. You could use a prioritizer to influence the execution order of scans in the default scan executor, or setup different

RE: Issue with TableOperations

My guess is that the error is happening on the server side and being raised at https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/Base64.java#L713. If there is a stack trace in the server log, then that would help pinpoint the issue. I’m guessing here, but it