date:20151030

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

2015-10-30 Thread Rob Povey

Thanks, we’ll give this a try next week, and see if the issue is in fact the 
VFS Jar version. We certainly commonly change several iterator Jars 
simultaneously.

Rob Povey

From: "dlmar...@comcast.net" 
>
Reply-To: "user@accumulo.apache.org" 
>
Date: Friday, October 30, 2015 at 9:33 AM
To: "user@accumulo.apache.org" 
>
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars

Also, turn the logging on the tservers up to DEBUG for 
org.apache.accumulo.start.classloader.*. You should see a line in the log that 
starts with "monitoring "



From: dlmar...@comcast.net
To: user@accumulo.apache.org
Sent: Friday, October 30, 2015 12:22:53 PM
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been 
fixed, but one of them is that if more than one monitored resource changed then 
it would miss some of them.



From: "Rob Povey" >
To: user@accumulo.apache.org
Sent: Friday, October 30, 2015 11:57:27 AM
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars

Thanks for the help with this,

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change.

These are from my dev box, but the clusters look the same just with many more 
contexts

default| general.vfs.classpaths . |
site   | general.vfs.context.classpath.rob_maanaNgram ... |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
site   | general.vfs.context.classpath.rob_maanaSearch .. |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar

And then the context is set on the table like this


default| table.classpath.context  |
table  |@override ... | 
rob_maanaNgram



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the  config classpaths.

  
tserver.memory.maps.max
2G
  


  
tserver.memory.maps.native.enabled
true
  


  
tserver.cache.data.size
4G
  


  
tserver.cache.index.size
3G
  


  
trace.token.property.password
maana
  


  
trace.user
root
  


  
tserver.sort.buffer.size
200M
  


  
tserver.walog.max.size
1G
  


  
general.classpaths

  $ACCUMULO_HOME/lib/accumulo-server.jar,
  $ACCUMULO_HOME/lib/accumulo-core.jar,
  $ACCUMULO_HOME/lib/accumulo-start.jar,
  $ACCUMULO_HOME/lib/accumulo-fate.jar,
  $ACCUMULO_HOME/lib/accumulo-proxy.jar,
  $ACCUMULO_HOME/lib/[^.].*.jar,
  $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
  $HADOOP_CONF_DIR,
  $HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
  /usr/hdp/current/hadoop-client/[^.].*.jar,
  /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar,
  /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar,
  /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar,
  /usr/hdp/current/hadoop-yarn-client/[^.].*.jar,
  /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar,
  $HADOOP_PREFIX/[^.].*.jar,
  $HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar,
  /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar,

Classpaths that accumulo checks for updates and class 
files.
  


  
  instance.volumes
  hdfs://maana2/apps/accumulo
  









On 10/29/15, 5:27 PM, "dlmar...@comcast.net" 
> wrote:

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the

Transaction type query in accumulo

2015-10-30 Thread shweta.agrawal


Hi,

Is transaction type facility available in Accumulo?
I have read about transaction in accumulo which says " Accumulo 
guarantees these ACID properties for a single mutation (a set of changes 
for a single row) but does not provide support for atomic updates across 
multiple rows"


In my case:
If one thread is updating the fields of a document then this document 
should be locked so that other thread cannot modify that document.


I am trying to achieve this by a query through conditional mutation. I 
am checking whether the particular entry exist or not  then updating. 
But the problem is I am doing this through 150 threads. If one thread 
finds and updating particular entry then other thread should not get it.


So is this the case in conditional write?

We are achieving same thing through mongoDB by find and modify feature.

If one thread get particular document to update from conditional write 
then other thread should get that particular document.


Please provide your inputs

Thanks
Shweta

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

2015-10-30 Thread Max Jordan

We've gotten a lot of mileage out of versioning product and iterator 
folders.  Every new set of iterators gets a new folder and therefore 
classpath.  This forces VFS to load new iterators.  In HDFS we'll have 
many different folders; /product/product-version/iterator-version


Max

On 10/30/2015 11:57 AM, Rob Povey wrote:

Thanks for the help with this,

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change.

These are from my dev box, but the clusters look the same just with many more 
contexts

default| general.vfs.classpaths . |
site   | general.vfs.context.classpath.rob_maanaNgram ... |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
site   | general.vfs.context.classpath.rob_maanaSearch .. |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar

And then the context is set on the table like this


default| table.classpath.context  |
table  |@override ... | 
rob_maanaNgram



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the  config classpaths.

   
 tserver.memory.maps.max
 2G
   


   
 tserver.memory.maps.native.enabled
 true
   


   
 tserver.cache.data.size
 4G
   


   
 tserver.cache.index.size
 3G
   


   
 trace.token.property.password
 maana
   


   
 trace.user
 root
   


   
 tserver.sort.buffer.size
 200M
   


   
 tserver.walog.max.size
 1G
   


   
 general.classpaths
 
   $ACCUMULO_HOME/lib/accumulo-server.jar,
   $ACCUMULO_HOME/lib/accumulo-core.jar,
   $ACCUMULO_HOME/lib/accumulo-start.jar,
   $ACCUMULO_HOME/lib/accumulo-fate.jar,
   $ACCUMULO_HOME/lib/accumulo-proxy.jar,
   $ACCUMULO_HOME/lib/[^.].*.jar,
   $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
   $HADOOP_CONF_DIR,
   $HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
   $HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
   $HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
   $HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
   $HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
   /usr/hdp/current/hadoop-client/[^.].*.jar,
   /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar,
   /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar,
   /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar,
   /usr/hdp/current/hadoop-yarn-client/[^.].*.jar,
   /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar,
   $HADOOP_PREFIX/[^.].*.jar,
   $HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar,
   /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar,
 
 Classpaths that accumulo checks for updates and class 
files.
   


   
   instance.volumes
   hdfs://maana2/apps/accumulo
   









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote:


So, without seeing your configuration, I would suggest trying something before upgrading 
to 1.7. In 1.5 we changed the classloader so that it could load from different locations. 
At the same time, we added the concept of classloader contexts which are basically names 
for locations for jars. Table(s) can be configured to use a classloader context allowing 
you to deploy server side code for different applications in different locations. This 
new classloader does "reload" jars on the classpath when they change; the same 
behavior with the older classloader reading from lib/ext. You can read more about this 
feature at [1].

We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
fixed and you will have a better experience if you replace the VFS jar in the 
lib directory with a snapshot of the 2.1 release[2].

[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader
[2] 
https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target



-Original Message-
From: dlmar...@comcast.net [mailto:dlmar...@comcast.net]
Sent: Thursday, October 29, 2015 8:04 PM
To: user@accumulo.apache.org
Subject: RE: Accumulo Iterator painful development because TS don't pick up
changes to Jars

Can you provide the relevant classpath sections of your accumulo-site.xml
file?


-Original Message-
From: Rob Povey [mailto:r...@maana.io]
Sent: Thursday, October 29, 2015 8:01 PM
To: user@accumulo.apache.org

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

2015-10-30 Thread dlmarion

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been 
fixed, but one of them is that if more than one monitored resource changed then 
it would miss some of them. 


- Original Message -

From: "Rob Povey"  
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 11:57:27 AM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Thanks for the help with this, 

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change. 

These are from my dev box, but the clusters look the same just with many more 
contexts 

default | general.vfs.classpaths . | 
site | general.vfs.context.classpath.rob_maanaNgram ... | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
 
site | general.vfs.context.classpath.rob_maanaSearch .. | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar
 

And then the context is set on the table like this 


default | table.classpath.context  | 
table | @override ... | rob_maanaNgram 



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the config classpaths. 

 
tserver.memory.maps.max 
2G 
 


 
tserver.memory.maps.native.enabled 
true 
 


 
tserver.cache.data.size 
4G 
 


 
tserver.cache.index.size 
3G 
 


 
trace.token.property.password 
maana 
 


 
trace.user 
root 
 


 
tserver.sort.buffer.size 
200M 
 


 
tserver.walog.max.size 
1G 
 


 
general.classpaths 
 
$ACCUMULO_HOME/lib/accumulo-server.jar, 
$ACCUMULO_HOME/lib/accumulo-core.jar, 
$ACCUMULO_HOME/lib/accumulo-start.jar, 
$ACCUMULO_HOME/lib/accumulo-fate.jar, 
$ACCUMULO_HOME/lib/accumulo-proxy.jar, 
$ACCUMULO_HOME/lib/[^.].*.jar, 
$ZOOKEEPER_HOME/zookeeper[^.].*.jar, 
$HADOOP_CONF_DIR, 
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar, 
/usr/hdp/current/hadoop-client/[^.].*.jar, 
/usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, 
/usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, 
$HADOOP_PREFIX/[^.].*.jar, 
$HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar, 
 
Classpaths that accumulo checks for updates and class 
files. 
 


 
instance.volumes 
hdfs://maana2/apps/accumulo 
 









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote: 

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the concept of classloader 
>contexts which are basically names for locations for jars. Table(s) can be 
>configured to use a classloader context allowing you to deploy server side 
>code for different applications in different locations. This new classloader 
>does "reload" jars on the classpath when they change; the same behavior with 
>the older classloader reading from lib/ext. You can read more about this 
>feature at [1]. 
> 
>We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
>fixed and you will have a better experience if you replace the VFS jar in the 
>lib directory with a snapshot of the 2.1 release[2]. 
> 
>[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader 
>[2] 
>https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target
> 
> 
> 
>> -Original Message- 
>> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net] 
>> Sent: Thursday, October 29, 2015 8:04 PM 
>> To: user@accumulo.apache.org 
>> Subject: RE: Accumulo Iterator painful development because TS don't pick up 
>> changes to Jars 
>> 
>> Can you provide the relevant classpath sections of your accumulo-site.xml 
>> file? 
>> 
>> > -Original Message- 
>> > From: Rob Povey [mailto:r...@maana.io] 
>> > Sent: Thursday, October 29, 2015 8:01 PM 
>> > To: user@accumulo.apache.org 
>> > Subject: Accumulo Iterator painful development because TS don't pick 
>> > up changes to Jars 
>> >

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

2015-10-30 Thread dlmarion

Also, turn the logging on the tservers up to DEBUG for 
org.apache.accumulo.start.classloader.*. You should see a line in the log that 
starts with "monitoring " 


- Original Message -

From: dlmar...@comcast.net 
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 12:22:53 PM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Try replacing the vfs jar in lib with a 2.1-SNAPSHOT. Several issues have been 
fixed, but one of them is that if more than one monitored resource changed then 
it would miss some of them. 


- Original Message -

From: "Rob Povey"  
To: user@accumulo.apache.org 
Sent: Friday, October 30, 2015 11:57:27 AM 
Subject: Re: Accumulo Iterator painful development because TS don't pick up 
changes to Jars 

Thanks for the help with this, 

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change. 

These are from my dev box, but the clusters look the same just with many more 
contexts 

default | general.vfs.classpaths . | 
site | general.vfs.context.classpath.rob_maanaNgram ... | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
 
site | general.vfs.context.classpath.rob_maanaSearch .. | 
system | @override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar
 

And then the context is set on the table like this 


default | table.classpath.context  | 
table | @override ... | rob_maanaNgram 



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the config classpaths. 

 
tserver.memory.maps.max 
2G 
 


 
tserver.memory.maps.native.enabled 
true 
 


 
tserver.cache.data.size 
4G 
 


 
tserver.cache.index.size 
3G 
 


 
trace.token.property.password 
maana 
 


 
trace.user 
root 
 


 
tserver.sort.buffer.size 
200M 
 


 
tserver.walog.max.size 
1G 
 


 
general.classpaths 
 
$ACCUMULO_HOME/lib/accumulo-server.jar, 
$ACCUMULO_HOME/lib/accumulo-core.jar, 
$ACCUMULO_HOME/lib/accumulo-start.jar, 
$ACCUMULO_HOME/lib/accumulo-fate.jar, 
$ACCUMULO_HOME/lib/accumulo-proxy.jar, 
$ACCUMULO_HOME/lib/[^.].*.jar, 
$ZOOKEEPER_HOME/zookeeper[^.].*.jar, 
$HADOOP_CONF_DIR, 
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar, 
$HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar, 
/usr/hdp/current/hadoop-client/[^.].*.jar, 
/usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hadoop-hdfs-client/[^.].*.jar, 
/usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/[^.].*.jar, 
/usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar, 
$HADOOP_PREFIX/[^.].*.jar, 
$HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar, 
/usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar, 
 
Classpaths that accumulo checks for updates and class 
files. 
 


 
instance.volumes 
hdfs://maana2/apps/accumulo 
 









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote: 

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the concept of classloader 
>contexts which are basically names for locations for jars. Table(s) can be 
>configured to use a classloader context allowing you to deploy server side 
>code for different applications in different locations. This new classloader 
>does "reload" jars on the classpath when they change; the same behavior with 
>the older classloader reading from lib/ext. You can read more about this 
>feature at [1]. 
> 
>We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
>fixed and you will have a better experience if you replace the VFS jar in the 
>lib directory with a snapshot of the 2.1 release[2]. 
> 
>[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader 
>[2] 
>https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target
> 
> 
> 
>> -Original Message- 
>> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net] 
>> Sent: Thursday, October 29, 2015 8:04 PM 
>> To: user@accumulo.apache.org 
>> Subject: RE: Accumulo Iterator painful development because TS

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

2015-10-30 Thread Rob Povey

Thanks for the help with this, 

To be clear I believe we are using the context class loader for each of the 
sets of tables, and we don’t see the jar reloaded reliably when they are 
changed. This behavior is consistent running just a 1 node stop on my local 
machine for development or on the cluster. Copying the jars into the lib/ext 
directory however always seems to pick up the change. 

These are from my dev box, but the clusters look the same just with many more 
contexts

default| general.vfs.classpaths . |
site   | general.vfs.context.classpath.rob_maanaNgram ... |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaNgram/maana-iterators-plugins-core_2.11-assembly.jar
site   | general.vfs.context.classpath.rob_maanaSearch .. |
system |@override ... | 
hdfs://localhost/user/maana/rob/iterators/maanaSearch/maana-iterators-core_2.11-1.0-SNAPSHOT-assembly.jar

And then the context is set on the table like this


default| table.classpath.context  |
table  |@override ... | 
rob_maanaNgram



And below is most of the accumulo site.xml minus the secrets and Zookeeper 
sections, but none of the iterators are in the  config classpaths.

  
tserver.memory.maps.max
2G
  


  
tserver.memory.maps.native.enabled
true
  


  
tserver.cache.data.size
4G
  


  
tserver.cache.index.size
3G
  


  
trace.token.property.password
maana
  


  
trace.user
root
  


  
tserver.sort.buffer.size
200M
  


  
tserver.walog.max.size
1G
  


  
general.classpaths

  $ACCUMULO_HOME/lib/accumulo-server.jar,
  $ACCUMULO_HOME/lib/accumulo-core.jar,
  $ACCUMULO_HOME/lib/accumulo-start.jar,
  $ACCUMULO_HOME/lib/accumulo-fate.jar,
  $ACCUMULO_HOME/lib/accumulo-proxy.jar,
  $ACCUMULO_HOME/lib/[^.].*.jar,
  $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
  $HADOOP_CONF_DIR,
  $HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
  $HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
  /usr/hdp/current/hadoop-client/[^.].*.jar,
  /usr/hdp/current/hadoop-client/lib/(?!slf4j)[^.].*.jar,
  /usr/hdp/current/hadoop-hdfs-client/[^.].*.jar,
  /usr/hdp/current/hadoop-mapreduce-client/[^.].*.jar,
  /usr/hdp/current/hadoop-yarn-client/[^.].*.jar,
  /usr/hdp/current/hadoop-yarn-client/lib/jersey.*.jar,
  $HADOOP_PREFIX/[^.].*.jar,
  $HADOOP_PREFIX/lib/(?!slf4j)[^.].*.jar,
  /usr/hdp/current/hive-client/lib/hive-accumulo-handler.jar,

Classpaths that accumulo checks for updates and class 
files.
  


  
  instance.volumes
  hdfs://maana2/apps/accumulo
  









On 10/29/15, 5:27 PM, "dlmar...@comcast.net"  wrote:

>So, without seeing your configuration, I would suggest trying something before 
>upgrading to 1.7. In 1.5 we changed the classloader so that it could load from 
>different locations. At the same time, we added the concept of classloader 
>contexts which are basically names for locations for jars. Table(s) can be 
>configured to use a classloader context allowing you to deploy server side 
>code for different applications in different locations. This new classloader 
>does "reload" jars on the classpath when they change; the same behavior with 
>the older classloader reading from lib/ext. You can read more about this 
>feature at [1].
>
>We currently depend on Commons VFS 2.0 for this feature. Some bugs have been 
>fixed and you will have a better experience if you replace the VFS jar in the 
>lib directory with a snapshot of the 2.1 release[2].
>
>[1] https://blogs.apache.org/accumulo/entry/the_accumulo_classloader
>[2] 
>https://continuum-ci.apache.org/continuum/workingCopy.action?projectId=129=Apache+Commons+VFS=dist/target
>
>
>> -Original Message-
>> From: dlmar...@comcast.net [mailto:dlmar...@comcast.net]
>> Sent: Thursday, October 29, 2015 8:04 PM
>> To: user@accumulo.apache.org
>> Subject: RE: Accumulo Iterator painful development because TS don't pick up
>> changes to Jars
>> 
>> Can you provide the relevant classpath sections of your accumulo-site.xml
>> file?
>> 
>> > -Original Message-
>> > From: Rob Povey [mailto:r...@maana.io]
>> > Sent: Thursday, October 29, 2015 8:01 PM
>> > To: user@accumulo.apache.org
>> > Subject: Accumulo Iterator painful development because TS don't pick
>> > up changes to Jars
>> >
>> > Caveat I’m still running 1.6.2 internally here, and things may have
>> > changed and I could simply “be doing it wrong”, or have missed the
>> > solution in the docs. It’s also probably not a typical use case.
>> >

Re: Transaction type query in accumulo

2015-10-30 Thread Josh Elser


Hi Shweta,

shweta.agrawal wrote:

Hi,

Is transaction type facility available in Accumulo?
I have read about transaction in accumulo which says " Accumulo
guarantees these ACID properties for a single mutation (a set of changes
for a single row) but does not provide support for atomic updates across
multiple rows"


This might be easier to reason about if you consider the Java API.

When you make a Mutation, all updates in that mutation will be applied 
or rejected.


Mutation m = new Mutation("row".getBytes());
m.put("cf", "cq1", "value1");
m.put("cf", "cq2", "value2");
batchwriter.put(m);
batchwriter.close();

In this case, Accumulo will either have 2 K/V pairs in "row" ("cf:cq1" 
=> "value1", "cf:cq2" => "value2") or no K/V pairs in "row".




In my case:
If one thread is updating the fields of a document then this document
should be locked so that other thread cannot modify that document.

I am trying to achieve this by a query through conditional mutation. I
am checking whether the particular entry exist or not then updating. But
the problem is I am doing this through 150 threads. If one thread finds
and updating particular entry then other thread should not get it.

So is this the case in conditional write?

We are achieving same thing through mongoDB by find and modify feature.

If one thread get particular document to update from conditional write
then other thread should get that particular document.


I'm not 100% sure how best to go about this. Maybe you could use a 
special column in the row to do the exclusion?


write cf:lock 
_update columns_
delete cf:lock 

Keith probably has a better suggestion :)


Please provide your inputs

Thanks
Shweta

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Transaction type query in accumulo

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Re: Accumulo Iterator painful development because TS don't pick up changes to Jars

Re: Transaction type query in accumulo

7 matches

Site Navigation

Mail list logo

Footer information