Re: Using Bucket Data Types slowed insert performance

[Alexander Sicular]

Let's talk about Riak data types for a moment. Riak data types are collectively 
implementations of what academia refer to as CRDT's (convergent or conflict 
free replicated data types.) The key benefit a CRDT offers, over a traditional 
KV by contrast, is in automatic conflict resolution. The various CRDT's 
provided in Riak have specific conflict resolution strategies. This does not 
come for free. There is a computational cost associated with CRDT's. If your 
use case requires automated conflict resolution strategies than CRDT's are a 
good fit. Internally CRDT's rely on vector clocks (see DVV's in the 
documentation) to resolve conflict. 

Considering your ETL use case I'm going to presume that your data is immutable 
(I could very well be wrong here.) If your data is immutable I would consider 
simply using a KV and not paying the CRDT computational penalty (and possibly 
even the write once bucket.) The CRDT penalty you pay is obviously subjective 
to your use case, configuration, hw deployment etc. 

Hope that helps!
-Alexander 

@siculars
http://siculars.posthaven.com

Sent from my iRotaryPhone

> On Oct 20, 2015, at 12:39, Dennis Nicolay  wrote:
> 
> Hi Alexander,
>  
> I’m parsing the file and storing each row with own key in a map datatype 
> bucket and each column is a register. 
>  
> Thanks,
> Dennis
>  
> From: Alexander Sicular [mailto:sicul...@gmail.com] 
> Sent: Tuesday, October 20, 2015 10:34 AM
> To: Dennis Nicolay
> Cc: Christopher Mancini; riak-users@lists.basho.com
> Subject: Re: Using Bucket Data Types slowed insert performance
>  
> Hi Dennis,
>  
> It's a bit unclear what you are trying to do here. Are you 1. uploading the 
> entire file and saving it to one key with the value being the file? Or are 
> you 2. parsing the file and storing each row as a register in a map? 
>  
> Either of those approaches are not appropriate in Riak KV. For the first case 
> I would point you to Riak S2 which is designed to manage large binary object 
> storage. You can keep the large file as a single addressable entity and 
> access it via Amazon S3 or Swift protocol. For the second case I would 
> consider maintaining one key (map) per row in the file and have a register 
> per column in the row. Or not use Riak data types (maps, sets, registers, 
> flags and counters) and simply keep each row in the file as a KV in Riak 
> either as a raw string or as a serialized json string. ETL'ing out of 
> relational databases and into Riak is a very common use case and often 
> implemented in the fashion I described. 
>  
> As Chris mentioned, soft upper bound on value size should be 1MB. I say soft 
> because we won't enforce it although there are settings in the config that 
> can be changed to enforce it (default 5MB warning, 50MB reject I believe.) 
> 
> Best,
> Alexander
> 
> 
> @siculars
> http://siculars.posthaven.com
>  
> Sent from my iRotaryPhone
> 
> On Oct 20, 2015, at 10:22, Christopher Mancini  wrote:
> 
> Hi Dennis,
> 
> I am not the most experienced, but what I do know is that a file that size 
> causes a great deal of network chatter because it has to handoff that data to 
> the other nodes in the network and will cause delays in Riak's ability to 
> send and confirm consistency across the ring. Typically we recommend that you 
> try to structure your objects to around 1mb or less to ensure consistent 
> performance. That max object size can vary of course based on your network / 
> server specs and configuration.
> 
> I hope this helps.
> 
> Chris
>  
> On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay  wrote:
> Hi,
>  
> I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows 
> of data into riak.  I switched to a map bucket data type with registers.   It 
> is taking about 3 times longer to insert into this bucket vs non data typed 
> bucket.  Any suggestions?
>  
> Thanks in advance,
> Dennis
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Using Bucket Data Types slowed insert performance

[Dennis Nicolay]

 

  ResultObject cdr;

while (queued.TryDequeue(out cdr))

{

long beforeProcessing = DateTime.Now.Ticks;

UpdateMap.Builder builder = BuildMapObject(bucket, cdr);

UpdateMap cmd = builder.Build();

RiakResult rslt = client.Execute(cmd); 

 

 

 

 

private static UpdateMap.Builder BuildMapObject(string bucketname, ResultObject 
cdr )

{

  

var builder = new UpdateMap.Builder()

   .WithBucketType("maps")

   .WithBucket(bucketname)

   .WithKey(cdr.CdrKey);   

var mapOperation = new UpdateMap.MapOperation();

mapOperation.SetRegister("FileTimeStamp", 
cdr.CdrValue.FileTimeStamp.ToString());

mapOperation.SetRegister("AuditId", 
cdr.CdrValue.AuditId.ToString());

mapOperation.SetRegister("CdrId", cdr.CdrValue.CdrId.ToString());

mapOperation.SetRegister("IsBillable", 
cdr.CdrValue.IsBillable.ToString());

mapOperation.SetRegister("SwitchId", 
cdr.CdrValue.SwitchId.ToString());

mapOperation.SetRegister("SwitchDescription", 
cdr.CdrValue.SwitchDescription.ToString());

mapOperation.SetRegister("SequenceNumber", 
cdr.CdrValue.SequenceNumber.ToString());

mapOperation.SetRegister("CallDirection", 
cdr.CdrValue.CallDirection.ToString());

mapOperation.SetRegister("CallTypeId", 
cdr.CdrValue.CallTypeId.ToString());

mapOperation.SetRegister("Partition", 
cdr.CdrValue.Partition.ToString());

mapOperation.SetRegister("CustomerTrunkId", 
cdr.CdrValue.CustomerTrunkId.ToString());

mapOperation.SetRegister("OrigIpAddress", 
cdr.CdrValue.OrigIpAddress.ToString());

mapOperation.SetRegister("OrigPort", 
cdr.CdrValue.OrigPort.ToString());

mapOperation.SetRegister("SupplierTrunkId", 
cdr.CdrValue.SupplierTrunkId.ToString());

mapOperation.SetRegister("TermIpAddress", 
cdr.CdrValue.TermIpAddress.ToString());

mapOperation.SetRegister("TermPort", 
cdr.CdrValue.TermPort.ToString());

mapOperation.SetRegister("Ani", cdr.CdrValue.Ani.ToString());

mapOperation.SetRegister("OutpulseNumber", 
cdr.CdrValue.OutpulseNumber.ToString());

mapOperation.SetRegister("SubscriberNumber", 
cdr.CdrValue.SupplierTrunkId.ToString());

mapOperation.SetRegister("CallingNoa", 
cdr.CdrValue.CallingNoa.ToString());

mapOperation.SetRegister("DialedNoa", 
cdr.CdrValue.DialedNoa.ToString());

mapOperation.SetRegister("OutpulseNoa", 
cdr.CdrValue.OutpulseNumber.ToString());

mapOperation.SetRegister("TreatmentCode", 
cdr.CdrValue.TreatmentCode.ToString());

mapOperation.SetRegister("CompletionCode", 
cdr.CdrValue.CompletionCode.ToString());

mapOperation.SetRegister("CustomerName", 
cdr.CdrValue.CustomerName.ToString());

mapOperation.SetRegister("CustId", cdr.CdrValue.CustId.ToString());

mapOperation.SetRegister("CustContractId", 
cdr.CdrValue.CustContractId.ToString());

mapOperation.SetRegister("CustCountryCode", 
cdr.CdrValue.CustCountryCode.ToString());

mapOperation.SetRegister("CustDuration", 
cdr.CdrValue.CustDuration.ToString());

mapOperation.SetRegister("Price", cdr.CdrValue.Price.ToString());

mapOperation.SetRegister("BasePrice", 
cdr.CdrValue.BasePrice.ToString());

mapOperation.SetRegister("BillingDestinationName", 
cdr.CdrValue.BillingDestinationName.ToString());

mapOperation.SetRegister("BillingGroupId", 
cdr.CdrValue.BillingGroupId.ToString());

mapOperation.SetRegister("SupplierName", 
cdr.CdrValue.SupplierName.ToString());

mapOperation.SetRegister("SuppId", cdr.CdrValue.SuppId.ToString());

mapOperation.SetRegister("SuppContractId", 
cdr.CdrValue.SuppContractId.ToString());

mapOperation.SetRegister("SuppCountryCode", 
cdr.CdrValue.SuppCountryCode.ToString());

mapOperation.SetRegister("SuppDuration", 
cdr.CdrValue.SuppDuration.ToString());

mapOperation.SetRegister("Cost", cdr.CdrValue.Cost.ToString());

mapOperation.SetRegister("BaseCost", 
cdr.CdrValue.BaseCost.ToString());

mapOperation.SetRegister("RoutingDestinationName", 
cdr.CdrValue.RoutingDestinationName.ToString());

mapOperation.SetRegister("RoutingGroupId", 
cdr.CdrValue.RoutingGroupId.ToString());

mapOperation.SetRegister("RouteToCountryCode", 
cdr.CdrValue.RouteToCountryCode.ToString());

mapOperation.SetRegister("Pdd", cdr.CdrValue.Pdd.ToString());

mapOperation.SetRegister("RealDuration", 
cdr.CdrValue.RealDuration.ToString());

mapOperation.SetRegister("StartTime", 

Re: Using Bucket Data Types slowed insert performance

[Alexander Sicular]

Hi Dennis,

It's a bit unclear what you are trying to do here. Are you 1. uploading the 
entire file and saving it to one key with the value being the file? Or are you 
2. parsing the file and storing each row as a register in a map? 

Either of those approaches are not appropriate in Riak KV. For the first case I 
would point you to Riak S2 which is designed to manage large binary object 
storage. You can keep the large file as a single addressable entity and access 
it via Amazon S3 or Swift protocol. For the second case I would consider 
maintaining one key (map) per row in the file and have a register per column in 
the row. Or not use Riak data types (maps, sets, registers, flags and counters) 
and simply keep each row in the file as a KV in Riak either as a raw string or 
as a serialized json string. ETL'ing out of relational databases and into Riak 
is a very common use case and often implemented in the fashion I described. 

As Chris mentioned, soft upper bound on value size should be 1MB. I say soft 
because we won't enforce it although there are settings in the config that can 
be changed to enforce it (default 5MB warning, 50MB reject I believe.) 

Best,
Alexander

@siculars
http://siculars.posthaven.com

Sent from my iRotaryPhone

> On Oct 20, 2015, at 10:22, Christopher Mancini  wrote:
> 
> Hi Dennis,
> 
> I am not the most experienced, but what I do know is that a file that size 
> causes a great deal of network chatter because it has to handoff that data to 
> the other nodes in the network and will cause delays in Riak's ability to 
> send and confirm consistency across the ring. Typically we recommend that you 
> try to structure your objects to around 1mb or less to ensure consistent 
> performance. That max object size can vary of course based on your network / 
> server specs and configuration.
> 
> I hope this helps.
> 
> Chris
> 
>> On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay  wrote:
>> Hi,
>> 
>>  
>> 
>> I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows 
>> of data into riak.  I switched to a map bucket data type with registers.   
>> It is taking about 3 times longer to insert into this bucket vs non data 
>> typed bucket.  Any suggestions?
>> 
>>  
>> 
>> Thanks in advance,
>> 
>> Dennis
>> 
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Using Bucket Data Types slowed insert performance

[Dennis Nicolay]

Hi Alexander,

 

I’m parsing the file and storing each row with own key in a map datatype bucket 
and each column is a register.  

 

Thanks,

Dennis

 

From: Alexander Sicular [mailto:sicul...@gmail.com] 
Sent: Tuesday, October 20, 2015 10:34 AM
To: Dennis Nicolay
Cc: Christopher Mancini; riak-users@lists.basho.com
Subject: Re: Using Bucket Data Types slowed insert performance

 

Hi Dennis,

 

It's a bit unclear what you are trying to do here. Are you 1. uploading the 
entire file and saving it to one key with the value being the file? Or are you 
2. parsing the file and storing each row as a register in a map? 

 

Either of those approaches are not appropriate in Riak KV. For the first case I 
would point you to Riak S2 which is designed to manage large binary object 
storage. You can keep the large file as a single addressable entity and access 
it via Amazon S3 or Swift protocol. For the second case I would consider 
maintaining one key (map) per row in the file and have a register per column in 
the row. Or not use Riak data types (maps, sets, registers, flags and counters) 
and simply keep each row in the file as a KV in Riak either as a raw string or 
as a serialized json string. ETL'ing out of relational databases and into Riak 
is a very common use case and often implemented in the fashion I described. 

 

As Chris mentioned, soft upper bound on value size should be 1MB. I say soft 
because we won't enforce it although there are settings in the config that can 
be changed to enforce it (default 5MB warning, 50MB reject I believe.) 

Best,

Alexander





@siculars

http://siculars.posthaven.com

 

Sent from my iRotaryPhone


On Oct 20, 2015, at 10:22, Christopher Mancini  > wrote:

Hi Dennis,

I am not the most experienced, but what I do know is that a file that size 
causes a great deal of network chatter because it has to handoff that data to 
the other nodes in the network and will cause delays in Riak's ability to send 
and confirm consistency across the ring. Typically we recommend that you try to 
structure your objects to around 1mb or less to ensure consistent performance. 
That max object size can vary of course based on your network / server specs 
and configuration.

I hope this helps.

Chris

 

On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay  > wrote:

Hi,

 

I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows of 
data into riak.  I switched to a map bucket data type with registers.   It is 
taking about 3 times longer to insert into this bucket vs non data typed 
bucket.  Any suggestions?

 

Thanks in advance,

Dennis

___
riak-users mailing list
riak-users@lists.basho.com  
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com  
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

RE: Using Bucket Data Types slowed insert performance

[Mark Schmidt]

Hi folks, sorry for the confusion.

 

Our scenario is as follows:

 

We have a 6 node development cluster running on its own network segment using 
HAProxy to facilitate load-balancing across the nodes. A single Riak-dot-NET 
client service is performing the insert operations from dedicated hardware 
located within the same network segment. We have basic network throughput 
capabilities of 100 Mbit with an average speed achievable of 75 Mbit.

 

The data we are attempting to insert is composed of phone call record receipts 
from telephone carriers. These records are batched and written to a flat file 
for incorporation into our reporting engine. 1) Our Riak client process takes a 
flat file (In this case, a 40MB collection of records, each record being 
approximately 5k in size) and parses the entire file so each record can be 
added to a local .NET queue.

2) Once the entire file has been parsed and each record loaded into the local 
queue, 20 threads are spawned and connections are opened to our Riak nodes via 
the HAProxy.

3) Each thread will pull a 5k record from the queue on a first come first 
served basis and perform a put to the Riak environment.

 

When first testing our client insert process, we were pushing the 5K records as 
whole strings into the Riak environment. Network throughput topped out at 
around 80 Mbits with a total load time of 90 seconds for 149k records. When the 
client process was modified (same queuing and de-queuing methods) so that a map 
datatype bucket would be created and keys stored as registers, we saw network 
throughput drop to around 10 Mbit with total upload time increase to around 270 
seconds for the 149k records.  

 

It appears as though we’ve either encountered a potential bottleneck unrelated 
to network throughput, or we’re just seeing an expected processing penalty for 
our use of Riak datatypes. Please note, we’re configuring Zabbix so we can 
monitor disk IO on each node as processor and memory resources don’t appear to 
be the culprit either.

 

If the reduction in processing speed is a natural consequence to utilizing Riak 
data types, is the inter-node network the optimum place to increase resources? 
Our eventual datacenter implementation will support speeds of over 40 Gbit for 
inter-node communication. We’re just trying to identify which levers from an 
operational standpoint we can throw to boost performance, or if our client 
implementation is suspect. 

 

You bring up some excellent points regarding our use of CRDTs. In our case, the 
call data records are mutable as they are subject to changes by phone carriers 
for billing error corrections, incorrect data and a host of other reasons. We 
may be better served by treating the records as immutable and performing wide 
scale record removal and “reprocessing” in the event changes to existing 
records are received/requested.

 

Thank you,

 

Mark Schmidt

 

From: Alexander Sicular [mailto:sicul...@gmail.com] 
Sent: Tuesday, October 20, 2015 10:55 AM
To: Dennis Nicolay 
Cc: Christopher Mancini ; riak-users@lists.basho.com; Mark 
Schmidt 
Subject: Re: Using Bucket Data Types slowed insert performance

 

Let's talk about Riak data types for a moment. Riak data types are collectively 
implementations of what academia refer to as CRDT's (convergent or conflict 
free replicated data types.) The key benefit a CRDT offers, over a traditional 
KV by contrast, is in automatic conflict resolution. The various CRDT's 
provided in Riak have specific conflict resolution strategies. This does not 
come for free. There is a computational cost associated with CRDT's. If your 
use case requires automated conflict resolution strategies than CRDT's are a 
good fit. Internally CRDT's rely on vector clocks (see DVV's in the 
documentation) to resolve conflict. 

 

Considering your ETL use case I'm going to presume that your data is immutable 
(I could very well be wrong here.) If your data is immutable I would consider 
simply using a KV and not paying the CRDT computational penalty (and possibly 
even the write once bucket.) The CRDT penalty you pay is obviously subjective 
to your use case, configuration, hw deployment etc. 

 

Hope that helps!
-Alexander 





@siculars

http://siculars.posthaven.com

 

Sent from my iRotaryPhone


On Oct 20, 2015, at 12:39, Dennis Nicolay  > wrote:

Hi Alexander,

 

I’m parsing the file and storing each row with own key in a map datatype bucket 
and each column is a register.  

 

Thanks,

Dennis

 

From: Alexander Sicular [mailto:sicul...@gmail.com] 
Sent: Tuesday, October 20, 2015 10:34 AM
To: Dennis Nicolay
Cc: Christopher Mancini; riak-users@lists.basho.com 
 
Subject: Re: Using Bucket Data Types slowed insert performance

 

Hi Dennis,

 

It's a bit unclear what you are trying to do here. Are you 1. uploading 

Re: Using Bucket Data Types slowed insert performance

[Christopher Mancini]

Hi Dennis,

I am not the most experienced, but what I do know is that a file that size
causes a great deal of network chatter because it has to handoff that data
to the other nodes in the network and will cause delays in Riak's ability
to send and confirm consistency across the ring. Typically we recommend
that you try to structure your objects to around 1mb or less to ensure
consistent performance. That max object size can vary of course based on
your network / server specs and configuration.

I hope this helps.

Chris

On Tue, Oct 20, 2015 at 8:18 AM Dennis Nicolay 
wrote:

> Hi,
>
>
>
> I’m using .net RiakClient 2.0 to insert a 44mb delimited file with 139k
> rows of data into riak.  I switched to a map bucket data type with
> registers.   It is taking about 3 times longer to insert into this bucket
> vs non data typed bucket.  Any suggestions?
>
>
>
> Thanks in advance,
>
> Dennis
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak S2 2.1 is available

[Matthew Brender]

Hello all,

Riak S2 2.1 (aka Riak CS) is available. Thanks to all who made it happen!

Here are the highlights of the release:
* New metrics that enable you to determine the health of your Riak S2
system, as well as get reports on your storage utilization per bucket
or user.
* Garbage collection refinements
* A module-level hook point for limiting user access and quota usage
* General availability of multi-bag, now known as supercluster

Links:
Riak S2 (Riak CS) 2.1.0 Release Notes [1]
Riak open source downloads [2]
Riak S2 Enterprise downloads (Enterprise customers only) [3]

[1] 
http://docs.basho.com/riakcs/latest/cookbooks/Riak-CS-Release-Notes/#Riak-S2-Riak-CS-2-1-0-Release-Notes
[2] http://docs.basho.com/riakcs/latest/riakcs-downloads/
[3] https://basho.zendesk.com/entries/105842243

Best,
Matt Brender
Developer Advocate

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Using Bucket Data Types slowed insert performance

[Dennis Nicolay]

Hi,

 

I'm using .net RiakClient 2.0 to insert a 44mb delimited file with 139k rows
of data into riak.  I switched to a map bucket data type with registers.
It is taking about 3 times longer to insert into this bucket vs non data
typed bucket.  Any suggestions?

 

Thanks in advance,

Dennis

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Riak in Docker - Error folding keys - incomplete_hint

[Toby Corkindale]

 Hi,
I've been working on getting Riak to run inside Docker containers - in a
multi-machine cluster. (Previous work I've seen has only run Riak as a
cluster all on the same machine.)
I thought I had it cracked, although I tripped up on the existing issue
with Riak and lockfiles[1]. But the nodes have been generating an awful lot
of errors like the below, and I wondered if anyone here can give me an
explanation? (And, is it a problem?)

2015-10-21 01:19:23.567 [error] <0.24495.0> Error folding keys for
"/var/lib/riak/bitcask.1h/2283596
30832953580969325755111919221821239459840/2.bitcask.data":
{incomplete_hint,4}

1: Related issues to the lockfiles --
I note that many are closed, but the problem still exists, and is
particularly triggered by using Docker and stopping/killing Riak more
violently than it likes.
https://github.com/basho/bitcask/issues/163 (closed)
https://github.com/basho/riak/issues/535 (open)
https://github.com/basho/bitcask/issues/167 (closed)
https://github.com/basho/bitcask/issues/99 (closed)
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: Using Bucket Data Types slowed insert performance

[Alex Moore]

Hi Dennis & Mark,

I noticed some timing code in your snippets:

 long beforeProcessing = DateTime.Now.Ticks;

Do you have any numbers on what an individual operation for KV vs CRDTs
looks like on your system? (Mean, percentiles if possible)
Also, how big are your KV objects?

CRDTs will take extra processing on Riak's side, so I'm wondering if you're
getting limited by a longer RTT + your 20 threads.  One easy thing to try
would be to double the thread pool (and connections) and see if that shaves
off any overall time by overlapping the time we're waiting for Riak to
respond.

If it doesn't, then we can look in other directions :)

Thanks,
Alex


On Tue, Oct 20, 2015 at 3:25 PM, Dennis Nicolay 
wrote:

>
>
>   ResultObject cdr;
>
> while (queued.TryDequeue(out cdr))
>
> {
>
> long beforeProcessing = DateTime.Now.Ticks;
>
> UpdateMap.Builder builder = BuildMapObject(bucket,
> cdr);
>
> UpdateMap cmd = builder.Build();
>
> RiakResult rslt = client.Execute(cmd);
>
>
>
>
>
>
>
>
>
> *private static UpdateMap.Builder BuildMapObject(string bucketname,
> ResultObject cdr )*
>
> *{*
>
>
>
> *var builder = new UpdateMap.Builder()*
>
> *   .WithBucketType("maps")*
>
> *   .WithBucket(bucketname)*
>
> *   .WithKey(cdr.CdrKey);   *
>
> *var mapOperation = new UpdateMap.MapOperation();*
>
> *mapOperation.SetRegister("FileTimeStamp",
> cdr.CdrValue.FileTimeStamp.ToString());*
>
> *mapOperation.SetRegister("AuditId",
> cdr.CdrValue.AuditId.ToString());*
>
> *mapOperation.SetRegister("CdrId",
> cdr.CdrValue.CdrId.ToString());*
>
> *mapOperation.SetRegister("IsBillable",
> cdr.CdrValue.IsBillable.ToString());*
>
> *mapOperation.SetRegister("SwitchId",
> cdr.CdrValue.SwitchId.ToString());*
>
> *mapOperation.SetRegister("SwitchDescription",
> cdr.CdrValue.SwitchDescription.ToString());*
>
> *mapOperation.SetRegister("SequenceNumber",
> cdr.CdrValue.SequenceNumber.ToString());*
>
> *mapOperation.SetRegister("CallDirection",
> cdr.CdrValue.CallDirection.ToString());*
>
> *mapOperation.SetRegister("CallTypeId",
> cdr.CdrValue.CallTypeId.ToString());*
>
> *mapOperation.SetRegister("Partition",
> cdr.CdrValue.Partition.ToString());*
>
> *mapOperation.SetRegister("CustomerTrunkId",
> cdr.CdrValue.CustomerTrunkId.ToString());*
>
> *mapOperation.SetRegister("OrigIpAddress",
> cdr.CdrValue.OrigIpAddress.ToString());*
>
> *mapOperation.SetRegister("OrigPort",
> cdr.CdrValue.OrigPort.ToString());*
>
> *mapOperation.SetRegister("SupplierTrunkId",
> cdr.CdrValue.SupplierTrunkId.ToString());*
>
> *mapOperation.SetRegister("TermIpAddress",
> cdr.CdrValue.TermIpAddress.ToString());*
>
> *mapOperation.SetRegister("TermPort",
> cdr.CdrValue.TermPort.ToString());*
>
> *mapOperation.SetRegister("Ani", cdr.CdrValue.Ani.ToString());*
>
> *mapOperation.SetRegister("OutpulseNumber",
> cdr.CdrValue.OutpulseNumber.ToString());*
>
> *mapOperation.SetRegister("SubscriberNumber",
> cdr.CdrValue.SupplierTrunkId.ToString());*
>
> *mapOperation.SetRegister("CallingNoa",
> cdr.CdrValue.CallingNoa.ToString());*
>
> *mapOperation.SetRegister("DialedNoa",
> cdr.CdrValue.DialedNoa.ToString());*
>
> *mapOperation.SetRegister("OutpulseNoa",
> cdr.CdrValue.OutpulseNumber.ToString());*
>
> *mapOperation.SetRegister("TreatmentCode",
> cdr.CdrValue.TreatmentCode.ToString());*
>
> *mapOperation.SetRegister("CompletionCode",
> cdr.CdrValue.CompletionCode.ToString());*
>
> *mapOperation.SetRegister("CustomerName",
> cdr.CdrValue.CustomerName.ToString());*
>
> *mapOperation.SetRegister("CustId",
> cdr.CdrValue.CustId.ToString());*
>
> *mapOperation.SetRegister("CustContractId",
> cdr.CdrValue.CustContractId.ToString());*
>
> *mapOperation.SetRegister("CustCountryCode",
> cdr.CdrValue.CustCountryCode.ToString());*
>
> *mapOperation.SetRegister("CustDuration",
> cdr.CdrValue.CustDuration.ToString());*
>
> *mapOperation.SetRegister("Price",
> cdr.CdrValue.Price.ToString());*
>
> *mapOperation.SetRegister("BasePrice",
> cdr.CdrValue.BasePrice.ToString());*
>
> *mapOperation.SetRegister("BillingDestinationName",
> cdr.CdrValue.BillingDestinationName.ToString());*
>
> *mapOperation.SetRegister("BillingGroupId",
> cdr.CdrValue.BillingGroupId.ToString());*
>
> *mapOperation.SetRegister("SupplierName",
> cdr.CdrValue.SupplierName.ToString());*
>
> *mapOperation.SetRegister("SuppId",
> cdr.CdrValue.SuppId.ToString());*
>
> *

counter crdt in riak

[David Byron]

Apologies if this is basic question, or one that's already been 
answered.  I've done a fair amount of digging (e.g. 
http://thread.gmane.org/gmane.comp.db.riak.user/12808/focus=12816), but 
I'm still confused about the documentation regarding conflict resolution 
for riak's crdt counter.  Specifically this: 
https://github.com/basho/basho_docs/blame/riak/2.1.1/source/languages/en/riak/theory/concepts/crdts.md#L238 
(using the blame view to get a line number reference in markdown), which 
says:


"Counters | Each actor keeps an independent count for increments and 
decrements; upon merge, the pairwise maximum of the counts for each 
actor will win (e.g. if one count for an actor holds 172 and the other 
holds 173, 173 will win upon merge)"


This makes it sound to me like an increment from one of the actors gets 
dropped.  Fingers crossed this isn't actually what happens.


I think this is saying the same thing that the cdrt paper (section 3.1.2 
State-based increment-only Counter (G-Counter) of 
https://hal.inria.fr/file/index/docid/88/filename/techreport.pdf) 
says -- that increments that happen during a partition eventually 
resolve correctly.  I could use a more step-by-step illustration though 
-- pairwise maximum of the counts for each actor is a little dense.


Apologies for ugly formatting, but an example like this would make more 
sense to me.


- actor a increments counter
  a {a: 1}   b {}

- actor b increments counter
  a {a: 1}   b {b: 1}

- a merges b's info
  a {a: 1, b: 1} b {b: 1}

- b merges a's info
  a:{a: 1, b: 1} b {a: 1, b: 1}

so eventually this all resolves such that a and b both increment the 
counter by 2, right?  Is this how riak's crdt counter works?  If so I'll 
breathe a sigh of relief and make a pull request against the docs if you 
like.


Thanks much for your help.

-DB

___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: ERROR: S3 error: 403 (AccessDenied): Access Denied with s3cmd

[Devin Butterfield]

Sorry, please disregard. After careful double-checking I found an error in my 
stanchion.conf. Fixed, restarted and now all is working great.

Thanks for the awesome work on risk-cs/s2!
—
Regards, Devin 

> On Oct 20, 2015, at 10:21 PM, Devin Butterfield  
> wrote:
> 
> Hi,
> 
> I’ve setup risk-cs (2.1.0) on a linux (ubuntu 15.04) box to start learning 
> and followed the Riak Fast Track guide. I am trying to use s3cmd (version 
> 1.5.0-rc1) to interact with riak-cs in a proxy configuration (not direct).
> 
> However, any attempt to create a bucket fails with 403. My s3cmd config looks 
> like:
> 
> [default]
> access_key = IIQHN7RPFVCIGO6AWUPC
> secret_key = E60Y6vojRnZXMZRILe-Dw3V7azVnE9IjI6XaeQ==
> bucket_location = US
> cloudfront_host = cloudfront.amazonaws.com
> cloudfront_resource = /2010-07-15/distribution
> default_mime_type = binary/octet-stream
> delete_removed = False
> dry_run = False
> enable_multipart = False
> encoding = UTF-8
> encrypt = False
> follow_symlinks = False
> force = False
> get_continue = False
> gpg_command = /usr/local/bin/gpg
> gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes 
> --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
> gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes 
> --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
> gpg_passphrase = password
> guess_mime_type = True
> host_base = s3.amazonaws.com
> host_bucket = %(bucket)s.s3.amazonaws.com
> human_readable_sizes = False
> list_md5 = False
> log_target_prefix =
> preserve_attrs = True
> progress_meter = True
> proxy_host = localhost
> proxy_port = 8080
> recursive = False
> recv_chunk = 4096
> reduced_redundancy = False
> send_chunk = 4096
> simpledb_host = sdb.amazonaws.com
> skip_existing = False
> socket_timeout = 300
> urlencoding_mode = normal
> use_https = False
> verbosity = WARNING
> signature_v2 = True
> 
> I have riak, stanchion, and risk-cs configured per the Fast Track guide, and 
> all are up and running (can ping, etc.).
> 
> Any ideas how I might figure out what the issue is?
> 
> Thanks!
> —
> Regards, Devin


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com

Re: counter crdt in riak

[David Byron]

On 10/20/15 9:26 PM, xia...@xiaclo.net wrote:

This gives a little more detail about each of the CRDT types:
https://github.com/basho/riak/issues/354


Thanks much.  That link is very helpful.


Your assumptions are correct.  Within Riak, an actor is a vnode, and
counters behave as G-Counters.

Keep in mind that Riak counters store both an increment and decrement
for each actor, which allows decrements and negative counters as well
while preserving the accuracy of G-Counters.


I'd read in 
https://github.com/basho/riak/blob/develop/releasenotes/riak-1.4.md#pn-counters 
that riak's counters were PN...which I think is what you're 
saying...except the part about them behaving as G-Counters.


Sorry if I'm a little slow on the uptake here, and thanks again for your 
help.


-DB


Hope this helps,
Jason

From: David Byron >
Date: Wed Oct 21 2015 15:03:04 GMT+1100 (AUS Eastern Summer Time)
To: riak-users@lists.basho.com >
Subject: counter crdt in riak

Apologies if this is basic question, or one that's already been
answered.  I've done a fair amount of digging (e.g.
http://thread.gmane.org/gmane.comp.db.riak.user/12808/focus=12816), but
  I'm still confused about the documentation regarding conflict
resolution  for riak's crdt counter.  Specifically this:
https://github.com/basho/basho_docs/blame/riak/2.1.1/source/languages/en/riak/theory/concepts/crdts.md#L238
  (using the blame view to get a line number reference in markdown),
which  says:

"Counters | Each actor keeps an independent count for increments and
decrements; upon merge, the pairwise maximum of the counts for each
actor will win (e.g. if one count for an actor holds 172 and the other
holds 173, 173 will win upon merge)"

This makes it sound to me like an increment from one of the actors gets
  dropped.  Fingers crossed this isn't actually what happens.

I think this is saying the same thing that the cdrt paper (section 3.1.2
  State-based increment-only Counter (G-Counter) of
https://hal.inria.fr/file/index/docid/88/filename/techreport.pdf)
says -- that increments that happen during a partition eventually
resolve correctly.  I could use a more step-by-step illustration though
  -- pairwise maximum of the counts for each actor is a little dense.

Apologies for ugly formatting, but an example like this would make more
  sense to me.

- actor a increments counter
a {a: 1}   b {}

- actor b increments counter
a {a: 1}   b {b: 1}

- a merges b's info
a {a: 1, b: 1} b {b: 1}

- b merges a's info
a:{a: 1, b: 1} b {a: 1, b: 1}

so eventually this all resolves such that a and b both increment the
counter by 2, right?  Is this how riak's crdt counter works?  If so I'll
  breathe a sigh of relief and make a pull request against the docs if
you  like.

Thanks much for your help.

-DB


___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com