Re: Counter values become under-counted when running repair.

[Dikang Gu]

@Aleksey, sure, here is the jira:
https://issues.apache.org/jira/browse/CASSANDRA-11432

Thanks!

On Thu, Mar 24, 2016 at 5:32 PM, Aleksey Yeschenko 
wrote:

> Best open a JIRA ticket and I’ll have a look at what could be the reason.
>
> --
> AY
>
> On 24 March 2016 at 23:20:55, Dikang Gu (dikan...@gmail.com) wrote:
>
> @Aleksey, we are writing to cluster with CL = 2, and reading with CL = 1.
> And overall we have 6 copies across 3 different regions. Do you have
> comments about our setup?
>
> During the repair, the counter value become inaccurate, we are still
> playing with the repair, will keep you update with more experiments. But
> do
> you have any theory around that?
>
> Thanks a lot!
> Dikang.
>
> On Thu, Mar 24, 2016 at 11:02 AM, Aleksey Yeschenko 
> wrote:
>
> > After repair is over, does the value settle? What CLs do you write to
> your
> > counters with? What CLs are you reading with?
> >
> > --
> > AY
> >
> > On 24 March 2016 at 06:17:27, Dikang Gu (dikan...@gmail.com) wrote:
> >
> > Hello there,
> >
> > We are experimenting Counters in Cassandra 2.2.5. Our setup is that we
> > have
> > 6 nodes, across three different regions, and in each region, the
> > replication factor is 2. Basically, each nodes holds a full copy of the
> > data.
> >
> > When are doing 30k/s counter increment/decrement per node, and at the
> > meanwhile, we are double writing to our mysql tier, so that we can
> measure
> > the accuracy of C* counter, compared to mysql.
> >
> > The experiment result was great at the beginning, the counter value in
> C*
> > and mysql are very close. The difference is less than 0.1%.
> >
> > But when we start to run the repair on one node, the counter value in C*
> > become much less than the value in mysql, the difference becomes larger
> > than 1%.
> >
> > My question is that is it a known problem that the counter value will
> > become under-counted if repair is running? Should we avoid running
> repair
> > for counter tables?
> >
> > Thanks.
> >
> > --
> > Dikang
> >
> >
>
>
> --
> Dikang
>
>


-- 
Dikang

Re: Counter values become under-counted when running repair.

[Aleksey Yeschenko]

Best open a JIRA ticket and I’ll have a look at what could be the reason.

-- 
AY

On 24 March 2016 at 23:20:55, Dikang Gu (dikan...@gmail.com) wrote:

@Aleksey, we are writing to cluster with CL = 2, and reading with CL = 1.  
And overall we have 6 copies across 3 different regions. Do you have  
comments about our setup?  

During the repair, the counter value become inaccurate, we are still  
playing with the repair, will keep you update with more experiments. But do  
you have any theory around that?  

Thanks a lot!  
Dikang.  

On Thu, Mar 24, 2016 at 11:02 AM, Aleksey Yeschenko   
wrote:  

> After repair is over, does the value settle? What CLs do you write to your  
> counters with? What CLs are you reading with?  
>  
> --  
> AY  
>  
> On 24 March 2016 at 06:17:27, Dikang Gu (dikan...@gmail.com) wrote:  
>  
> Hello there,  
>  
> We are experimenting Counters in Cassandra 2.2.5. Our setup is that we  
> have  
> 6 nodes, across three different regions, and in each region, the  
> replication factor is 2. Basically, each nodes holds a full copy of the  
> data.  
>  
> When are doing 30k/s counter increment/decrement per node, and at the  
> meanwhile, we are double writing to our mysql tier, so that we can measure  
> the accuracy of C* counter, compared to mysql.  
>  
> The experiment result was great at the beginning, the counter value in C*  
> and mysql are very close. The difference is less than 0.1%.  
>  
> But when we start to run the repair on one node, the counter value in C*  
> become much less than the value in mysql, the difference becomes larger  
> than 1%.  
>  
> My question is that is it a known problem that the counter value will  
> become under-counted if repair is running? Should we avoid running repair  
> for counter tables?  
>  
> Thanks.  
>  
> --  
> Dikang  
>  
>  


--  
Dikang  

Re: Counter values become under-counted when running repair.

[Dikang Gu]

@Aleksey, we are writing to cluster with CL = 2, and reading with CL = 1.
And overall we have 6 copies across 3 different regions. Do you have
comments about our setup?

During the repair, the counter value become inaccurate, we are still
playing with the repair, will keep you update with more experiments. But do
you have any theory around that?

Thanks a lot!
Dikang.

On Thu, Mar 24, 2016 at 11:02 AM, Aleksey Yeschenko 
wrote:

> After repair is over, does the value settle? What CLs do you write to your
> counters with? What CLs are you reading with?
>
> --
> AY
>
> On 24 March 2016 at 06:17:27, Dikang Gu (dikan...@gmail.com) wrote:
>
> Hello there,
>
> We are experimenting Counters in Cassandra 2.2.5. Our setup is that we
> have
> 6 nodes, across three different regions, and in each region, the
> replication factor is 2. Basically, each nodes holds a full copy of the
> data.
>
> When are doing 30k/s counter increment/decrement per node, and at the
> meanwhile, we are double writing to our mysql tier, so that we can measure
> the accuracy of C* counter, compared to mysql.
>
> The experiment result was great at the beginning, the counter value in C*
> and mysql are very close. The difference is less than 0.1%.
>
> But when we start to run the repair on one node, the counter value in C*
> become much less than the value in mysql, the difference becomes larger
> than 1%.
>
> My question is that is it a known problem that the counter value will
> become under-counted if repair is running? Should we avoid running repair
> for counter tables?
>
> Thanks.
>
> --
> Dikang
>
>


-- 
Dikang

Re: Counter values become under-counted when running repair.

[Aleksey Yeschenko]

After repair is over, does the value settle? What CLs do you write to your 
counters with? What CLs are you reading with?

-- 
AY

On 24 March 2016 at 06:17:27, Dikang Gu (dikan...@gmail.com) wrote:

Hello there,  

We are experimenting Counters in Cassandra 2.2.5. Our setup is that we have  
6 nodes, across three different regions, and in each region, the  
replication factor is 2. Basically, each nodes holds a full copy of the  
data.  

When are doing 30k/s counter increment/decrement per node, and at the  
meanwhile, we are double writing to our mysql tier, so that we can measure  
the accuracy of C* counter, compared to mysql.  

The experiment result was great at the beginning, the counter value in C*  
and mysql are very close. The difference is less than 0.1%.  

But when we start to run the repair on one node, the counter value in C*  
become much less than the value in mysql, the difference becomes larger  
than 1%.  

My question is that is it a known problem that the counter value will  
become under-counted if repair is running? Should we avoid running repair  
for counter tables?  

Thanks.  

--  
Dikang  

Counter values become under-counted when running repair.

[Dikang Gu]

Hello there,

We are experimenting Counters in Cassandra 2.2.5. Our setup is that we have
6 nodes, across three different regions, and in each region, the
replication factor is 2. Basically, each nodes holds a full copy of the
data.

When are doing 30k/s counter increment/decrement per node, and at the
meanwhile, we are double writing to our mysql tier, so that we can measure
the accuracy of C* counter, compared to mysql.

The experiment result was great at the beginning, the counter value in C*
and mysql are very close. The difference is less than 0.1%.

But when we start to run the repair on one node, the counter value in C*
become much less than the value in mysql,  the difference becomes larger
than 1%.

My question is that is it a known problem that the counter value will
become under-counted if repair is running? Should we avoid running repair
for counter tables?

Thanks.

-- 
Dikang