Re: WARN Writing large partition for materialized views

[Robert Sicoie]

Thanks!

Robert Sicoie

On Thu, Sep 29, 2016 at 12:49 PM, Alexander Dejanovski <
a...@thelastpickle.com> wrote:

> Hi Robert,
>
> Materialized Views are regular C* tables underneath, so based on their PK
> they can generate big partitions.
> It is often advised to keep partition size under 100MB because larger
> partitions are hard to read and compact. They usually put pressure on the
> heap and lead to long GC pauses  + laggy compactions.
> You could possibly OOM while trying to fully read a partition that is way
> too big for your heap.
>
> It is indeed a schema problem and you most likely have to bucket your MV
> in order to split those partitions into smaller chunks. In the case of MV,
> you possibly need to add a bucketing field to the table it relies on (if
> you don't have one already), and add it to the MV partition key.
>
> You should try to use cassandra-stress to test your bucket sizes :
> https://docs.datastax.com/en/cassandra/3.x/cassandra/
> tools/toolsCStress.html
> In your schema definition you can now specify the creation of a MV.
>
> Cheers,
>
>
> On Wed, Sep 28, 2016 at 7:35 PM Robert Sicoie 
> wrote:
>
>> Hi guys,
>>
>> I run a cluster with 5 nodes, cassandra version 3.0.5.
>>
>> I get this warning:
>> 2016-09-28 17:22:18,480 BigTableWriter.java:171 - Writing large
>> partition...
>>
>> for some materialized view. Some have values over 500MB. How this affects
>> performance? What can/should be done? I suppose is a problem in the schema
>> design.
>>
>> Thanks,
>> Robert Sicoie
>>
> --
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>

Re: WARN Writing large partition for materialized views

[Alexander Dejanovski]

Hi Robert,

Materialized Views are regular C* tables underneath, so based on their PK
they can generate big partitions.
It is often advised to keep partition size under 100MB because larger
partitions are hard to read and compact. They usually put pressure on the
heap and lead to long GC pauses  + laggy compactions.
You could possibly OOM while trying to fully read a partition that is way
too big for your heap.

It is indeed a schema problem and you most likely have to bucket your MV in
order to split those partitions into smaller chunks. In the case of MV, you
possibly need to add a bucketing field to the table it relies on (if you
don't have one already), and add it to the MV partition key.

You should try to use cassandra-stress to test your bucket sizes :
https://docs.datastax.com/en/cassandra/3.x/cassandra/tools/toolsCStress.html
In your schema definition you can now specify the creation of a MV.

Cheers,


On Wed, Sep 28, 2016 at 7:35 PM Robert Sicoie 
wrote:

> Hi guys,
>
> I run a cluster with 5 nodes, cassandra version 3.0.5.
>
> I get this warning:
> 2016-09-28 17:22:18,480 BigTableWriter.java:171 - Writing large
> partition...
>
> for some materialized view. Some have values over 500MB. How this affects
> performance? What can/should be done? I suppose is a problem in the schema
> design.
>
> Thanks,
> Robert Sicoie
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

WARN Writing large partition for materialized views

[Robert Sicoie]

Hi guys,

I run a cluster with 5 nodes, cassandra version 3.0.5.

I get this warning:
2016-09-28 17:22:18,480 BigTableWriter.java:171 - Writing large partition...

for some materialized view. Some have values over 500MB. How this affects
performance? What can/should be done? I suppose is a problem in the schema
design.

Thanks,
Robert Sicoie