Re: Reason for Trace Message Drop

[Varun Barala]

Thanks Eric Stevens for your detailed reply!!

I got your points.


On Thu, Jun 16, 2016 at 11:49 PM, Eric Stevens  wrote:

> Are you executing all queries with tracing enabled?  If so that introduces
> overhead you probably don't want.  Most people probably don't see this log
> very often because it's the exception to query with tracing enabled, and
> not the rule (it's a diagnostic thing usually turned on only when
> troubleshooting another problem).
>
> The trace messages being dropped aren't in-and-of-themselves a problem, it
> just means that your traces would be incomplete.  However this indicates a
> cluster that's either unhealthy or getting close to it.
>
> Since you have a single node, run nodetool tpstats (if you had a cluster
> you'd want to do this on each node).  Look at the bottom section, ideally
> it looks like this:
> Message type   Dropped
> READ 0
> RANGE_SLICE  0
> _TRACE   0
> MUTATION 0
> COUNTER_MUTATION 0
> BINARY   0
> REQUEST_RESPONSE 0
> PAGED_RANGE  0
> READ_REPAIR  0
>
> If you're seeing MUTATIONs or COUNTER_MUTATIONs dropped, your data
> integrity is at risk, you need to reduce pressure in some way, as your
> cluster is falling behind.  That might be expanding your node count
> *above* your RF, or it might be tuning your application to reduce read or
> write pressure (note that expanding a cluster involves read pressure on the
> existing nodes, so to successfully expand your cluster you probably *also* 
> need
> to reduce pressure from the application, because the act of growing your
> cluster will cause you to get even further behind).
>
> On a single node, like yours, mutations dropped is data lost.  On a
> cluster with RF > 1, there's just a *chance* that you have lost data (the
> same mutation needs to be dropped by all replicas for it to be
> irrecoverably lost).  Dropped mutations are always bad.
>
> In this scenario you probably also want to double check nodetool
> compactionstats, and ideally you have < 4 or 5 compactions.  Lower is
> always better, 4 or 5 is getting worrying if it remains in that range.  If
> you're above 10, you're falling behind on compaction too, and your read
> performance will be suffering.
>
> Finally RE your JVM settings, with a 30GB node, I think you could turn
> that up a bit if you convert to G1GC.  JVM tuning is definitely not my
> strong point, there are others on this list who will be able to help you do
> a better job of it.  If you're seeing big GC pauses, then you do need to
> work on this, even if they're not the direct cause of the dropped traces.
> With that column family count you'll be under more GC pressure than you
> would be with a lower CF count (there is a fixed memory cost per CF).
> Reconsider your data model, usually this many column families suggests
> dynamically creating CF's (eg to solve multi-tenancy).  If your CF count
> will grow steadily over time at any appreciable rate, that's an
> anti-pattern.
>
> On Thu, Jun 16, 2016 at 2:40 AM Varun Barala 
> wrote:
>
>> Thanks!! Eric Stevens for your reply.
>>
>> We have following JVM settings :-
>> -
>> *memtable_offheap_space_in_mb: 15360  (*found in casandra.yaml
>> *)*
>> *MAX_HEAP_SIZE="16G"  (*found in cassandra-env.sh
>> *)*-
>>
>> And I also found big GC in log. But messages and big GC were logged at
>> different-different time in system.log. I was expecting to happen them at
>> same time after reading your reply. I also manually triggered GC but
>> messages were not dropped.
>>
>> is *TRACE message drop *harmful or it's okay we can neglect them?
>>
>> Thank you!!
>>
>>
>> On Wed, Jun 15, 2016 at 8:45 PM, Eric Stevens  wrote:
>>
>>> This is better kept to the User groups.
>>>
>>> What are your JVM memory settings for Cassandra, and have you seen big
>>> GC's in your logs?
>>>
>>> The reason I ask is because that's a large number of column families,
>>> which produces memory pressure, and at first blush that strikes me as a
>>> likely cause.
>>>
>>>
>>> On Wed, Jun 15, 2016 at 3:23 AM Varun Barala 
>>> wrote:
>>>
 Hi all,

 Can anyone tell me that what are all possible reasons for below log:-


 *"INFO  [ScheduledTasks:1] 2016-06-14 06:27:39,498
 MessagingService.java:929 - _TRACE messages were dropped in last 5000 ms:
 928 for internal timeout and 0 for cross node timeout".*
 I searched online for the same and found some reasons like:-

 * Disk is not able to keep up with your ingest
 * Resources are not able to support all parallel running tasks
 * If other nodes are down then due to large hint replay
 * Heavy workload

 But in this case other kind of messages 

Re: Multi-DC Cluster w/ non-replicated Keyspace

[Jason J. W. Williams]

It was my mistake. Before I checked the trace I assume it meant the data
also had been replicated to the remote cluster, which is why it could
answer the request. Thank you for responding so quickly and helping correct
my misunderstanding. As long as the data isn't being replicated, everything
else is fine.

-J

On Thu, Jun 16, 2016 at 8:05 PM, Ben Slater 
wrote:

> That’s the behaviour I would have expected. I’m not aware of anyway to
> prevent this and would be surprised if there is one (but I’ve never tried
> to find one either so it might be possible).
>
> Cheers
> Ben
>
> On Fri, 17 Jun 2016 at 12:02 Jason J. W. Williams <
> jasonjwwilli...@gmail.com> wrote:
>
>> Hey Ben,
>>
>> Looks like just the schema. I was surprised that running SELECTs against
>> the DC which should not have any data (because it's not specified in
>> NetworkTopologyStrategy), actually returned data. But looking at the query
>> trace it looks like its forwarding the queries to the other DC.
>>
>> -J
>>
>> On Thu, Jun 16, 2016 at 7:55 PM, Ben Slater 
>> wrote:
>>
>>> Do you mean the data is getting replicated or just the schema?
>>>
>>> On Fri, 17 Jun 2016 at 11:48 Jason J. W. Williams <
>>> jasonjwwilli...@gmail.com> wrote:
>>>
 Hi Guys,

 We have a 2 DC cluster where the keyspaces are replicated between the
 2. Is it possible to add a keyspace to one of the DCs that won't be
 replicated to the other?

 Whenever we add a new keyspace it seems to get replicated even if we
 don't specify the other DC in the keyspace's NetworkTopologyStrategy.

 -J

>>> --
>>> 
>>> Ben Slater
>>> Chief Product Officer
>>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>>> +61 437 929 798
>>>
>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>

Re: Multi-DC Cluster w/ non-replicated Keyspace

[Ben Slater]

That’s the behaviour I would have expected. I’m not aware of anyway to
prevent this and would be surprised if there is one (but I’ve never tried
to find one either so it might be possible).

Cheers
Ben

On Fri, 17 Jun 2016 at 12:02 Jason J. W. Williams 
wrote:

> Hey Ben,
>
> Looks like just the schema. I was surprised that running SELECTs against
> the DC which should not have any data (because it's not specified in
> NetworkTopologyStrategy), actually returned data. But looking at the query
> trace it looks like its forwarding the queries to the other DC.
>
> -J
>
> On Thu, Jun 16, 2016 at 7:55 PM, Ben Slater 
> wrote:
>
>> Do you mean the data is getting replicated or just the schema?
>>
>> On Fri, 17 Jun 2016 at 11:48 Jason J. W. Williams <
>> jasonjwwilli...@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> We have a 2 DC cluster where the keyspaces are replicated between the 2.
>>> Is it possible to add a keyspace to one of the DCs that won't be replicated
>>> to the other?
>>>
>>> Whenever we add a new keyspace it seems to get replicated even if we
>>> don't specify the other DC in the keyspace's NetworkTopologyStrategy.
>>>
>>> -J
>>>
>> --
>> 
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798
>>
>
> --

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Re: Multi-DC Cluster w/ non-replicated Keyspace

[Jason J. W. Williams]

Hey Ben,

Looks like just the schema. I was surprised that running SELECTs against
the DC which should not have any data (because it's not specified in
NetworkTopologyStrategy), actually returned data. But looking at the query
trace it looks like its forwarding the queries to the other DC.

-J

On Thu, Jun 16, 2016 at 7:55 PM, Ben Slater 
wrote:

> Do you mean the data is getting replicated or just the schema?
>
> On Fri, 17 Jun 2016 at 11:48 Jason J. W. Williams <
> jasonjwwilli...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have a 2 DC cluster where the keyspaces are replicated between the 2.
>> Is it possible to add a keyspace to one of the DCs that won't be replicated
>> to the other?
>>
>> Whenever we add a new keyspace it seems to get replicated even if we
>> don't specify the other DC in the keyspace's NetworkTopologyStrategy.
>>
>> -J
>>
> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>

Re: Multi-DC Cluster w/ non-replicated Keyspace

[Ben Slater]

Do you mean the data is getting replicated or just the schema?

On Fri, 17 Jun 2016 at 11:48 Jason J. W. Williams 
wrote:

> Hi Guys,
>
> We have a 2 DC cluster where the keyspaces are replicated between the 2.
> Is it possible to add a keyspace to one of the DCs that won't be replicated
> to the other?
>
> Whenever we add a new keyspace it seems to get replicated even if we don't
> specify the other DC in the keyspace's NetworkTopologyStrategy.
>
> -J
>
-- 

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Multi-DC Cluster w/ non-replicated Keyspace

[Jason J. W. Williams]

Hi Guys,

We have a 2 DC cluster where the keyspaces are replicated between the 2. Is
it possible to add a keyspace to one of the DCs that won't be replicated to
the other?

Whenever we add a new keyspace it seems to get replicated even if we don't
specify the other DC in the keyspace's NetworkTopologyStrategy.

-J

Re: Backup strategy

[Dennis Lovely]

Snapshot would flush your memtable to disk and you could stream your
sstables out.  Incremental backups would be the differences that have
occurred since your last snapshot as far as I'm aware.  Since it's
reasonably unfeasible to constantly stream out full snapshots (depending on
the density of your data on disk), incremental backups are a faster
approach to keeping a remote location synched with your sstable changes,
which would make it much more likely to succesfully restore to points in
time.

On Thu, Jun 16, 2016 at 4:35 PM, Rakesh Kumar 
wrote:

> On Thu, Jun 16, 2016 at 7:30 PM, Bhuvan Rawal  wrote:
> > 2. Snapshotting : Hardlinks of sstables will get created. This is a very
> > fast process and latest data is captured into sstables after flushing
> > memtables, snapshots will be created in snapshots directory. But snapshot
> > does not provide you the feature to go back to a certain point in time
> but
> > incremental backups give you that feature.
>
> Does that mean that the only point-in-time recovery possible is using
> incremental backup. In other words C* does not have a concept of
> rolling forward commit logs to a point in time (like RDBMS do). Pls
> clarify.  thanks
>

Re: Backup strategy

[Dennis Lovely]

Periodic snapshots + incremental backups I think are pretty good in terms
of restoring to point in time.  But you must manage cleaning up your
snapshots + incremental backups on your own.  I believe that tablesnap (
https://github.com/JeremyGrosser/tablesnap) is a pretty decent approach in
terms of keeping your sstables, per node, synched to a location off of your
host (on S3 in fact).  Not sure how portable it is to other block storage
services however.  S3+Lifecycle policy to go to Glacier would likely be the
most cost effective for long term retention.

On Thu, Jun 16, 2016 at 4:30 PM, Bhuvan Rawal  wrote:

> Also if we talk about backup strategy for Cassandra Data then essentially
> there are couple of strategies that are adopted:
>
> 1. Incremental Backups. The old sstables will remain inside a backup
> directory and can be shipped to a storage location like AWS Glacier, etc.
> 2. Snapshotting : Hardlinks of sstables will get created. This is a very
> fast process and latest data is captured into sstables after flushing
> memtables, snapshots will be created in snapshots directory. But snapshot
> does not provide you the feature to go back to a certain point in time but
> incremental backups give you that feature.
>
> Depending on the use case, you can use 1 or 2 or both.
>
> On Fri, Jun 17, 2016 at 4:46 AM, Bhuvan Rawal  wrote:
>
>> What kind of data are we talking here?
>> Is it time series data with infrequent updates and only inserts or
>> frequently updated data. How frequently is old data read. I ask this
>> because your Node size planning and Compaction Strategy will essentially
>> depend on these.
>>
>> I have known people go upto 3-5 TB per node if data is not updated
>> frequently.
>>
>> Regards,
>> Bhuvan
>>
>> On Fri, Jun 17, 2016 at 4:31 AM,  wrote:
>>
>>> Bhuvan,
>>>
>>> Thanks for the info but actually I'm not looking for migration strategy.
>>> just want to backup strategy and retention policy best practices
>>>
>>> Thanks,
>>> Vasu
>>>
>>> On Jun 16, 2016, at 6:51 PM, Bhuvan Rawal  wrote:
>>>
>>> Hi Vasu,
>>>
>>> Planet Cassandra has a documentation page for basic info about migrating
>>> to cassandra from MySQL. What to expect and what not to. It can be found
>>> here .
>>>
>>> I had a look at this slide
>>> 
>>>  a
>>> while back. It provides a pretty reliable 4 Phase Sync strategy, starting
>>> from Slide 31. Also the QA session of the talk is informative too -
>>> http://www.doanduyhai.com/blog/?p=1757.
>>>
>>> Best Regards,
>>> Bhuvan
>>>
>>> On Fri, Jun 17, 2016 at 4:03 AM,  wrote:
>>>
 Hi ,

 I'm from relational world recently started working on Cassandra. I'm
 just wondering what is backup best practices for DB around 100 Tb with
 multi DC setup.


 Thanks,
 Vasu
>>>
>>>
>>>
>>
>

Re: Backup strategy

[Rakesh Kumar]

On Thu, Jun 16, 2016 at 7:30 PM, Bhuvan Rawal  wrote:
> 2. Snapshotting : Hardlinks of sstables will get created. This is a very
> fast process and latest data is captured into sstables after flushing
> memtables, snapshots will be created in snapshots directory. But snapshot
> does not provide you the feature to go back to a certain point in time but
> incremental backups give you that feature.

Does that mean that the only point-in-time recovery possible is using
incremental backup. In other words C* does not have a concept of
rolling forward commit logs to a point in time (like RDBMS do). Pls
clarify.  thanks

Re: Backup strategy

[Bhuvan Rawal]

Also if we talk about backup strategy for Cassandra Data then essentially
there are couple of strategies that are adopted:

1. Incremental Backups. The old sstables will remain inside a backup
directory and can be shipped to a storage location like AWS Glacier, etc.
2. Snapshotting : Hardlinks of sstables will get created. This is a very
fast process and latest data is captured into sstables after flushing
memtables, snapshots will be created in snapshots directory. But snapshot
does not provide you the feature to go back to a certain point in time but
incremental backups give you that feature.

Depending on the use case, you can use 1 or 2 or both.

On Fri, Jun 17, 2016 at 4:46 AM, Bhuvan Rawal  wrote:

> What kind of data are we talking here?
> Is it time series data with infrequent updates and only inserts or
> frequently updated data. How frequently is old data read. I ask this
> because your Node size planning and Compaction Strategy will essentially
> depend on these.
>
> I have known people go upto 3-5 TB per node if data is not updated
> frequently.
>
> Regards,
> Bhuvan
>
> On Fri, Jun 17, 2016 at 4:31 AM,  wrote:
>
>> Bhuvan,
>>
>> Thanks for the info but actually I'm not looking for migration strategy.
>> just want to backup strategy and retention policy best practices
>>
>> Thanks,
>> Vasu
>>
>> On Jun 16, 2016, at 6:51 PM, Bhuvan Rawal  wrote:
>>
>> Hi Vasu,
>>
>> Planet Cassandra has a documentation page for basic info about migrating
>> to cassandra from MySQL. What to expect and what not to. It can be found
>> here .
>>
>> I had a look at this slide
>> 
>>  a
>> while back. It provides a pretty reliable 4 Phase Sync strategy, starting
>> from Slide 31. Also the QA session of the talk is informative too -
>> http://www.doanduyhai.com/blog/?p=1757.
>>
>> Best Regards,
>> Bhuvan
>>
>> On Fri, Jun 17, 2016 at 4:03 AM,  wrote:
>>
>>> Hi ,
>>>
>>> I'm from relational world recently started working on Cassandra. I'm
>>> just wondering what is backup best practices for DB around 100 Tb with
>>> multi DC setup.
>>>
>>>
>>> Thanks,
>>> Vasu
>>
>>
>>
>

Re: Backup strategy

[Bhuvan Rawal]

What kind of data are we talking here?
Is it time series data with infrequent updates and only inserts or
frequently updated data. How frequently is old data read. I ask this
because your Node size planning and Compaction Strategy will essentially
depend on these.

I have known people go upto 3-5 TB per node if data is not updated
frequently.

Regards,
Bhuvan

On Fri, Jun 17, 2016 at 4:31 AM,  wrote:

> Bhuvan,
>
> Thanks for the info but actually I'm not looking for migration strategy.
> just want to backup strategy and retention policy best practices
>
> Thanks,
> Vasu
>
> On Jun 16, 2016, at 6:51 PM, Bhuvan Rawal  wrote:
>
> Hi Vasu,
>
> Planet Cassandra has a documentation page for basic info about migrating
> to cassandra from MySQL. What to expect and what not to. It can be found
> here .
>
> I had a look at this slide
> 
>  a
> while back. It provides a pretty reliable 4 Phase Sync strategy, starting
> from Slide 31. Also the QA session of the talk is informative too -
> http://www.doanduyhai.com/blog/?p=1757.
>
> Best Regards,
> Bhuvan
>
> On Fri, Jun 17, 2016 at 4:03 AM,  wrote:
>
>> Hi ,
>>
>> I'm from relational world recently started working on Cassandra. I'm just
>> wondering what is backup best practices for DB around 100 Tb with multi DC
>> setup.
>>
>>
>> Thanks,
>> Vasu
>
>
>

Re: Backup strategy

[vasu . nosql]

Bhuvan,

Thanks for the info but actually I'm not looking for migration strategy. just 
want to backup strategy and retention policy best practices 

Thanks,
Vasu

> On Jun 16, 2016, at 6:51 PM, Bhuvan Rawal  wrote:
> 
> Hi Vasu,
> 
> Planet Cassandra has a documentation page for basic info about migrating to 
> cassandra from MySQL. What to expect and what not to. It can be found here.
> 
> I had a look at this slide a while back. It provides a pretty reliable 4 
> Phase Sync strategy, starting from Slide 31. Also the QA session of the talk 
> is informative too - http://www.doanduyhai.com/blog/?p=1757. 
> 
> Best Regards,
> Bhuvan
> 
>> On Fri, Jun 17, 2016 at 4:03 AM,  wrote:
>> Hi ,
>> 
>> I'm from relational world recently started working on Cassandra. I'm just 
>> wondering what is backup best practices for DB around 100 Tb with multi DC 
>> setup.
>> 
>> 
>> Thanks,
>> Vasu
> 

Re: Backup strategy

[Bhuvan Rawal]

Hi Vasu,

Planet Cassandra has a documentation page for basic info about migrating to
cassandra from MySQL. What to expect and what not to. It can be found here
.

I had a look at this slide

a
while back. It provides a pretty reliable 4 Phase Sync strategy, starting
from Slide 31. Also the QA session of the talk is informative too -
http://www.doanduyhai.com/blog/?p=1757.

Best Regards,
Bhuvan

On Fri, Jun 17, 2016 at 4:03 AM,  wrote:

> Hi ,
>
> I'm from relational world recently started working on Cassandra. I'm just
> wondering what is backup best practices for DB around 100 Tb with multi DC
> setup.
>
>
> Thanks,
> Vasu

Backup strategy

[vasu . nosql]

Hi ,

I'm from relational world recently started working on Cassandra. I'm just 
wondering what is backup best practices for DB around 100 Tb with multi DC 
setup.


Thanks,
Vasu

Re: StreamCoordinator.ConnectionsPerHost set to 1

[Paulo Motta]

Increasing the number of threads alone won't help, because you need to add
connectionsPerHost-awareness to StreamPlan.requestRanges (otherwise only a
single connection per host is created) similar to what was done to
StreamPlan.transferFiles by CASSANDRA-3668, but maybe bit trickier. There's
an open ticket to support that on CASSANDRA-4663

There's also another discussion on improving rebuild parallelism on
CASSANDRA-12015.

2016-06-16 14:43 GMT-03:00 Anubhav Kale :

> Hello,
>
>
>
> I noticed that StreamCoordinator.ConnectionsPerHost is always set to 1
> (Cassandra 2.1.13). If I am reading the code correctly, this means there
> will always be just one socket (well, 2 technically for each direction)
> between nodes when rebuilding thus the data will always be serialized.
>
>
>
> Have folks experimented with increasing this ? It appears that some
> parallelism here might help rebuilds in a significant way assuming we
> aren’t hitting bandwidth caps (it’s a pain for us at the moment to rebuild
> nodes holding 500GB).
>
>
>
> I’m going to try to patch our cluster with a change to test this out, but
> wanted to hear from experts as well.
>
>
>
> Thanks !
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Dennis Lovely]

I believe you want to set memoryFraction higher, not lower.  These two
older threads seem to have similar issues you are experiencing:

https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCAHUQ+_ZqaWFs_MJ=+V49bD2paKvjLErPKMEW5duLO1jAo4=d...@mail.gmail.com%3E
https://www.mail-archive.com/user@spark.apache.org/msg44793.html

More info on tuning shuffle behavior:
https://spark.apache.org/docs/1.5.1/configuration.html#shuffle-behavior

On Thu, Jun 16, 2016 at 1:57 PM, Cassa L  wrote:

> Hi Dennis,
>
> On Wed, Jun 15, 2016 at 11:39 PM, Dennis Lovely  wrote:
>
>> You could try tuning spark.shuffle.memoryFraction and
>> spark.storage.memoryFraction (both of which have been deprecated in 1.6),
>> but ultimately you need to find out where you are bottlenecked and address
>> that as adjusting memoryFraction will only be a stopgap.  both shuffle and
>> storage memoryFractions default to 0.6
>>
>> I have set above parameters to 0.5. Does it need to increased?
>
> Thanks.
>
>> On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:
>>
>>> Hi,
>>>  I did set  --driver-memory 4G. I still run into this issue after 1
>>> hour of data load.
>>>
>>> I also tried version 1.6 in test environment. I hit this issue much
>>> faster than in 1.5.1 setup.
>>> LCassa
>>>
>>> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
>>> wrote:
>>>
 try setting the option --driver-memory 4G

 On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater  wrote:

> A high level shot in the dark but in our testing we found Spark 1.6 a
> lot more reliable in low memory situations (presumably due to
> https://issues.apache.org/jira/browse/SPARK-1). If it’s an
> option, probably worth a try.
>
> Cheers
> Ben
>
> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>
>> Hi,
>> I would appreciate any clue on this. It has become a bottleneck for
>> our spark job.
>>
>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>
>>> Hi,
>>>
>>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
>>> and writing it into Cassandra after processing it. Spark job starts 
>>> fine and runs all good for some time until I start getting below 
>>> errors. Once these errors come, job start to lag behind and I see that 
>>> job has scheduling and processing delays in streaming  UI.
>>>
>>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>>> memoryFraction parameters. Nothing works.
>>>
>>>
>>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>>> curMem=565394, maxMem=2778495713
>>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
>>> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 
>>> 69652 took 2 ms
>>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>>> broadcast_69652 in memory! (computed 496.0 B so far)
>>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 
>>> 2.6 GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit 
>>> = 2.6 GB.
>>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>>> disk instead.
>>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 
>>> (TID 452316). 2043 bytes result sent to driver
>>>
>>>
>>> Thanks,
>>>
>>> L
>>>
>>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>


>>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi Dennis,

On Wed, Jun 15, 2016 at 11:39 PM, Dennis Lovely  wrote:

> You could try tuning spark.shuffle.memoryFraction and
> spark.storage.memoryFraction (both of which have been deprecated in 1.6),
> but ultimately you need to find out where you are bottlenecked and address
> that as adjusting memoryFraction will only be a stopgap.  both shuffle and
> storage memoryFractions default to 0.6
>
> I have set above parameters to 0.5. Does it need to increased?

Thanks.

> On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:
>
>> Hi,
>>  I did set  --driver-memory 4G. I still run into this issue after 1 hour
>> of data load.
>>
>> I also tried version 1.6 in test environment. I hit this issue much
>> faster than in 1.5.1 setup.
>> LCassa
>>
>> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
>> wrote:
>>
>>> try setting the option --driver-memory 4G
>>>
>>> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
>>> wrote:
>>>
 A high level shot in the dark but in our testing we found Spark 1.6 a
 lot more reliable in low memory situations (presumably due to
 https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
 probably worth a try.

 Cheers
 Ben

 On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:

> Hi,
> I would appreciate any clue on this. It has become a bottleneck for
> our spark job.
>
> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>
>> Hi,
>>
>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
>> and writing it into Cassandra after processing it. Spark job starts fine 
>> and runs all good for some time until I start getting below errors. Once 
>> these errors come, job start to lag behind and I see that job has 
>> scheduling and processing delays in streaming  UI.
>>
>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>> memoryFraction parameters. Nothing works.
>>
>>
>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>> curMem=565394, maxMem=2778495713
>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
>> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 
>> 69652 took 2 ms
>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>> broadcast_69652 in memory! (computed 496.0 B so far)
>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
>> GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 
>> 2.6 GB.
>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>> disk instead.
>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>> 452316). 2043 bytes result sent to driver
>>
>>
>> Thanks,
>>
>> L
>>
>>
> --
 
 Ben Slater
 Chief Product Officer
 Instaclustr: Cassandra + Spark - Managed | Consulting | Support
 +61 437 929 798

>>>
>>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

On Thu, Jun 16, 2016 at 5:27 AM, Deepak Goel  wrote:

> What is your hardware configuration like which you are running Spark on?
>
> It  is 24core, 128GB RAM

> Hey
>
> Namaskara~Nalama~Guten Tag~Bonjour
>
>
>--
> Keigu
>
> Deepak
> 73500 12833
> www.simtree.net, dee...@simtree.net
> deic...@gmail.com
>
> LinkedIn: www.linkedin.com/in/deicool
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
>
> "Contribute to the world, environment and more :
> http://www.gridrepublic.org
> "
>
> On Thu, Jun 16, 2016 at 5:33 PM, Jacek Laskowski  wrote:
>
>> Hi,
>>
>> What do you see under Executors and Details for Stage (for the
>> affected stages)? Anything weird memory-related?
>>
>> How does your "I am reading data from Kafka into Spark and writing it
>> into Cassandra after processing it." pipeline look like?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
>> > Hi,
>> >
>> > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark
>> and
>> > writing it into Cassandra after processing it. Spark job starts fine and
>> > runs all good for some time until I start getting below errors. Once
>> these
>> > errors come, job start to lag behind and I see that job has scheduling
>> and
>> > processing delays in streaming  UI.
>> >
>> > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
>> > memoryFraction parameters. Nothing works.
>> >
>> >
>> > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
>> > curMem=565394, maxMem=2778495713
>> > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored
>> as
>> > bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable
>> 69652
>> > took 2 ms
>> > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
>> > threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
>> > broadcast_69652 in memory! (computed 496.0 B so far)
>> > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) +
>> 2.6 GB
>> > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6
>> GB.
>> > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to
>> disk
>> > instead.
>> > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
>> > 452316). 2043 bytes result sent to driver
>> >
>> >
>> > Thanks,
>> >
>> > L
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,

>
> What do you see under Executors and Details for Stage (for the
> affected stages)? Anything weird memory-related?
>
Under executor Tab, logs throw these warning -

16/06/16 20:45:40 INFO TorrentBroadcast: Reading broadcast variable
422145 took 1 ms
16/06/16 20:45:40 WARN MemoryStore: Failed to reserve initial memory
threshold of 1024.0 KB for computing block broadcast_422145 in memory.
16/06/16 20:45:40 WARN MemoryStore: Not enough space to cache
broadcast_422145 in memory! (computed 496.0 B so far)
16/06/16 20:45:40 INFO MemoryStore: Memory use = 147.9 KB (blocks) +
2.2 GB (scratch space shared across 0 tasks(s)) = 2.2 GB. Storage
limit = 2.2 GB.
16/06/16 20:45:40 WARN MemoryStore: Persisting block broadcast_422145
to disk instead.
16/06/16 20:45:40 INFO MapOutputTrackerWorker: Don't have map outputs
for shuffle 70278, fetching them

16/06/16 20:45:40 INFO MapOutputTrackerWorker: Doing the fetch; tracker
endpoint = AkkaRpcEndpointRef(Actor[akka.tcp://
sparkDriver@17.40.240.71:46187/user/MapOutputTracker#-1794035569])

I dont see any memory related errors on 'stages' Tab.

>
> How does your "I am reading data from Kafka into Spark and writing it
> into Cassandra after processing it." pipeline look like?
>
> This part has no issues. Reading from Kafka is always up to date. There
are no offset lags. Writting to Cassandra is also fine with less than 1ms
to write data.


> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
> > Hi,
> >
> > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark
> and
> > writing it into Cassandra after processing it. Spark job starts fine and
> > runs all good for some time until I start getting below errors. Once
> these
> > errors come, job start to lag behind and I see that job has scheduling
> and
> > processing delays in streaming  UI.
> >
> > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
> > memoryFraction parameters. Nothing works.
> >
> >
> > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
> > curMem=565394, maxMem=2778495713
> > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored
> as
> > bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652
> > took 2 ms
> > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
> > threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
> > broadcast_69652 in memory! (computed 496.0 B so far)
> > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6
> GB
> > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6
> GB.
> > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to
> disk
> > instead.
> > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
> > 452316). 2043 bytes result sent to driver
> >
> >
> > Thanks,
> >
> > L
>

StreamCoordinator.ConnectionsPerHost set to 1

[Anubhav Kale]

Hello,

I noticed that StreamCoordinator.ConnectionsPerHost is always set to 1 
(Cassandra 2.1.13). If I am reading the code correctly, this means there will 
always be just one socket (well, 2 technically for each direction) between 
nodes when rebuilding thus the data will always be serialized.

Have folks experimented with increasing this ? It appears that some parallelism 
here might help rebuilds in a significant way assuming we aren't hitting 
bandwidth caps (it's a pain for us at the moment to rebuild nodes holding 
500GB).

I'm going to try to patch our cluster with a change to test this out, but 
wanted to hear from experts as well.

Thanks !

Re: Reason for Trace Message Drop

[Eric Stevens]

Are you executing all queries with tracing enabled?  If so that introduces
overhead you probably don't want.  Most people probably don't see this log
very often because it's the exception to query with tracing enabled, and
not the rule (it's a diagnostic thing usually turned on only when
troubleshooting another problem).

The trace messages being dropped aren't in-and-of-themselves a problem, it
just means that your traces would be incomplete.  However this indicates a
cluster that's either unhealthy or getting close to it.

Since you have a single node, run nodetool tpstats (if you had a cluster
you'd want to do this on each node).  Look at the bottom section, ideally
it looks like this:
Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
MUTATION 0
COUNTER_MUTATION 0
BINARY   0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0

If you're seeing MUTATIONs or COUNTER_MUTATIONs dropped, your data
integrity is at risk, you need to reduce pressure in some way, as your
cluster is falling behind.  That might be expanding your node count
*above* your
RF, or it might be tuning your application to reduce read or write pressure
(note that expanding a cluster involves read pressure on the existing
nodes, so to successfully expand your cluster you probably *also* need to
reduce pressure from the application, because the act of growing your
cluster will cause you to get even further behind).

On a single node, like yours, mutations dropped is data lost.  On a cluster
with RF > 1, there's just a *chance* that you have lost data (the same
mutation needs to be dropped by all replicas for it to be irrecoverably
lost).  Dropped mutations are always bad.

In this scenario you probably also want to double check nodetool
compactionstats, and ideally you have < 4 or 5 compactions.  Lower is
always better, 4 or 5 is getting worrying if it remains in that range.  If
you're above 10, you're falling behind on compaction too, and your read
performance will be suffering.

Finally RE your JVM settings, with a 30GB node, I think you could turn that
up a bit if you convert to G1GC.  JVM tuning is definitely not my strong
point, there are others on this list who will be able to help you do a
better job of it.  If you're seeing big GC pauses, then you do need to work
on this, even if they're not the direct cause of the dropped traces.  With
that column family count you'll be under more GC pressure than you would be
with a lower CF count (there is a fixed memory cost per CF).  Reconsider
your data model, usually this many column families suggests dynamically
creating CF's (eg to solve multi-tenancy).  If your CF count will grow
steadily over time at any appreciable rate, that's an anti-pattern.

On Thu, Jun 16, 2016 at 2:40 AM Varun Barala 
wrote:

> Thanks!! Eric Stevens for your reply.
>
> We have following JVM settings :-
> -
> *memtable_offheap_space_in_mb: 15360  (*found in casandra.yaml
> *)*
> *MAX_HEAP_SIZE="16G"  (*found in cassandra-env.sh
> *)*-
>
> And I also found big GC in log. But messages and big GC were logged at
> different-different time in system.log. I was expecting to happen them at
> same time after reading your reply. I also manually triggered GC but
> messages were not dropped.
>
> is *TRACE message drop *harmful or it's okay we can neglect them?
>
> Thank you!!
>
>
> On Wed, Jun 15, 2016 at 8:45 PM, Eric Stevens  wrote:
>
>> This is better kept to the User groups.
>>
>> What are your JVM memory settings for Cassandra, and have you seen big
>> GC's in your logs?
>>
>> The reason I ask is because that's a large number of column families,
>> which produces memory pressure, and at first blush that strikes me as a
>> likely cause.
>>
>>
>> On Wed, Jun 15, 2016 at 3:23 AM Varun Barala 
>> wrote:
>>
>>> Hi all,
>>>
>>> Can anyone tell me that what are all possible reasons for below log:-
>>>
>>>
>>> *"INFO  [ScheduledTasks:1] 2016-06-14 06:27:39,498
>>> MessagingService.java:929 - _TRACE messages were dropped in last 5000 ms:
>>> 928 for internal timeout and 0 for cross node timeout".*
>>> I searched online for the same and found some reasons like:-
>>>
>>> * Disk is not able to keep up with your ingest
>>> * Resources are not able to support all parallel running tasks
>>> * If other nodes are down then due to large hint replay
>>> * Heavy workload
>>>
>>> But in this case other kind of messages (mutation, read, write etc)
>>>  should be dropped by *C** but It doesn't happen.
>>>
>>> -
>>> Cluster Specifications
>>> --
>>> number of nodes = 1
>>> total number of CF = 2000
>>>
>>> -
>>> Machine Specifications
>>> 

Re: Spark Memory Error - Not enough space to cache broadcast

[Jacek Laskowski]

Hi,

What do you see under Executors and Details for Stage (for the
affected stages)? Anything weird memory-related?

How does your "I am reading data from Kafka into Spark and writing it
into Cassandra after processing it." pipeline look like?

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and
> writing it into Cassandra after processing it. Spark job starts fine and
> runs all good for some time until I start getting below errors. Once these
> errors come, job start to lag behind and I see that job has scheduling and
> processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as
> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
> broadcast_69652 in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB
> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk
> instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L

Re: Streaming from 1 node only when adding a new DC

[Fabien Rousseau]

Thanks,

Created the issue: https://issues.apache.org/jira/browse/CASSANDRA-12015

2016-06-15 15:25 GMT+02:00 Paulo Motta :

> For rebuild, replace and -Dcassandra.consistent.rangemovement=false in
> general we currently pick the closest replica (as indicated by the Snitch)
> which has the range, what will often map to the same node due to the
> dynamic snitch, specially when N=RF. This is good for picking a node in the
> same DC or rack for transferring, but we can probably improve this to
> distribute streaming load more evenly within candidate source nodes in the
> same rack/DC.
>
> Would you mind opening a ticket for improving this?
>
>
> 2016-06-14 17:35 GMT-03:00 Fabien Rousseau :
>
>> We've tested with C* 2.1.14 version
>> Yes VNodes with 256 tokens
>> Once all the nodes in dc2 are added, schema is modified to have RF=3 in
>> dc1 and RF=3 in dc2.
>> Then on each nodes of dc2:
>> nodetool rebuild dc1
>> Le 14 juin 2016 10:39, "kurt Greaves"  a écrit :
>>
>>> What version of Cassandra are you using? Also what command are you using
>>> to run the rebuilds? Are you using vnodes?
>>>
>>> On 13 June 2016 at 09:01, Fabien Rousseau  wrote:
>>>
 Hello,

 We've tested adding a new DC from an existing DC having 3 nodes and
 RF=3 (ie all nodes have all data).
 During the rebuild process, only one node of the first DC streamed data
 to the 3 nodes of the second DC.

 Our goal is to minimise the time it takes to rebuild a DC and would
 like to be able to stream from all nodes.

 Starting C* with debug logs, it appears that all nodes, when computing
 their "streaming plan" returns the same node for all ranges.
 This is probably because all nodes in DC2 have the same view of the
 ring.

 I understand that when bootstrapping a new node, it's preferable to
 stream from the node being replaced, but when rebuilding a new DC, it
 should probably select sources "randomly" (rather than always selecting the
 same source for a specific range).
 What do you think ?

 Best Regards,
 Fabien

>>>
>>>
>>>
>>> --
>>> Kurt Greaves
>>> k...@instaclustr.com
>>> www.instaclustr.com
>>>
>>
>

Re: Reason for Trace Message Drop

[Varun Barala]

Thanks!! Eric Stevens for your reply.

We have following JVM settings :-
-
*memtable_offheap_space_in_mb: 15360  (*found in casandra.yaml
*)*
*MAX_HEAP_SIZE="16G"  (*found in cassandra-env.sh
*)*-

And I also found big GC in log. But messages and big GC were logged at
different-different time in system.log. I was expecting to happen them at
same time after reading your reply. I also manually triggered GC but
messages were not dropped.

is *TRACE message drop *harmful or it's okay we can neglect them?

Thank you!!


On Wed, Jun 15, 2016 at 8:45 PM, Eric Stevens  wrote:

> This is better kept to the User groups.
>
> What are your JVM memory settings for Cassandra, and have you seen big
> GC's in your logs?
>
> The reason I ask is because that's a large number of column families,
> which produces memory pressure, and at first blush that strikes me as a
> likely cause.
>
>
> On Wed, Jun 15, 2016 at 3:23 AM Varun Barala 
> wrote:
>
>> Hi all,
>>
>> Can anyone tell me that what are all possible reasons for below log:-
>>
>>
>> *"INFO  [ScheduledTasks:1] 2016-06-14 06:27:39,498
>> MessagingService.java:929 - _TRACE messages were dropped in last 5000 ms:
>> 928 for internal timeout and 0 for cross node timeout".*
>> I searched online for the same and found some reasons like:-
>>
>> * Disk is not able to keep up with your ingest
>> * Resources are not able to support all parallel running tasks
>> * If other nodes are down then due to large hint replay
>> * Heavy workload
>>
>> But in this case other kind of messages (mutation, read, write etc)
>>  should be dropped by *C** but It doesn't happen.
>>
>> -
>> Cluster Specifications
>> --
>> number of nodes = 1
>> total number of CF = 2000
>>
>> -
>> Machine Specifications
>> --
>> RAM 30 GB
>> hard disk SSD
>> ubuntu 14.04
>>
>>
>> Thanks in advance!!
>>
>> Regards,
>> Varun Barala
>>
>

RE: how to force cassandra-stress to actually generate enough data

[Peter Kovgan]

Thank you, guys.
I will try all proposals.
The limitation, mentioned by Benedict, is huge.
But anyway, there is something to do around.

From: Peter Kovgan
Sent: Wednesday, June 15, 2016 3:25 PM
To: 'user@cassandra.apache.org'
Subject: how to force cassandra-stress to actually generate enough data

Hi,

The cassandra-stress is not helping really to populate the disk sufficiently.

I tried several table structures, providing

cluster: UNIFORM(1..100)  on clustering parts of the PK.

Partition part of PK makes about 660 000 partitions.

The hope was create enough cells in a row, make the row really WIDE.

No matter what I tried, does no matter how long it runs, I see maximum 2-3 
SSTables per node and maximum 300Mb of data per node.

(I have 6 nodes and very active 400 threads stress)

It looks, like It is impossible to make the row really wide and disk really 
full.

Is it intentional?

I mean, if there was an intention to avoid really wide rows, why there is no 
hint on this in docs?

Do you have similar experience and do you know how resolve that?

Thanks.

**
This communication and all or some of the information contained therein may be 
confidential and is subject to our Terms and Conditions. If you have received 
this
communication in error, please destroy all electronic and paper copies and 
notify the sender immediately. Unless specifically indicated, this 
communication is 
not a confirmation, an offer to sell or solicitation of any offer to buy any 
financial product, or an official statement of ICAP or its affiliates. 
Non-Transactable Pricing Terms and Conditions apply to any non-transactable 
pricing provided. All terms and conditions referenced herein available
at www.icapterms.com. Please notify us by reply message if this link does not 
work.
**

Re: Spark Memory Error - Not enough space to cache broadcast

[Dennis Lovely]

You could try tuning spark.shuffle.memoryFraction and
spark.storage.memoryFraction (both of which have been deprecated in 1.6),
but ultimately you need to find out where you are bottlenecked and address
that as adjusting memoryFraction will only be a stopgap.  both shuffle and
storage memoryFractions default to 0.6

On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:

> Hi,
>  I did set  --driver-memory 4G. I still run into this issue after 1 hour
> of data load.
>
> I also tried version 1.6 in test environment. I hit this issue much faster
> than in 1.5.1 setup.
> LCassa
>
> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
> wrote:
>
>> try setting the option --driver-memory 4G
>>
>> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
>> wrote:
>>
>>> A high level shot in the dark but in our testing we found Spark 1.6 a
>>> lot more reliable in low memory situations (presumably due to
>>> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
>>> probably worth a try.
>>>
>>> Cheers
>>> Ben
>>>
>>> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>>>
 Hi,
 I would appreciate any clue on this. It has become a bottleneck for our
 spark job.

 On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:

> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
> and writing it into Cassandra after processing it. Spark job starts fine 
> and runs all good for some time until I start getting below errors. Once 
> these errors come, job start to lag behind and I see that job has 
> scheduling and processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
> broadcast_69652 in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
> GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 
> GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
> disk instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L
>
>
 --
>>> 
>>> Ben Slater
>>> Chief Product Officer
>>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>>> +61 437 929 798
>>>
>>
>>
>