Re: spark cassandra questiom

Yes, it should be good to use Spark for this use case in my opinion. You can look into using the Cassandra Spark connector for persisting your updated data into Cassandra. Cheers, Sonal Nube Technologies Join me at Data Con LA Oct 23 | Big Data Conference Europe. Nov 24 |

Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2?

2.4.3 Binary is out now and they did change back to 2.11. https://www.apache.org/dyn/closer.lua/spark/spark-2.4.3/spark-2.4.3-bin-hadoop2.7.tgz On Mon, May 6, 2019 at 9:21 PM Russell Spitzer wrote: > Spark 2.4.2 was incorrectly released with the default package binaries set > to Scala 2.12 >

Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2?

Actually i just checked the release, they only changed the pyspark part. So the download on the website will still be 2.12 so you'll need to build the scala 2.11 version of Spark if you want to use the connector. Or Submit a PR for scala 2.12 support On Mon, May 6, 2019 at 9:21 PM Russell Spitzer

Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2?

Spark 2.4.2 was incorrectly released with the default package binaries set to Scala 2.12 instead of scala 2.11.12 which was supposed to be the case. See the 2.4.3 vote

Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2?

Thanks for the reply. Unfortunately this is the highest version available for Cassandra connector.  One thing I don’t quite understand is that it worked perfectly under Spark 2.4.0. I thought support for Scala 2.11 only became deprecated starting spark 2.4.1, will be removed after spark 3.0 Se

Re: spark-cassandra-connector_2.1 caused java.lang.NoClassDefFoundError under Spark 2.4.2?

Scala version mismatched Spark is shown at 2.12, the connector only has a 2.11 release On Mon, May 6, 2019, 7:59 PM Richard Xin wrote: > > org.apache.spark > spark-core_2.12 > 2.4.0 > compile > > > org.apache.spark > spark-sql_2.12 > 2.4.0 > > > com.datast

Re: spark cassandra issue

Hi Russell. if possible pleae help me to solve the below issue. val df = sqlContext.read. format("org.apache.spark.sql.cassandra"). options(Map("c_table"->"restt","keyspace"->"sss")). load() com.datastax.driver.core.TransportException: [/192.23.2.100:9042] Cannot connect at com.datastax

Re: spark cassandra issue

This would also be a better question for the SCC user list :) https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-connector-user On Sun, Sep 4, 2016 at 9:31 AM Russell Spitzer wrote: > > https://github.com/datastax/spark-cassandra-connector/blob/v1.3.1/doc/14_data_frames.md > In S

Re: spark cassandra issue

https://github.com/datastax/spark-cassandra-connector/blob/v1.3.1/doc/14_data_frames.md In Spark 1.3 it was illegal to use "table" as a key in Spark SQL so in that version of Spark the connector needed to use the option "c_table" val df = sqlContext.read. | format("org.apache.spark.sql.cassan

Re: spark cassandra issue

and your Cassandra table is there etc? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw * http://talebzadehmich.wordpress.com *Disclaimer:* Use

Re: spark cassandra issue

Hey Mich, I am using the same one right now. Thanks for the reply. import org.apache.spark.sql.cassandra._ import com.datastax.spark.connector._ //Loads implicit functions sc.cassandraTable("keyspace name", "table name") On Sun, Sep 4, 2016 at 8:48 PM, Mich Talebzadeh wrote: > Hi Selvan. > > I

Re: spark cassandra issue

Hi Selvan. I don't deal with Cassandra but have you tried other options as described here https://github.com/datastax/spark-cassandra-connector/blob/master/doc/2_loading.md To get a Spark RDD that represents a Cassandra table, call the cassandraTable method on the SparkContext object. import co

Re: spark cassandra issue

its very urgent. please help me guys. On Sun, Sep 4, 2016 at 8:05 PM, Selvam Raman wrote: > Please help me to solve the issue. > > spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.10:1.3.0 > --conf spark.cassandra.connection.host=** > > val df = sqlContext.read. > |

RE: spark-cassandra-connector BulkOutputWriter

Alex – I suggest posting this question on the Spark Cassandra Connector mailing list. The SCC developers are pretty responsive. Mohammed Author: Big Data Analytics with Spark From: Alexandr Dzhagriev [mailto:dzh...@gma

RE: spark-cassandra

Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Gerard Maas [mailto:gerard.m...@gmail.com] Sent: Wednesday, February 3, 2016 4:56 AM To: Madabhattula Rajesh Kumar Cc: user@spark.apache.org Subject: Re: spark-cassandra NoSuchMethodError usually refer

Re: spark-cassandra

NoSuchMethodError usually refers to a version conflict. Probably your job was built against a higher version of the cassandra connector than what's available on the run time. Check that the versions are aligned. -kr, Gerard. On Wed, Feb 3, 2016 at 1:37 PM, Madabhattula Rajesh Kumar < mrajaf...@gm

Re: Spark Cassandra clusters

mazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, January 22, 2016 6:37 PM To: vivek.meghanat...@wipro.com Cc: user Subject: Re: Spark Cassandra clusters I am not Cassandra developer :-) Can you use http://search-hadoop.co

RE: Spark Cassandra clusters

Spark<http://www.amazon.com/Big-Data-Analytics-Spark-Practitioners/dp/1484209656/> From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, January 22, 2016 6:37 PM To: vivek.meghanat...@wipro.com Cc: user Subject: Re: Spark Cassandra clusters I am not Cassandra developer :-) Can you us

Re: Spark Cassandra clusters

Vivek: I searched for 'cassandra gc pause' and found a few hits. e.g. : http://search-hadoop.com/m/qZFqM1c5nrn1Ihwf6&subj=Re+GC+pauses+affecting+entire+cluster+ Keep in mind the effect of GC on shared nodes. FYI On Fri, Jan 22, 2016 at 7:09 PM, Mohammed Guller wrote: > For data locality, it is

RE: Spark Cassandra clusters

For data locality, it is recommended to run the Spark workers and Cassandra on the same nodes. Mohammed Author: Big Data Analytics with Spark From: vivek.meghanat...@wipro.com [mailto:vivek.meghanat...@wipro.com] Sent:

Re: Spark Cassandra clusters

I am not Cassandra developer :-) Can you use http://search-hadoop.com/ or ask on Cassandra mailing list. Cheers On Fri, Jan 22, 2016 at 6:35 PM, wrote: > Thanks Ted, also what is the suggested memory setting for Cassandra > process? > > Regards > Vivek > On Sat, Jan 23, 2016 at 7:57 am, Ted Yu

Re: Spark Cassandra clusters

Thanks Ted, also what is the suggested memory setting for Cassandra process? Regards Vivek On Sat, Jan 23, 2016 at 7:57 am, Ted Yu mailto:yuzhih...@gmail.com>> wrote: >From your description, putting Cassandra daemon on Spark cluster should be >feasible. One aspect to be measured is how much l

Re: Spark Cassandra clusters

>From your description, putting Cassandra daemon on Spark cluster should be feasible. One aspect to be measured is how much locality can be achieved in this setup - Cassandra is distributed NoSQL store. Cheers On Fri, Jan 22, 2016 at 6:13 PM, wrote: > + spark standalone cluster > On Sat, Jan 2

Re: Spark Cassandra clusters

+ spark standalone cluster On Sat, Jan 23, 2016 at 7:33 am, Vivek Meghanathan (WT01 - NEP) mailto:vivek.meghanat...@wipro.com>> wrote: We have the setup on Google cloud platform. Each node has 8 CPU + 30GB memory. 10 nodes for spark another 9nodes for Cassandra. We are using spark 1.3.0 and Da

Re: Spark Cassandra clusters

Thanks. We are using spark - Cassandra connector aligned for spark 1.3. Regards Vivek On Sat, Jan 23, 2016 at 7:27 am, Durgesh Verma mailto:dv21...@gmail.com>> wrote: This may be useful, you can try connectors. https://academy.datastax.com/demos/getting-started-apache-spark-and-cassandra https

Re: Spark Cassandra clusters

We have the setup on Google cloud platform. Each node has 8 CPU + 30GB memory. 10 nodes for spark another 9nodes for Cassandra. We are using spark 1.3.0 and Datastax bundle 4.5.9(which has 2.0.x Cassandra). Spark master and worker daemon uses Xmx & Xms 4G. We have not changed the default setting

Re: Spark Cassandra clusters

This may be useful, you can try connectors. https://academy.datastax.com/demos/getting-started-apache-spark-and-cassandra https://spark-summit.org/2015/events/cassandra-and-spark-optimizing-for-data-locality/ Thanks, -Durgesh > On Jan 22, 2016, at 8:37 PM, > wrote: > > Hi All, > What is the

Re: Spark Cassandra clusters

Can you give us a bit more information ? How much memory does each node have ? What's the current heap allocation for Cassandra process and executor ? Spark / Cassandra release you are using Thanks On Fri, Jan 22, 2016 at 5:37 PM, wrote: > Hi All, > What is the right spark Cassandra cluster se

Re: Spark Cassandra Java Connector: records missing despite consistency=ALL

Hi Anthony, no, the logging is not done via Spark (but PHP). But that does not really matter, as the records are eventually there. So it is the READ_CONSISTENCY=ALL that is not working. Btw. it seems that using withReadConf() and setting the consistency level there is working but I need to w

Re: Spark Cassandra Java Connector: records missing despite consistency=ALL

So is the logging to Cassandra being done via Spark ? On Wed, Jan 13, 2016 at 7:17 AM, Dennis Birkholz wrote: > Hi together, > > we Cassandra to log event data and process it every 15 minutes with Spark. > We are using the Cassandra Java Connector for Spark. > > Randomly our Spark runs produce t

Re: Spark-Cassandra-connector

Have you considered asking this question on https://groups.google.com/a/lists.datastax.com/forum/#!forum/spark-connector-user ? Cheers On Thu, Aug 20, 2015 at 10:57 PM, Samya wrote: > Hi All, > > I need to write an RDD to Cassandra & using the sparkCassandraConnector > from > DataStax. My appli

Re: Spark Cassandra Connector issue

HI, Can we apply *saveToCassandra method to a JdbcRDD * Code: *Code:* *import* *org.apache*.spark.SparkContext *import* *org.apache*.spark.SparkContext._ *import* *org.apache*.spark.SparkConf *import* *org.apache*.spark.rdd.JdbcRDD *import* *com.datastax*.spark.connector._ *import* com.dat

Re: Spark Cassandra Connector issue

HI All, I have tried Commands as mentioned below but still it is errors dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars /home/missingmerch/ postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark- cassandra-connector-java_2.10-1.1.1.jar /hom

Re: Spark Cassandra Connector issue

I don't know if DSE changed spark-submit, but you have to use a comma-separated list of jars to --jars. It probably looked for HelloWorld in the second one, the dse.jar file. Do this: dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars /home/missingmerch/ postgresql-9.4-1

Re: Spark Cassandra Connector issue

Hi, Thanks for quick input, now I am getting class not found error *Command:* dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar ///home/missingmerch/dse.jar ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1

Re: Spark Cassandra Connector issue

Add the other Cassandra dependencies (dse.jar, spark-cassandra-connect-java_2.10) to your --jars argument on the command line. Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition (O'Reilly) Typesafe @deanwampler

Re: Spark Cassandra connector number of Tasks

Looking for help with this. Thank you! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Cassandra-connector-number-of-Tasks-tp22820p22839.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark Cassandra Connector

1.2.0-rc3 can be found here: http://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.10 Can you use maven to build your project ? Cheers > On Apr 18, 2015, at 9:02 PM, DStrip wrote: > > Hello, > > I am facing some difficulties on installing the Cassandra Spark connect

RE: Spark Cassandra Guava version issues

Did the workaround work for you? Doesn't seem to work for me. Date: Mon, 24 Nov 2014 16:44:17 +0100 Subject: Re: Spark Cassandra Guava version issues From: shahab.mok...@gmail.com To: as...@live.com CC: user@spark.apache.org I faced same problem, and s work around solution is here :

Re: Spark Cassandra Guava version issues

I faced same problem, and s work around solution is here : https://github.com/datastax/spark-cassandra-connector/issues/292 best, /Shahab On Mon, Nov 24, 2014 at 3:21 PM, Ashic Mahtab wrote: > I've got a Cassandra 2.1.1 + Spark 1.1.0 cluster running. I'm using > sbt-assembly to create a uber

RE: Spark Cassandra Connector proper usage

Hi Gerard, I've gone with option 1, and seems to be working well. Option 2 is also quite interesting. Thanks for your help in this. Regards, Ashic. From: gerard.m...@gmail.com Date: Thu, 23 Oct 2014 17:07:56 +0200 Subject: Re: Spark Cassandra Connector proper usage To: as...@live.com CC:

Re: Spark Cassandra Connector proper usage

his > is happening in parallel on multiple machines, that would likely cause > discrepancies where a node will read and update to older values. Hence my > question about session management in order to issue custom update queries. > > Thanks, > Ashic. > > -

RE: Spark Cassandra Connector proper usage

order to issue custom update queries. Thanks, Ashic. Date: Thu, 23 Oct 2014 14:27:47 +0200 Subject: Re: Spark Cassandra Connector proper usage From: gerard.m...@gmail.com To: as...@live.com Ashic, With the Spark-cassandra connector you would typically create an RDD from the source table, upd

Re: Spark Cassandra connector issue

Is this because I am calling a transformation function on an rdd from inside another transformation function? Is it not allowed? Thanks Ankut On Oct 21, 2014 1:59 PM, "Ankur Srivastava" wrote: > Hi Gerard, > > this is the code that may be helpful. > > public class ReferenceDataJoin implements S