Re: How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)

2018-04-15 Thread Dinesh Joshi
Hi Revanth,
I took a quick look and don't think you can override existing mappings. You 
should see a warning like this -
18/04/15 19:50:27 WARN CodecRegistry: Ignoring codec CustomTSTypeCodec [date 
<-> java.lang.Long] because it collides with previously registered codec 
CustomTSTypeCodec [date <-> java.lang.Long]
The issue is that CodecRegistry does not allow you to override existing type 
mappings[1][2]. You can however register a codec that converts between a 
CqlType and your custom class.
Dinesh
[1] 
https://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/[2] 
https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/CodecRegistry.java#L357




 

On Sunday, April 15, 2018, 1:06:53 AM PDT, Revanth Reddy 
 wrote:  
 
 Hi Dinesh,
Thanks a lot for your response . I had tried with the existing codec 
(SimpleTimestampCodec) ,but it says the codec has been registered but it is not 
converting the timestamp column to long . also i don't observe any error.
Below are the steps that i had followed and also attached the code that i had 
used:
I had used this property to set the codec in spark 
config("spark.cassandra.connection.factory", 
"com.rasa.devops.cassandra.export.CustomConnectionFactory")

Since there is no direct way to refer the custom codec in spark , i had 
followed the above approach.
I had used the following dependencies :
               com.datastax.spark 
spark-cassandra-connector_2.11 
2.0.1   
com.datastax.cassandra 
cassandra-driver-extras 3.1.4 
  joda-time 
joda-time 2.9.1 
Cassandra table used for testing :

CREATE TABLE dev.my_table (

    partition_key text PRIMARY KEY,

    some_timestamp timestamp

);






cqlsh:dev> select * from my_table;




 partition_key | some_timestamp

---+--

           key | 2015-12-25 18:30:00+

           foo | 2018-04-11 06:22:29+









Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login










    










                                                                                
           

On Sun, Apr 15, 2018 at 11:36 AM, Dinesh Joshi  wrote:

Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried 
using a pre-existing codec? It would be helpful if you can give more 
information.
DineshOn Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy 
 wrote:  
 
 

Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my 
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another 
while reading from cassandra. for example i have a timestamp column in 
cassandra table ,but i want to read it as long and save as parquet file to HDFS 
, then this cutom codecs will be usefull instead of doing an additional 
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
 I had tried with solution provided in the below link but it does not work , 
any ideas around this will be helpfull.
https://stackoverflow.com/ questions/40363611/adding- custom-codec-to- 
cassandraconnector/49750791? noredirect=1#comment86515847_ 49750791



Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/ crazyusers/login










    










                                                                                
           
  

  

Re: How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)

2018-04-15 Thread Dinesh Joshi
Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried 
using a pre-existing codec? It would be helpful if you can give more 
information.
DineshOn Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy 
 wrote:  
 
 

Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my 
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another 
while reading from cassandra. for example i have a timestamp column in 
cassandra table ,but i want to read it as long and save as parquet file to HDFS 
, then this cutom codecs will be usefull instead of doing an additional 
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
 I had tried with solution provided in the below link but it does not work , 
any ideas around this will be helpfull.
https://stackoverflow.com/questions/40363611/adding-custom-codec-to-cassandraconnector/49750791?noredirect=1#comment86515847_49750791



Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login










    










                                                                                
           
  

How to register cassandra custom codecs in spark? (https://github.com/datastax/spark-cassandra-connector/issues/1173)

2018-04-14 Thread Revanth Reddy
Hi Team ,

I want to write a custom cassandra codec and i want to use that codec in my
spark application while reading the data from cassandra table .

Basically the custom codecs are used to convert one column type to another
while reading from cassandra. for example i have a timestamp column in
cassandra table ,but i want to read it as long and save as parquet file to
HDFS , then this cutom codecs will be usefull instead of doing an
additional transformation in the spark code.

I want to know how to set/register this custom cassandra codecs in spark ?

 I had tried with solution provided in the below link but it does not work
, any ideas around this will be helpfull.

https://stackoverflow.com/questions/40363611/adding-custom-codec-to-cassandraconnector/49750791?noredirect=1#comment86515847_49750791




*Thanks & Regards*
*S.Revanth kumar reddy*
https://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login