Hi Revanth,
I took a quick look and don't think you can override existing mappings. You 
should see a warning like this -
18/04/15 19:50:27 WARN CodecRegistry: Ignoring codec CustomTSTypeCodec [date 
<-> java.lang.Long] because it collides with previously registered codec 
CustomTSTypeCodec [date <-> java.lang.Long]
The issue is that CodecRegistry does not allow you to override existing type 
mappings[1][2]. You can however register a codec that converts between a 
CqlType and your custom class.
Dinesh
[1] 
https://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/[2] 
https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/CodecRegistry.java#L357




 

    On Sunday, April 15, 2018, 1:06:53 AM PDT, Revanth Reddy 
<revanthkuma...@gmail.com> wrote:  
 
 Hi Dinesh,
Thanks a lot for your response . I had tried with the existing codec 
(SimpleTimestampCodec) ,but it says the codec has been registered but it is not 
converting the timestamp column to long . also i don't observe any error.
Below are the steps that i had followed and also attached the code that i had 
used:
I had used this property to set the codec in spark 
config("spark.cassandra.connection.factory", 
"com.rasa.devops.cassandra.export.CustomConnectionFactory")

Since there is no direct way to refer the custom codec in spark , i had 
followed the above approach.
I had used the following dependencies :
              <dependency> <groupId>com.datastax.spark</groupId> 
<artifactId>spark-cassandra-connector_2.11</artifactId> 
<version>2.0.1</version> </dependency> <dependency> 
<groupId>com.datastax.cassandra</groupId> 
<artifactId>cassandra-driver-extras</artifactId> <version>3.1.4</version> 
</dependency> <dependency> <groupId>joda-time</groupId> 
<artifactId>joda-time</artifactId> <version>2.9.1</version> </dependency>
Cassandra table used for testing :

CREATE TABLE dev.my_table (

    partition_key text PRIMARY KEY,

    some_timestamp timestamp

);






cqlsh:dev> select * from my_table;




 partition_key | some_timestamp

---------------+--------------------------

           key | 2015-12-25 18:30:00+0000

           foo | 2018-04-11 06:22:29+0000









Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login










    










                                                                                
           

On Sun, Apr 15, 2018 at 11:36 AM, Dinesh Joshi <dinesh.jo...@yahoo.com> wrote:

Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried 
using a pre-existing codec? It would be helpful if you can give more 
information.
Dinesh    On Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy 
<revanthkuma...@gmail.com> wrote:  
 
 

Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my 
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another 
while reading from cassandra. for example i have a timestamp column in 
cassandra table ,but i want to read it as long and save as parquet file to HDFS 
, then this cutom codecs will be usefull instead of doing an additional 
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
 I had tried with solution provided in the below link but it does not work , 
any ideas around this will be helpfull.
https://stackoverflow.com/ questions/40363611/adding- custom-codec-to- 
cassandraconnector/49750791? noredirect=1#comment86515847_ 49750791



Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/ crazyusers/login










    










                                                                                
           
  

  

Reply via email to