Hi Revanth,
I took a quick look and don't think you can override existing mappings. You
should see a warning like this -
18/04/15 19:50:27 WARN CodecRegistry: Ignoring codec CustomTSTypeCodec [date
<-> java.lang.Long] because it collides with previously registered codec
CustomTSTypeCodec [date <-> java.lang.Long]
The issue is that CodecRegistry does not allow you to override existing type
mappings[1][2]. You can however register a codec that converts between a
CqlType and your custom class.
Dinesh
[1]
https://docs.datastax.com/en/developer/java-driver/3.1/manual/custom_codecs/[2]
https://github.com/datastax/java-driver/blob/3.x/driver-core/src/main/java/com/datastax/driver/core/CodecRegistry.java#L357
On Sunday, April 15, 2018, 1:06:53 AM PDT, Revanth Reddy
wrote:
Hi Dinesh,
Thanks a lot for your response . I had tried with the existing codec
(SimpleTimestampCodec) ,but it says the codec has been registered but it is not
converting the timestamp column to long . also i don't observe any error.
Below are the steps that i had followed and also attached the code that i had
used:
I had used this property to set the codec in spark
config("spark.cassandra.connection.factory",
"com.rasa.devops.cassandra.export.CustomConnectionFactory")
Since there is no direct way to refer the custom codec in spark , i had
followed the above approach.
I had used the following dependencies :
com.datastax.spark
spark-cassandra-connector_2.11
2.0.1
com.datastax.cassandra
cassandra-driver-extras 3.1.4
joda-time
joda-time 2.9.1
Cassandra table used for testing :
CREATE TABLE dev.my_table (
partition_key text PRIMARY KEY,
some_timestamp timestamp
);
cqlsh:dev> select * from my_table;
partition_key | some_timestamp
---+--
key | 2015-12-25 18:30:00+
foo | 2018-04-11 06:22:29+
Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/crazyusers/login
On Sun, Apr 15, 2018 at 11:36 AM, Dinesh Joshi wrote:
Hi Revanth,
How do you register the custom codec? Do you get any errors? Have you tried
using a pre-existing codec? It would be helpful if you can give more
information.
DineshOn Saturday, April 14, 2018, 7:29:30 PM PDT, Revanth Reddy
wrote:
Hi Team ,
I want to write a custom cassandra codec and i want to use that codec in my
spark application while reading the data from cassandra table .
Basically the custom codecs are used to convert one column type to another
while reading from cassandra. for example i have a timestamp column in
cassandra table ,but i want to read it as long and save as parquet file to HDFS
, then this cutom codecs will be usefull instead of doing an additional
transformation in the spark code.
I want to know how to set/register this custom cassandra codecs in spark ?
I had tried with solution provided in the below link but it does not work ,
any ideas around this will be helpfull.
https://stackoverflow.com/ questions/40363611/adding- custom-codec-to-
cassandraconnector/49750791? noredirect=1#comment86515847_ 49750791
Thanks & RegardsS.Revanth kumar reddyhttps://github.com/Re1tReddy/
http://34.230.100.5:8080/ crazyusers/login