Ronak Thakrar created FLINK-13479:
-------------------------------------
Summary: Cassandra POJO Sink - Prepared Statement query does not
have deterministic ordering of columns - causing prepared statement cache
overflow
Key: FLINK-13479
URL: https://issues.apache.org/jira/browse/FLINK-13479
Project: Flink
Issue Type: Improvement
Components: Connectors / Cassandra
Reporter: Ronak Thakrar
While using Cassandra POJO Sink as part of Flink Jobs - prepared statements
query string which is automatically generated while inserting the data(using
Mapper.saveQuery method), Cassandra entity does not have deterministic ordering
enforced-so every time column position is changed a new prepared statement is
generated and used. As an effect of that prepared statement query cache is
overflown because every time when insert statement query string is generated by
- columns are in random order.
Following is the detailed explanation for what happens inside the Datastax java
driver([https://datastax-oss.atlassian.net/browse/JAVA-1587]):
The current Mapper uses random ordering of columns when it creates prepared
queries. This is fine when only 1 java client is accessing a cluster (and
assuming the application developer does the correct thing by re-using a
Mapper), since each Mapper will reused prepared statement. However when you
have many java clients accessing a cluster, they will each create their own
permutations of column ordering, and can thrash the prepared statement cache on
the cluster.
I propose that the Mapper uses a TreeMap instead of a HashMap when it builds
its set of AliasedMappedProperty - sorted by the column name
(col.mappedProperty.getMappedName()). This would create a deterministic
ordering of columns, and all java processes accessing the same cluster would
end up with the same prepared queries for the same entities.
This issue is already fixed in the Datastax java driver update version(3.3.1)
which is not used by Flink Cassandra connector (using 3.0.0).
I upgraded the driver version to 3.3.1 locally in Flink Cassandra connector and
tested, it stopped creating new prepared statements with different ordering of
column for the same entity. I have the fix for this issue and would like to
contribute the change and will raise the PR request for the same.
Flink Cassandra Connector Version: flink-connector-cassandra_2.11
Flink Version: 1.7.1
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
