[
https://issues.apache.org/jira/browse/FLINK-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17518077#comment-17518077
]
Etienne Chauchot edited comment on FLINK-26793 at 4/7/22 4:14 PM:
------------------------------------------------------------------
[~bumblebee], in the meantime, I made a [pure-flink test
pipeline|https://github.com/echauchot/flink-samples/blob/master/src/main/java/org/example/CassandraPojoSinkStreamingExample.java]
that runs in streaming mode (so it is infinite) with a fake source that
provides Pojos every 100 ms and write them to Cassandra. I've run this pipeline
locally for 2 hours with no perf log messages. I'll try to run it for 72 hours
or more on my Azure environment (hoping that I can get a Cassandra instance
there). That way, I'll try to reproduce the performance issue logs you had. If
I can't reproduce after some days of running this pipeline, I think the good
way to proceed is to take a look at your Flink restore stats hence my question
above or take a look at the Scylla connector you use.
was (Author: echauchot):
[~bumblebee], in the meantime, I made a [pure-flink test
pipeline|https://github.com/echauchot/flink/blob/FLINK-26793-cassandra-perf/flink-connectors/flink-connector-cassandra/src/test/java/org/apache/flink/streaming/connectors/cassandra/example/CassandraPojoSinkStreamingExample.java]
that runs in streaming mode (so it is infinite) with a fake source that
provides Pojos every 100 ms and write them to Cassandra. I've run this pipeline
locally for 2 hours with no perf log messages. I'll try to run it for 72 hours
or more on my Azure environment (hoping that I can get a Cassandra instance
there). That way, I'll try to reproduce the performance issue logs you had. If
I can't reproduce after some days of running this pipeline, I think the good
way to proceed is to take a look at your Flink restore stats hence my question
above or take a look at the Scylla connector you use.
> Flink Cassandra connector performance issue
> --------------------------------------------
>
> Key: FLINK-26793
> URL: https://issues.apache.org/jira/browse/FLINK-26793
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Cassandra
> Affects Versions: 1.14.4
> Reporter: Jay Ghiya
> Assignee: Etienne Chauchot
> Priority: Major
>
> A warning is observed during long runs of flink job stating “Insertions into
> scylla might be suffering. Expect performance problems unless this is
> resolved.”
> Upon initial analysis - “flink cassandra connector is not keeping instance of
> mapping manager that is used to convert a pojo to cassandra row. Ideally the
> mapping manager should have the same life time as cluster and session objects
> which are also created once when the driver is initialized”
> Reference:
> https://stackoverflow.com/questions/59203418/cassandra-java-driver-warning
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
