Vedran Ljubovic created IGNITE-19458:
----------------------------------------
Summary: Kafka Connect IgniteSinkConnector multiple topics &
caches do not work
Key: IGNITE-19458
URL: https://issues.apache.org/jira/browse/IGNITE-19458
Project: Ignite
Issue Type: Bug
Components: extensions
Affects Versions: 2.15
Reporter: Vedran Ljubovic
We are using Kafka Connect to stream 4 topics into 4 separate caches. We are
presently forced to use 4 separate instances of Kafka Connect because when 4
connectors are passed to one instance all messages will be streamed into the
same (first) Ignite cache. This works, but in the future we plan to add more
topics/caches and this will affect memory usage as well as our ability to
distribute workload in a cluster.
I think that the reason for this is in IgniteSinkTask.java where
IgniteDataStreamer is declared as private static final attribute of inner
class, which means that it will be initialized with the first passed cache and
cannot be changed later.
I'm not sure which is the best approach performance wise, but possible
solutions would be:
* It should be non-static attribute of IgniteSinkTask (different streamer for
each task)
* It should be created in IgniteSinkConnector and passed to IgniteSinkTask
(one streamer per connector)
* Have a Map of cacheNames to Streamers which ensures that only as many
Streamers will be created as there are different caches used.
Also I notice that cacheName, SingleTupleExtractor, stopped boolean, are all
declared as static which cause various problems when using multiple connectors
(messages being processed with wrong SimpleTupleExtractor etc.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
