Hi Randall, Many thanks for your and Gwen's help with this - it's very reassuring that help is at hand in such circumstances :) All the best, David
-----Original Message----- From: Randall Hauch [mailto:rha...@gmail.com] Sent: 02 May 2017 21:01 To: dev@kafka.apache.org Subject: Re: Kafka Connect and Partitions Hi, David. Excellent. I'm glad that you've solved the puzzle. Best regards, Randall On Tue, May 2, 2017 at 9:18 AM, <david.frank...@bt.com> wrote: > Hi Gwen/Randall, > > I think I've finally understood, more or less, how partitioning relates to > SourceRecords. > > Because I was using the SourceRecord constructor that doesn't provide > values for key and key schema, the key is null. The DefaultPartioner > appears to map null to a constant value rather than round-robin across all > of the partitions :( > SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> > sourceOffset, String topic, Schema valueSchema, Object value) > > Another SourceRecord constructor enables the partition to be specified but > I'd prefer not to use this as I don't want to couple the non-Kafka source > side to Kafka by making it aware of topic partitions - this would also > presumably involve coupling it to the Cluster so that the number of > partitions in a topic can be determined :( > SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> > sourceOffset, String topic, Integer partition, Schema keySchema, Object > key, Schema valueSchema, Object value) > > Instead, if I use the SourceRecord constructor that also takes arguments > for the key and key schema (making them take the same values as the value > and value schema in my application), then the custom partitioner / > producer.partitioner.class property is not required and the data is > distributed across the partitions :) > SourceRecord(Map<String, ?> sourcePartition, Map<String, ?> > sourceOffset, String topic, Integer partition, Schema keySchema, Object > key, Schema valueSchema, Object value) > > Many thanks once again for your guidance. I think this puzzle is now > solved :) > Best wishes, > David > > -----Original Message----- > From: Randall Hauch [mailto:rha...@gmail.com] > Sent: 28 April 2017 16:08 > To: dev@kafka.apache.org > Subject: Re: Kafka Connect and Partitions > > The source connector creates SourceRecord object and can set a number of > fields, including the message's key and value, the Kafka topic name and, if > desired, the Kafka topic partition number. If the connector does se the the > topic partition to a non-null value, then that's the partition to which > Kafka Connect will write the message; otherwise, the customer partitioner > (e.g., your custom partitioner) used by the Kafka Connect producer will > choose/compute the partition based purely upon the key and value byte > arrays. Note that if the connector doesn't set the topic partition number > and no special producer partitioner is specified, the default hash-based > Kafka partitioner will be used. > > So, the connector can certainly set the topic partition number, and it may > be easier to do it there since the connector actually has the key and > values before they are serialized. But no matter what, the connector is the > only thing that sets the message key in the source record. > > BTW, the SourceRecord's "source position" and "source offset" are actually > the connector-defined information about the source and where the connector > has read in that source. Don't confuse these with the topic name or topic > partition number. > > Hope that helps. > > Randall > > On Fri, Apr 28, 2017 at 7:15 AM, <david.frank...@bt.com> wrote: > > > Hi Gwen, > > > > Having added a custom partitioner (via the producer.partitioner.class > > property in worker.properties) that simply randomly selects a partition, > > the data is now written evenly across all the partitions :) > > > > The root of my confusion regarding why the default partitioner writes all > > data to the same partition is that I don't understand how the > SourceRecords > > returned in the source task poll() method are used by the partitioner. > The > > data that is passed to the partitioner includes a key Object (which is an > > empty byte array - presumably this is a bad idea!), and a value Object > > (which is a non-empty byte array): > > > > @Override > > public int partition(String topic, Object key, byte[] keyBytes, > Object > > value, byte[] valueBytes, Cluster cluster) { > > System.out.println(String.format( > > "### PARTITION key[%s][%s][%d] value[%s][%s][%d]", > > key, key.getClass().getSimpleName(), keyBytes.length, > > value, value.getClass().getSimpleName(), > > valueBytes.length)); > > > > => > > ### PARTITION key[[B@584f599f][byte[]][0] value[[B@73cc0cd8][byte[]][ > 236] > > > > However, I don't understand how the above key and value are derived from > > the SourceRecord attributes which, in my application's case, is as > follows: > > > > events.add(new SourceRecord( > > offsetKey(filename), > > offsetValue(++recordIndex), > > topicName, > > Schema.BYTES_SCHEMA, > > line)); > > System.out.println(String.format( > > "### PARTITION SourceRecord key[%s] value[%s] > > topic[%s] schema[%s], data[%s]", > > offsetKey(filename), > offsetValue(recordIndex), > > topicName, Schema.BYTES_SCHEMA, line)); > > > > => > > ### PARTITION SourceRecord key[{_taskFiles=e:\a\b\c\d.ser}] > > value[{_position=1}] topic[Topic1] schema[Schema{BYTES}], data[{"field1": > > value1, …, "fieldN": valueN}] > > > > In worker.properties I use the key.converter and value.converter > > properties to apply an Avro converter to the data written to Kafka. > Hence, > > I assume, the byte[] format of the key and the value. Though I don't > > understand why the key is empty and this, presumably, is why all data is > > mapped to the same Kafka partition. > > > > Could you explain how the SourceRecord is used to derive the partition > key > > please. Can you see from the above summary why the partition key is > null? > > It defeats me :( > > > > Have a good weekend, thanks, > > > > David > > > > -----Original Message----- > > From: Gwen Shapira [mailto:g...@confluent.io] > > Sent: 27 April 2017 17:44 > > To: dev@kafka.apache.org > > Subject: Re: Kafka Connect and Partitions > > > > That's great! So we tracked this down to the source connector not > properly > > partitioning data. > > > > Do you set both key and value? It sounds a bit like maybe all your > records > > have the exact same key, which means they all get hashed to the same > > partition. Can you check that? > > > > On Thu, Apr 27, 2017 at 3:22 AM, <david.frank...@bt.com> wrote: > > > > > Hi Gwen, > > > > > > Many thanks for your much appreciated offer to help with this. > > > > > > In answer to your questions: > > > * Are you writing a connector or trying to use an existing one? > > > I'm writing a new source/sink connector pipeline: folderToTopics piped > > > into topicsToFolders. > > > * Is the connector reading from the topic you think you are reading? > > > Yes > > > * Do you actually have 4 tasks? Are they all running? Are there errors? > > > Yes, Yes, No (see log output below) > > > * What happens if you stop the only task doing the work? > > > I'm not sure how to do this but am confident that the single effective > > > task does actually complete the job correctly. > > > * Is the one task subscribed to all partitions? How did you check that? > > > I don't think it is. In the log output below, the line tagged > > > 'Setting newly assigned partitions' shows the topics and their > > > partitions; the lines tagged 'Kafka Offset/Partition' show the > > > partition that contains the data read. > > > As you will see, only partition 31 is ever read, (interestingly) for > > > all topics. > > > * Do you have data in all 50 partitions? > > > No - only partition 31 contains any data, which therefore explains the > > > 'topicsToFolders' sink task behaviour. > > > This points the finger of suspicion at the 'foldersToTopics' source > task. > > > This may be the root of the problem but I'm not clear why the > > > DefaultPartitioner doesn't write the data across all partitions. > > > To simplify things, I've reduced my setup to 2 producer tasks, reading > > > data from files to generate events from, and 2 consumer tasks that > > > read events and serialize them to another set of files. > > > As it stands it's a perverse file copy! > > > * Anything interesting in the log? > > > Not that I can see. > > > > > > > > > I ran the following command across several of the topics and each gave > > > the same result: > > > only partition 31 contains any data! > > > $ ./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list > > > localhost:9092 -topic topicN --time -1 --offsets 5 > > > topicN:23:0 > > > topicN:32:0 > > > topicN:41:0 > > > topicN:17:0 > > > topicN:8:0 > > > topicN:44:0 > > > topicN:35:0 > > > topicN:26:0 > > > topicN:11:0 > > > topicN:29:0 > > > topicN:47:0 > > > topicN:38:0 > > > topicN:20:0 > > > topicN:2:0 > > > topicN:5:0 > > > topicN:14:0 > > > topicN:46:0 > > > topicN:40:0 > > > topicN:49:0 > > > topicN:13:0 > > > topicN:4:0 > > > topicN:31:91169 > > > topicN:22:0 > > > topicN:16:0 > > > topicN:7:0 > > > topicN:43:0 > > > topicN:25:0 > > > topicN:34:0 > > > topicN:10:0 > > > topicN:37:0 > > > topicN:1:0 > > > topicN:28:0 > > > topicN:19:0 > > > topicN:45:0 > > > topicN:36:0 > > > topicN:27:0 > > > topicN:9:0 > > > topicN:18:0 > > > topicN:21:0 > > > topicN:48:0 > > > topicN:3:0 > > > topicN:12:0 > > > topicN:30:0 > > > topicN:39:0 > > > topicN:15:0 > > > topicN:42:0 > > > topicN:24:0 > > > topicN:33:0 > > > topicN:6:0 > > > topicN:0:0 > > > > > > > > > Below is an edited version of the log output when I launch the Kafka > > > Connect source and sink connectors/tasks. > > > > > > Source Task [2134126231] retrieves 500 events, in 5 batches. > > > Source Task [1714504927] retrieves 500 events, in 5 batches. > > > > > > Sink Task [518962883] {id=topicsToFolders-0} processes none of the > > events. > > > Sink Task [602953950] {id=topicsToFolders-1} processes all of the 1000 > > > events that are written by the source tasks. > > > > > > It seems to me that the consumer tasks are behaving reasonably given > > > that all data for all topics is in a single partition. > > > But I don't understand why the producer tasks write to a single > > partition. > > > Does it suggest that the DefaultPartitioner is generating the same > > > hash value for the data I'm providing it and that I need to introduce > > > a custom partitioner? > > > > > > Once again, many thanks for your help in resolving this. > > > > > > Best wishes, > > > David > > > > > > > > > > > > offset.flush.timeout.ms = 10000 > > > offset.flush.interval.ms = 1000 > > > rest.port = 8084 > > > [41:24,897] INFO Kafka Connect starting (org.apache.kafka.connect. > > > runtime.Connect:52) > > > [41:24,897] INFO Herder starting (org.apache.kafka.connect. > > > runtime.standalone.StandaloneHerder:71) > > > [41:24,897] INFO Worker starting (org.apache.kafka.connect. > > > runtime.Worker:102) > > > [41:24,912] INFO ProducerConfig values: > > > partitioner.class = class org.apache.kafka.clients. > > > producer.internals.DefaultPartitioner > > > [41:24,990] INFO ProducerConfig values: > > > partitioner.class = class org.apache.kafka.clients. > > > producer.internals.DefaultPartitioner > > > [41:24,990] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common. > > > utils.AppInfoParser:83) > > > [41:24,990] INFO Starting FileOffsetBackingStore with file > > > \tmp\kafkaConnect\foldersToTopicsToFolders.offsets > > > (org.apache.kafka.connect.storage.FileOffsetBackingStore:60) > > > [41:25,006] INFO Worker started (org.apache.kafka.connect. > > > runtime.Worker:124) > > > [41:25,006] INFO Herder started (org.apache.kafka.connect. > > > runtime.standalone.StandaloneHerder:73) > > > [41:25,006] INFO Starting REST server (org.apache.kafka.connect. > > > runtime.rest.RestServer:98) > > > > > > [41:25,600] INFO Started > > > o.e.j.s.ServletContextHandler@14d14731{/,null,AVAILABLE} > > > (org.eclipse.jetty.server.handler.ContextHandler:744) > > > [41:25,615] INFO Started > > > ServerConnector@793138bd{HTTP/1.1}{0.0.0.0:8084} > > > (org.eclipse.jetty.server.ServerConnector:266) > > > [41:25,615] INFO Started @1365ms (org.eclipse.jetty.server.Server:379) > > > [41:25,615] INFO REST server listening at xxxx, advertising URL xxxx > > > (org.apache.kafka.connect.runtime.rest.RestServer:150) > > > [41:25,615] INFO Kafka Connect started (org.apache.kafka.connect. > > > runtime.Connect:58) > > > [41:25,615] INFO ConnectorConfig values: > > > connector.class = c.b.b.pp.kafkaConnect.connectors. > > > FoldersToTopicsSourceConnector > > > tasks.max = 2 > > > name = foldersToTopics > > > [41:25,631] INFO Creating connector foldersToTopics of type > > > c.b.b.pp.kafkaConnect.connectors.FoldersToTopicsSourceConnector > > > (org.apache.kafka.connect.runtime.Worker:168) > > > [41:25,631] INFO Instantiated connector foldersToTopics with version > > > 0.10.0.1 of type > > > c.b.b.pp.kafkaConnect.connectors.FoldersToTopicsSourceConnector > > > (org.apache.kafka.connect.runtime.Worker:176) > > > [41:25,631] INFO ### FoldersToTopicsSourceConnector is monitoring > > > inbox [e:/events/sensor-103] and has [2] tasks (c.b.b.pp.kafkaConnect. > > > connectors.FoldersToTopicsSourceConnector:55) > > > [41:25,631] INFO Finished creating connector foldersToTopics > > > (org.apache.kafka.connect.runtime.Worker:181) > > > [41:25,631] INFO SourceConnectorConfig values: > > > connector.class = c.b.b.pp.kafkaConnect.connectors. > > > FoldersToTopicsSourceConnector > > > tasks.max = 2 > > > name = foldersToTopics > > > > > > [41:26,162] INFO ### FileLister found [1000] files under > > > [~eventSource103] > > > (c.b.b.pp.InboxMonitor:89) > > > [41:26,162] INFO ### Files in inbox [~eventSource103] have changed to > > > [1000] files - requesting task reconfiguration > > > (c.b.b.pp.InboxMonitor:61) [41:26,162] INFO ### inbox > > > [~eventSource103] contains [1000] files > > > (c.b.b.pp.kafkaConnect.connectors.FoldersToTopicsSourceConnector:76) > > > > > > [41:26,162] INFO Creating task foldersToTopics-0 > > (org.apache.kafka.connect. > > > runtime.Worker:315) > > > [41:26,162] INFO Instantiated task foldersToTopics-0 with version > > > 0.10.0.1 of type c.b.b.pp.kafkaConnect.tasks.FoldersToTopicsSourceTask > > > (org.apache.kafka.connect.runtime.Worker:326) > > > > > > [41:26,162] INFO Creating task foldersToTopics-1 > > (org.apache.kafka.connect. > > > runtime.Worker:315) > > > [41:26,162] INFO Instantiated task foldersToTopics-1 with version > > > 0.10.0.1 of type c.b.b.pp.kafkaConnect.tasks.FoldersToTopicsSourceTask > > > (org.apache.kafka.connect.runtime.Worker:326) > > > [41:26,162] INFO Created connector foldersToTopics > > > (org.apache.kafka.connect.cli.ConnectStandalone:91) > > > > > > [41:26,162] INFO ConnectorConfig values: > > > connector.class = c.b.b.pp.kafkaConnect.connectors. > > > TopicsToFoldersSinkConnector > > > tasks.max = 2 > > > name = topicsToFolders > > > (org.apache.kafka.connect.runtime.ConnectorConfig:178) > > > [41:26,162] INFO Creating connector topicsToFolders of type > > > c.b.b.pp.kafkaConnect.connectors.TopicsToFoldersSinkConnector > > > (org.apache.kafka.connect.runtime.Worker:168) > > > [41:26,162] INFO Instantiated connector topicsToFolders with version > > > 0.10.0.1 of type > > > c.b.b.pp.kafkaConnect.connectors.TopicsToFoldersSinkConnector > > > (org.apache.kafka.connect.runtime.Worker:176) > > > [41:26,162] INFO Finished creating connector topicsToFolders > > > (org.apache.kafka.connect.runtime.Worker:181) > > > [41:26,162] INFO SinkConnectorConfig values: > > > connector.class = c.b.b.pp.kafkaConnect.connectors. > > > TopicsToFoldersSinkConnector > > > tasks.max = 2 > > > topics = [topic1, topic2, topic3, topic4, topic5, topic6, > > > topic7, topic8, topic9, topic10, topic11] > > > name = topicsToFolders > > > [41:26,162] INFO ### Max tasks [2] (c.b.b.pp.kafkaConnect.connectors. > > > TopicsToFoldersSinkConnector:54) > > > [41:26,162] INFO ### Topics for task > > > [topic1,topic2,topic3,topic4,topic5,topic6] > > > (c.b.b.pp.kafkaConnect.connectors.TopicsToFoldersSinkConnector:61) > > > [41:26,162] INFO ### Topics for task > > > [topic7,topic8,topic10,topic9,topic11] > > > (c.b.b.pp.kafkaConnect.connectors.TopicsToFoldersSinkConnector:61) > > > > > > [41:26,162] INFO Creating task topicsToFolders-0 > > (org.apache.kafka.connect. > > > runtime.Worker:315) > > > [41:26,162] INFO Instantiated task topicsToFolders-0 with version > > > 0.10.0.1 of type c.b.b.pp.kafkaConnect.tasks.TopicsToFoldersSinkTask > > > (org.apache.kafka.connect.runtime.Worker:326) > > > [41:26,178] INFO ConsumerConfig values: > > > partition.assignment.strategy = [org.apache.kafka.clients. > > > consumer.RangeAssignor] > > > max.partition.fetch.bytes = 1048576 > > > max.poll.records = 1000 > > > session.timeout.ms = 30000 > > > > > > [41:26,209] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common. > > > utils.AppInfoParser:83) > > > [41:26,209] INFO Kafka commitId : a7a17cdec9eaa6c5 > > > (org.apache.kafka.common.utils.AppInfoParser:84) > > > [41:26,209] INFO TaskConfig values: > > > task.class = class c.b.b.pp.kafkaConnect.tasks. > > > TopicsToFoldersSinkTask > > > (org.apache.kafka.connect.runtime.TaskConfig:178) > > > > > > [41:26,209] INFO Creating task topicsToFolders-1 > > (org.apache.kafka.connect. > > > runtime.Worker:315) > > > [41:26,209] INFO Instantiated task topicsToFolders-1 with version > > > 0.10.0.1 of type c.b.b.pp.kafkaConnect.tasks.TopicsToFoldersSinkTask > > > (org.apache.kafka.connect.runtime.Worker:326) > > > [41:26,209] INFO ConsumerConfig values: > > > partition.assignment.strategy = [org.apache.kafka.clients. > > > consumer.RangeAssignor] > > > max.partition.fetch.bytes = 1048576 > > > max.poll.records = 1000 > > > session.timeout.ms = 30000 > > > > > > [41:26,225] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common. > > > utils.AppInfoParser:83) > > > > > > [41:26,225] INFO ### Task [518962883] is (ideally) assigned topics > > > [topic1,topic2,topic3,topic4,topic5,topic6] BUT ACTUALLY IT ISN'T!!! > > > (c.b.b.pp.kafkaConnect.tasks.TopicsToFoldersSinkTask:49) > > > [41:26,225] INFO Sink task WorkerSinkTask{id=topicsToFolders-0} > > > finished initialization and start (org.apache.kafka.connect. > > > runtime.WorkerSinkTask:208) > > > [41:26,225] INFO Created connector topicsToFolders > > > (org.apache.kafka.connect.cli.ConnectStandalone:91) > > > > > > [41:26,225] INFO ### Task [602953950] is (ideally) assigned topics > > > [topic7,topic8,topic10,topic9,topic11] BUT ACTUALLY IT ISN'T!!! > > > (c.b.b.pp.kafkaConnect.tasks.TopicsToFoldersSinkTask:49) > > > [41:26,225] INFO Sink task WorkerSinkTask{id=topicsToFolders-1} > > > finished initialization and start (org.apache.kafka.connect. > > > runtime.WorkerSinkTask:208) > > > > > > [41:26,240] INFO Source task WorkerSourceTask{id=foldersToTopics-1} > > > finished initialization and start (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:138) > > > [41:26,240] INFO Source task WorkerSourceTask{id=foldersToTopics-0} > > > finished initialization and start (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:138) > > > [41:26,365] INFO Discovered coordinator localhost:9092 (id: 2147483647 > > > rack: null) for group connect-topicsToFolders. > (org.apache.kafka.clients. > > > consumer.internals.AbstractCoordinator:505) > > > [41:26,365] INFO Discovered coordinator localhost:9092 (id: 2147483647 > > > rack: null) for group connect-topicsToFolders. > (org.apache.kafka.clients. > > > consumer.internals.AbstractCoordinator:505) > > > [41:26,365] INFO Revoking previously assigned partitions [] for group > > > connect-topicsToFolders (org.apache.kafka.clients.consumer.internals. > > > ConsumerCoordinator:292) > > > [41:26,365] INFO Revoking previously assigned partitions [] for group > > > connect-topicsToFolders (org.apache.kafka.clients.consumer.internals. > > > ConsumerCoordinator:292) > > > [41:26,365] INFO (Re-)joining group connect-topicsToFolders > > > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:326) > > > [41:26,365] INFO (Re-)joining group connect-topicsToFolders > > > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:326) > > > [41:26,365] INFO (Re-)joining group connect-topicsToFolders > > > (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:326) > > > [41:26,381] INFO Successfully joined group connect-topicsToFolders > > > with generation 2 (org.apache.kafka.clients.consumer.internals. > > > AbstractCoordinator:434) > > > [41:26,381] INFO Successfully joined group connect-topicsToFolders > > > with generation 2 (org.apache.kafka.clients.consumer.internals. > > > AbstractCoordinator:434) > > > [41:26,381] INFO Setting newly assigned partitions [topic7-18, > > > topic2-11, topic11-17, topic8-0, topic1-11, topic4-15, topic5-17, > > > topic6-0, topic7-2, topic10-16, topic11-1, topic8-16, topic6-16, > > > topic9-9, topic5-0, topic3-12, topic10-0, topic7-19, topic2-12, > > > topic6-1, topic1-12, topic11-16, topic8-1, topic4-14, topic9-24, > > > topic5-16, topic8-18, topic10-17, topic7-3, topic6-17, topic11-0, > > > topic8-17, topic9-8, topic3-11, topic10-1, topic4-17, topic7-20, > > > topic5-15, topic6-14, topic9-23, topic10-18, topic4-1, topic3-14, > > > topic7-4, topic8-14, topic11-15, topic1-13, topic2-9, topic9-7, > > > topic10-2, topic4-16, topic7-21, topic5-14, topic9-22, topic10-19, > > > topic4-0, topic6-15, topic7-5, topic8-15, topic1-14, topic11-14, > > > topic3-13, topic2-10, topic9-6, topic10-3, topic7-22, topic2-15, > > > topic3-0, topic5-13, topic6-12, topic9-21, topic8-12, topic7-6, > > > topic3-16, topic11-13, topic1-15, topic4-11, topic9-5, topic10-20, > > > topic7-23, topic2-16, topic6-13, topic1-0, topic9-20, topic10-4, > > > topic5-12, topic8-13, topic7-7, topic3-15, topic2-0, topic1-16, > > > topic11-12, topic7-24, topic4-10, topic10-21, topic9-4, topic2-13, > > > topic3-2, topic9-19, topic7-8, topic10-5, topic5-11, topic6-10, > > > topic1-1, topic3-18, topic9-3, topic8-10, topic4-13, topic11-11, > > > topic10-22, topic1-17, topic2-14, topic9-18, topic3-1, topic6-11, > > > topic7-9, topic1-2, topic10-6, topic5-10, topic9-2, topic8-11, > > > topic3-17, topic4-12, topic10-23, topic1-18, topic11-10, topic9-17, > > > topic3-4, topic7-10, topic2-19, topic1-3, topic10-7, topic4-23, > > > topic5-9, topic6-8, topic9-1, topic5-8, topic3-20, topic10-24, > > > topic2-3, topic11-9, topic8-8, topic1-19, topic4-7, topic6-24, > > > topic9-16, topic5-24, topic3-3, topic10-8, topic7-11, topic2-20, > > > topic6-9, topic1-4, topic11-24, topic4-22, topic5-7, topic9-0, > > > topic3-19, topic2-4, topic1-20, topic11-8, topic8-9, topic4-6, > > > topic5-23, topic6-6, topic11-23, topic1-5, topic2-17, topic9-15, > > > topic10-9, topic3-23, topic7-12, topic8-23, topic6-22, topic11-7, > > > topic5-6, topic1-21, topic2-1, topic4-9, topic3-6, topic8-6, topic1-6, > > > topic11-22, topic5-22, topic3-5, topic2-18, topic9-14, topic10-10, > > > topic8-24, topic3-22, topic4-24, topic6-7, topic7-13, topic5-5, > > > topic1-22, topic11-6, topic3-21, topic2-2, topic4-8, topic6-23, > > > topic8-7, topic11-21, topic1-7, topic4-19, topic5-21, topic6-4, > > > topic9-13, topic8-21, topic10-11, topic7-14, topic2-23, topic11-5, > > > topic1-23, topic4-3, topic6-20, topic10-12, topic5-4, topic8-4, > > > topic2-7, topic3-8, topic6-5, topic1-8, topic11-20, topic4-18, > > > topic9-12, topic5-20, topic7-15, topic8-22, topic3-24, topic2-24, > > > topic6-21, topic1-24, topic11-4, topic4-2, topic10-13, topic5-3, > > > topic8-5, topic3-7, topic2-8, topic4-21, topic7-16, topic5-19, > > > topic6-2, topic11-19, topic1-9, topic2-21, topic8-19, topic4-5, > > > topic7-0, topic6-18, topic11-3, topic10-14, topic5-2, topic2-5, > > > topic3-10, topic9-11, topic8-2, topic4-20, topic6-3, topic7-17, > > > topic1-10, topic11-18, topic5-18, topic8-20, topic2-22, topic4-4, > > > topic6-19, topic7-1, topic10-15, topic5-1, topic11-2, topic2-6, > > > topic9-10, topic8-3, topic3-9] for group connect-topicsToFolders > > > (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:231) > > > [41:26,381] INFO Setting newly assigned partitions [topic10-32, > > > topic2-44, topic6-33, topic1-44, topic4-48, topic8-33, topic9-25, > > > topic3-29, topic10-49, topic7-35, topic2-28, topic6-49, topic4-32, > > > topic8-49, topic1-27, topic5-33, topic11-34, topic9-42, topic3-45, > > > topic10-33, topic2-45, topic11-49, topic6-34, topic4-47, topic5-49, > > > topic3-28, topic1-29, topic2-29, topic11-33, topic1-28, topic7-36, > > > topic4-31, topic9-41, topic5-32, topic8-34, topic3-44, topic1-45, > > > topic10-34, topic11-48, topic3-31, topic6-31, topic8-31, topic5-48, > > > topic11-31, topic1-30, topic2-25, topic4-34, topic11-32, topic3-47, > > > topic6-47, topic7-37, topic8-47, topic5-31, topic9-40, topic6-30, > > > topic1-46, topic2-42, topic10-35, topic2-43, topic4-49, topic8-32, > > > topic3-30, topic6-32, topic5-47, topic1-31, topic11-30, topic2-26, > > > topic2-27, topic7-38, topic4-33, topic8-48, topic3-46, topic6-48, > > > topic1-47, topic11-47, topic5-30, topic9-39, topic8-29, topic2-48, > > > topic3-33, topic6-45, topic11-29, topic1-32, topic4-27, topic5-46, > > > topic10-36, topic8-45, topic7-39, topic2-32, topic3-49, topic1-48, > > > topic4-44, topic5-29, topic6-28, topic11-46, topic9-38, topic2-49, > > > topic8-30, topic3-32, topic6-46, topic11-28, topic7-40, topic5-45, > > > topic4-26, topic10-37, topic1-33, topic2-33, topic8-46, topic3-48, > > > topic6-29, topic11-45, topic4-43, topic9-37, topic5-28, topic1-49, > > > topic2-46, topic3-35, topic8-27, topic4-46, topic4-29, topic6-43, > > > topic7-41, topic5-44, topic11-27, topic1-34, topic10-38, topic2-30, > > > topic8-43, topic4-30, topic11-44, topic7-25, topic5-27, topic9-36, > > > topic6-26, topic8-28, topic3-34, topic2-47, topic7-42, topic4-28, > > > topic6-44, topic5-43, topic1-35, topic10-39, topic11-26, topic8-44, > > > topic9-35, topic2-31, topic7-26, topic4-45, topic6-27, topic11-43, > > > topic5-26, topic3-37, topic10-40, topic7-43, topic6-41, topic11-25, > > > topic1-36, topic8-25, topic5-42, topic9-34, topic7-27, topic2-36, > > > topic4-40, topic8-41, topic5-25, topic11-42, topic9-49, topic8-26, > > > topic3-36, topic1-37, topic10-41, topic6-42, topic7-44, topic5-41, > > > topic9-33, topic5-40, topic8-42, topic10-25, topic2-37, topic6-25, > > > topic11-41, topic7-28, topic4-39, topic9-48, topic1-38, topic10-42, > > > topic4-25, topic6-39, topic7-45, topic5-39, topic9-32, topic2-34, > > > topic10-26, topic4-42, topic11-40, topic3-39, topic7-29, topic8-39, > > > topic1-39, topic9-47, topic10-43, topic7-46, topic6-40, topic11-39, > > > topic5-38, topic10-44, topic9-31, topic10-27, topic2-35, topic7-30, > > > topic4-41, topic8-40, topic3-38, topic6-37, topic1-40, topic10-28, > > > topic9-46, topic7-47, topic3-25, topic4-36, topic5-37, topic11-38, > > > topic10-45, topic9-30, topic8-37, topic7-31, topic2-40, topic3-41, > > > topic6-38, topic7-48, topic9-45, topic10-29, topic1-41, topic11-37, > > > topic7-32, topic4-35, topic9-29, topic5-36, topic10-46, topic2-41, > > > topic8-38, topic3-40, topic6-35, topic7-49, topic9-44, topic1-42, > > > topic10-30, topic3-27, topic9-27, topic11-36, topic7-33, topic5-35, > > > topic9-28, topic10-47, topic1-25, topic2-38, topic3-43, topic8-35, > > > topic4-38, topic6-36, topic1-43, topic10-31, topic3-26, topic9-26, > > > topic7-34, topic4-37, topic10-48, topic11-35, topic1-26, topic5-34, > > > topic8-36, topic3-42, topic9-43, topic2-39] for group > > > connect-topicsToFolders (org.apache.kafka.clients.consumer.internals. > > > ConsumerCoordinator:231) > > > > > > [41:27,131] INFO ### Task [2134126231] retrieved [100] records > > > > > > [41:27,162] INFO ### [TopicsToFoldersSinkTask]:[518962883] received > > > [0] records [41:27,162] INFO ### Writing [0] events [41:27,162] INFO > > > ### Sink Task [518962883] has processed [0] events > > > > > > [41:27,162] INFO WorkerSinkTask{id=topicsToFolders-0} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > > > > [41:27,178] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,178] INFO ### Topic [topic1] Kafka Offset/Partition > > > [91152/31] [41:27,178] INFO ### Writing [1] events [41:27,272] INFO > > > ### Sink Task [602953950] has processed [1] events > > > > > > [41:27,272] INFO WorkerSinkTask{id=topicsToFolders-1} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > > > > [41:27,272] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,287] INFO ### Topic [topic11] Kafka > > > Offset/Partition [90542/31] [41:27,287] INFO ### Writing [1] events > > > [41:27,287] INFO ### Sink Task [602953950] has processed [2] events > > > > > > [41:27,312] INFO Finished WorkerSourceTask{id=foldersToTopics-1} > > > commitOffsets successfully in 134 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > > > > [41:27,354] INFO ### Task [1714504927] retrieved [100] records > > > > > > [41:27,372] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,372] INFO ### Topic [topic8] Kafka Offset/Partition > > > [91028/31] [41:27,372] INFO ### Writing [1] events [41:27,375] INFO > > > ### Sink Task [602953950] has processed [3] events > > > > > > [41:27,382] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,382] INFO ### Topic [topic5] Kafka Offset/Partition > > > [91169/31] [41:27,383] INFO ### Writing [1] events [41:27,386] INFO > > > ### Sink Task [602953950] has processed [4] events > > > > > > [41:27,459] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,459] INFO ### Topic [topic2] Kafka Offset/Partition > > > [90560/31] [41:27,459] INFO ### Writing [1] events [41:27,459] INFO > > > ### Sink Task [602953950] has processed [5] events > > > > > > [41:27,475] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [2] records [41:27,475] INFO ### Topic [topic2] Kafka Offset/Partition > > > [90561/31] [41:27,475] INFO ### Topic [topic10] Kafka Offset/Partition > > > [91454/31] [41:27,475] INFO ### Writing [2] events [41:27,475] INFO > > > ### Sink Task [602953950] has processed [7] events > > > > > > [41:27,568] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [2] records [41:27,568] INFO ### Topic [topic4] Kafka Offset/Partition > > > [91644/31] [41:27,568] INFO ### Topic [topic4] Kafka Offset/Partition > > > [91645/31] [41:27,568] INFO ### Writing [2] events [41:27,568] INFO > > > ### Sink Task [602953950] has processed [9] events > > > > > > [41:27,584] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [2] records [41:27,584] INFO ### Topic [topic4] Kafka Offset/Partition > > > [91646/31] [41:27,584] INFO ### Topic [topic5] Kafka Offset/Partition > > > [91170/31] [41:27,584] INFO ### Writing [2] events [41:27,584] INFO > > > ### Sink Task [602953950] has processed [11] events > > > > > > [41:27,678] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,678] INFO ### Topic [topic7] Kafka Offset/Partition > > > [90928/31] [41:27,678] INFO ### Writing [1] events [41:27,678] INFO > > > ### Sink Task [602953950] has processed [12] events > > > > > > [41:27,678] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [2] records [41:27,678] INFO ### Topic [topic4] Kafka Offset/Partition > > > [91647/31] [41:27,678] INFO ### Topic [topic9] Kafka Offset/Partition > > > [90943/31] [41:27,678] INFO ### Writing [2] events [41:27,693] INFO > > > ### Sink Task [602953950] has processed [14] events > > > > > > [41:27,787] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:27,787] INFO ### Topic [topic6] Kafka Offset/Partition > > > [91126/31] [41:27,787] INFO ### Writing [1] events [41:27,803] INFO > > > ### Sink Task [602953950] has processed [15] events > > > > > > [41:27,803] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [185] records [41:27,803] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90454/31] ... > > > [41:27,818] INFO ### Topic [topic3] Kafka Offset/Partition [90468/31] > > > [41:27,818] INFO ### Topic [topic4] Kafka Offset/Partition [91648/31] > > > ... > > > [41:27,818] INFO ### Topic [topic4] Kafka Offset/Partition [91665/31] > > > [41:27,818] INFO ### Topic [topic6] Kafka Offset/Partition [91127/31] > > > ... > > > [41:27,818] INFO ### Topic [topic6] Kafka Offset/Partition [91147/31] > > > [41:27,818] INFO ### Topic [topic2] Kafka Offset/Partition [90562/31] > > > ... > > > [41:27,818] INFO ### Topic [topic2] Kafka Offset/Partition [90577/31] > > > [41:27,818] INFO ### Topic [topic9] Kafka Offset/Partition [90944/31] > > > ... > > > [41:27,818] INFO ### Topic [topic9] Kafka Offset/Partition [90958/31] > > > [41:27,818] INFO ### Topic [topic10] Kafka Offset/Partition [91455/31] > > > ... > > > [41:27,818] INFO ### Topic [topic10] Kafka Offset/Partition [91466/31] > > > [41:27,818] INFO ### Topic [topic8] Kafka Offset/Partition [91029/31] > > > ... > > > [41:27,818] INFO ### Topic [topic8] Kafka Offset/Partition [91047/31] > > > [41:27,818] INFO ### Topic [topic5] Kafka Offset/Partition [91171/31] > > > ... > > > [41:27,818] INFO ### Topic [topic5] Kafka Offset/Partition [91186/31] > > > [41:27,818] INFO ### Topic [topic1] Kafka Offset/Partition [91153/31] > > > ... > > > [41:27,818] INFO ### Topic [topic1] Kafka Offset/Partition [91163/31] > > > [41:27,818] INFO ### Topic [topic11] Kafka Offset/Partition [90543/31] > > > ... > > > [41:27,834] INFO ### Topic [topic11] Kafka Offset/Partition [90564/31] > > > [41:27,834] INFO ### Topic [topic7] Kafka Offset/Partition [90929/31] > > > ... > > > [41:27,834] INFO ### Topic [topic7] Kafka Offset/Partition [90948/31] > > > [41:27,834] INFO ### Writing [185] events [41:28,053] INFO ### Sink > > > Task [602953950] has processed [200] events > > > > > > [41:28,178] INFO ### [TopicsToFoldersSinkTask]:[518962883] received > > > [0] records [41:28,178] INFO ### Writing [0] events [41:28,178] INFO > > > ### Sink Task [518962883] has processed [0] events > > > > > > [41:28,178] INFO WorkerSinkTask{id=topicsToFolders-0} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > [41:28,193] INFO Finished WorkerSourceTask{id=foldersToTopics-0} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > > > > [41:28,209] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [0] records [41:28,209] INFO ### Writing [0] events [41:28,209] INFO > > > ### Sink Task [602953950] has processed [200] events > > > > > > [41:28,209] INFO WorkerSinkTask{id=topicsToFolders-1} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > [41:28,209] INFO Reflections took 3141 ms to scan 61 urls, producing > > > 6178 keys and 37307 values (org.reflections.Reflections:229) > > > [41:28,318] INFO Finished WorkerSourceTask{id=foldersToTopics-1} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > > > > [41:28,521] INFO ### Task [2134126231] retrieved [100] records > > > > > > [41:28,537] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:28,537] INFO ### Topic [topic7] Kafka Offset/Partition > > > [90949/31] [41:28,537] INFO ### Writing [1] events [41:28,537] INFO > > > ### Sink Task [602953950] has processed [201] events > > > > > > [41:28,553] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:28,553] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90469/31] ... > > > [41:28,553] INFO ### Topic [topic3] Kafka Offset/Partition [90476/31] > > > [41:28,553] INFO ### Topic [topic4] Kafka Offset/Partition [91666/31] > > > ... > > > [41:28,553] INFO ### Topic [topic4] Kafka Offset/Partition [91681/31] > > > [41:28,553] INFO ### Topic [topic6] Kafka Offset/Partition [91148/31] > > > ... > > > [41:28,553] INFO ### Topic [topic6] Kafka Offset/Partition [91152/31] > > > [41:28,553] INFO ### Topic [topic2] Kafka Offset/Partition [90578/31] > > > ... > > > [41:28,553] INFO ### Topic [topic2] Kafka Offset/Partition [90587/31] > > > [41:28,553] INFO ### Topic [topic9] Kafka Offset/Partition [90959/31] > > > ... > > > [41:28,553] INFO ### Topic [topic9] Kafka Offset/Partition [90963/31] > > > [41:28,553] INFO ### Topic [topic10] Kafka Offset/Partition [91467/31] > > > ... > > > [41:28,553] INFO ### Topic [topic10] Kafka Offset/Partition [91473/31] > > > [41:28,553] INFO ### Topic [topic8] Kafka Offset/Partition [91048/31] > > > ... > > > [41:28,553] INFO ### Topic [topic8] Kafka Offset/Partition [91054/31] > > > [41:28,553] INFO ### Topic [topic5] Kafka Offset/Partition [91187/31] > > > ... > > > [41:28,568] INFO ### Topic [topic5] Kafka Offset/Partition [91199/31] > > > [41:28,568] INFO ### Topic [topic1] Kafka Offset/Partition [91164/31] > > > ... > > > [41:28,568] INFO ### Topic [topic1] Kafka Offset/Partition [91171/31] > > > [41:28,568] INFO ### Topic [topic11] Kafka Offset/Partition [90565/31] > > > ... > > > [41:28,568] INFO ### Topic [topic11] Kafka Offset/Partition [90572/31] > > > [41:28,568] INFO ### Topic [topic7] Kafka Offset/Partition [90950/31] > > > ... > > > [41:28,568] INFO ### Topic [topic7] Kafka Offset/Partition [90961/31] > > > [41:28,568] INFO ### Writing [99] events [41:28,662] INFO ### Sink > > > Task [602953950] has processed [300] events > > > > > > [41:28,834] INFO ### Task [1714504927] retrieved [100] records > > > > > > [41:28,850] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:28,850] INFO ### Topic [topic11] Kafka > > > Offset/Partition [90573/31] [41:28,850] INFO ### Writing [1] events > > > [41:28,850] INFO ### Sink Task [602953950] has processed [301] events > > > > > > [41:28,850] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:28,850] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90477/31] ... > > > [41:28,850] INFO ### Topic [topic3] Kafka Offset/Partition [90482/31] > > > [41:28,850] INFO ### Topic [topic4] Kafka Offset/Partition [91682/31] > > > ... > > > [41:28,850] INFO ### Topic [topic4] Kafka Offset/Partition [91691/31] > > > [41:28,865] INFO ### Topic [topic6] Kafka Offset/Partition [91153/31] > > > ... > > > [41:28,865] INFO ### Topic [topic6] Kafka Offset/Partition [91157/31] > > > [41:28,865] INFO ### Topic [topic2] Kafka Offset/Partition [90588/31] > > > ... > > > [41:28,865] INFO ### Topic [topic2] Kafka Offset/Partition [90596/31] > > > [41:28,865] INFO ### Topic [topic9] Kafka Offset/Partition [90964/31] > > > ... > > > [41:28,865] INFO ### Topic [topic9] Kafka Offset/Partition [90971/31] > > > [41:28,865] INFO ### Topic [topic10] Kafka Offset/Partition [91474/31] > > > ... > > > [41:28,865] INFO ### Topic [topic10] Kafka Offset/Partition [91485/31] > > > [41:28,865] INFO ### Topic [topic8] Kafka Offset/Partition [91055/31] > > > ... > > > [41:28,865] INFO ### Topic [topic8] Kafka Offset/Partition [91068/31] > > > [41:28,865] INFO ### Topic [topic5] Kafka Offset/Partition [91200/31] > > > ... > > > [41:28,865] INFO ### Topic [topic5] Kafka Offset/Partition [91210/31] > > > [41:28,865] INFO ### Topic [topic1] Kafka Offset/Partition [91172/31] > > > ... > > > [41:28,865] INFO ### Topic [topic1] Kafka Offset/Partition [91176/31] > > > [41:28,865] INFO ### Topic [topic11] Kafka Offset/Partition [90574/31] > > > ... > > > [41:28,865] INFO ### Topic [topic11] Kafka Offset/Partition [90587/31] > > > [41:28,865] INFO ### Topic [topic7] Kafka Offset/Partition [90962/31] > > > ... > > > [41:28,865] INFO ### Topic [topic7] Kafka Offset/Partition [90966/31] > > > [41:28,865] INFO ### Writing [99] events [41:28,943] INFO ### Sink > > > Task [602953950] has processed [400] events > > > > > > [41:29,162] INFO ### [TopicsToFoldersSinkTask]:[518962883] received > > > [0] records [41:29,162] INFO ### Writing [0] events [41:29,162] INFO > > > ### Sink Task [518962883] has processed [0] events > > > > > > [41:29,162] INFO WorkerSinkTask{id=topicsToFolders-0} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > > > > [41:29,178] INFO ### Task [2134126231] retrieved [100] records > > > > > > [41:29,178] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:29,178] INFO ### Topic [topic11] Kafka > > > Offset/Partition [90588/31] [41:29,178] INFO ### Writing [1] events > > > [41:29,178] INFO ### Sink Task [602953950] has processed [401] events > > > > > > [41:29,193] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:29,193] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90483/31] ... > > > [41:29,193] INFO ### Topic [topic3] Kafka Offset/Partition [90492/31] > > > [41:29,193] INFO ### Topic [topic4] Kafka Offset/Partition [91692/31] > > > ... > > > [41:29,193] INFO ### Topic [topic4] Kafka Offset/Partition [91699/31] > > > [41:29,193] INFO ### Topic [topic6] Kafka Offset/Partition [91158/31] > > > ... > > > [41:29,193] INFO ### Topic [topic6] Kafka Offset/Partition [91161/31] > > > [41:29,193] INFO ### Topic [topic2] Kafka Offset/Partition [90597/31] > > > ... > > > [41:29,193] INFO ### Topic [topic2] Kafka Offset/Partition [90607/31] > > > [41:29,193] INFO ### Topic [topic9] Kafka Offset/Partition [90972/31] > > > ... > > > [41:29,193] INFO ### Topic [topic9] Kafka Offset/Partition [90981/31] > > > [41:29,193] INFO ### Topic [topic10] Kafka Offset/Partition [91486/31] > > > ... > > > [41:29,193] INFO ### Topic [topic10] Kafka Offset/Partition [91492/31] > > > [41:29,193] INFO ### Topic [topic8] Kafka Offset/Partition [91069/31] > > > ... > > > [41:29,193] INFO ### Topic [topic8] Kafka Offset/Partition [91074/31] > > > [41:29,193] INFO ### Topic [topic5] Kafka Offset/Partition [91211/31] > > > ... > > > [41:29,193] INFO ### Topic [topic5] Kafka Offset/Partition [91218/31] > > > [41:29,193] INFO ### Topic [topic1] Kafka Offset/Partition [91177/31] > > > ... > > > [41:29,193] INFO ### Topic [topic1] Kafka Offset/Partition [91186/31] > > > [41:29,193] INFO ### Topic [topic11] Kafka Offset/Partition [90589/31] > > > ... > > > [41:29,193] INFO ### Topic [topic11] Kafka Offset/Partition [90603/31] > > > [41:29,193] INFO ### Topic [topic7] Kafka Offset/Partition [90967/31] > > > ... > > > [41:29,193] INFO ### Topic [topic7] Kafka Offset/Partition [90976/31] > > > [41:29,193] INFO ### Writing [99] events [41:29,271] INFO ### Sink > > > Task [602953950] has processed [500] events > > > > > > [41:29,209] INFO Finished WorkerSourceTask{id=foldersToTopics-0} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > [41:29,271] INFO WorkerSinkTask{id=topicsToFolders-1} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > [41:29,334] INFO Finished WorkerSourceTask{id=foldersToTopics-1} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > > > > [41:29,865] INFO ### Task [1714504927] retrieved [100] records > > > > > > [41:29,865] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:29,865] INFO ### Topic [topic5] Kafka Offset/Partition > > > [91219/31] [41:29,865] INFO ### Writing [1] events [41:29,865] INFO > > > ### Sink Task [602953950] has processed [501] events > > > > > > [41:29,881] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:29,881] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90493/31] ... > > > [41:29,881] INFO ### Topic [topic3] Kafka Offset/Partition [90502/31] > > > [41:29,881] INFO ### Topic [topic4] Kafka Offset/Partition [91700/31] > > > ... > > > [41:29,881] INFO ### Topic [topic4] Kafka Offset/Partition [91707/31] > > > [41:29,881] INFO ### Topic [topic6] Kafka Offset/Partition [91162/31] > > > ... > > > [41:29,881] INFO ### Topic [topic6] Kafka Offset/Partition [91168/31] > > > [41:29,881] INFO ### Topic [topic2] Kafka Offset/Partition [90608/31] > > > ... > > > [41:29,881] INFO ### Topic [topic2] Kafka Offset/Partition [90617/31] > > > [41:29,881] INFO ### Topic [topic9] Kafka Offset/Partition [90982/31] > > > ... > > > [41:29,881] INFO ### Topic [topic9] Kafka Offset/Partition [90991/31] > > > [41:29,881] INFO ### Topic [topic10] Kafka Offset/Partition [91493/31] > > > ... > > > [41:29,881] INFO ### Topic [topic10] Kafka Offset/Partition [91497/31] > > > [41:29,881] INFO ### Topic [topic8] Kafka Offset/Partition [91075/31] > > > ... > > > [41:29,881] INFO ### Topic [topic8] Kafka Offset/Partition [91081/31] > > > [41:29,881] INFO ### Topic [topic5] Kafka Offset/Partition [91220/31] > > > ... > > > [41:29,881] INFO ### Topic [topic5] Kafka Offset/Partition [91228/31] > > > [41:29,881] INFO ### Topic [topic1] Kafka Offset/Partition [91187/31] > > > ... > > > [41:29,881] INFO ### Topic [topic1] Kafka Offset/Partition [91197/31] > > > [41:29,881] INFO ### Topic [topic11] Kafka Offset/Partition [90604/31] > > > ... > > > [41:29,881] INFO ### Topic [topic11] Kafka Offset/Partition [90611/31] > > > [41:29,881] INFO ### Topic [topic7] Kafka Offset/Partition [90977/31] > > > ... > > > [41:29,881] INFO ### Topic [topic7] Kafka Offset/Partition [90990/31] > > > [41:29,881] INFO ### Writing [99] events [41:29,912] INFO ### Task > > > [2134126231] retrieved [100] records [41:29,943] INFO ### Sink Task > > > [602953950] has processed [600] events > > > > > > [41:29,943] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:29,943] INFO ### Topic [topic5] Kafka Offset/Partition > > > [91229/31] [41:29,943] INFO ### Writing [1] events .[41:29,943] INFO > > > ### Sink Task [602953950] has processed [601] events > > > > > > [41:29,943] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:29,943] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90503/31] ... > > > [41:29,943] INFO ### Topic [topic3] Kafka Offset/Partition [90513/31] > > > [41:29,943] INFO ### Topic [topic4] Kafka Offset/Partition [91708/31] > > > ... > > > [41:29,943] INFO ### Topic [topic4] Kafka Offset/Partition [91715/31] > > > [41:29,943] INFO ### Topic [topic6] Kafka Offset/Partition [91169/31] > > > ... > > > [41:29,943] INFO ### Topic [topic6] Kafka Offset/Partition [91177/31] > > > [41:29,943] INFO ### Topic [topic2] Kafka Offset/Partition [90618/31] > > > ... > > > [41:29,943] INFO ### Topic [topic2] Kafka Offset/Partition [90627/31] > > > [41:29,943] INFO ### Topic [topic9] Kafka Offset/Partition [90992/31] > > > ... > > > [41:29,943] INFO ### Topic [topic9] Kafka Offset/Partition [90994/31] > > > [41:29,943] INFO ### Topic [topic10] Kafka Offset/Partition [91498/31] > > > ... > > > [41:29,943] INFO ### Topic [topic10] Kafka Offset/Partition [91507/31] > > > [41:29,943] INFO ### Topic [topic8] Kafka Offset/Partition [91082/31] > > > ... > > > [41:29,943] INFO ### Topic [topic8] Kafka Offset/Partition [91093/31] > > > [41:29,943] INFO ### Topic [topic5] Kafka Offset/Partition [91230/31] > > > ... > > > [41:29,943] INFO ### Topic [topic5] Kafka Offset/Partition [91236/31] > > > [41:29,943] INFO ### Topic [topic1] Kafka Offset/Partition [91198/31] > > > ... > > > [41:29,943] INFO ### Topic [topic1] Kafka Offset/Partition [91208/31] > > > [41:29,943] INFO ### Topic [topic11] Kafka Offset/Partition [90612/31] > > > ... > > > [41:29,943] INFO ### Topic [topic11] Kafka Offset/Partition [90620/31] > > > [41:29,943] INFO ### Topic [topic7] Kafka Offset/Partition [90991/31] > > > ... > > > [41:29,943] INFO ### Topic [topic7] Kafka Offset/Partition [90999/31] > > > [41:29,943] INFO ### Writing [99] events [41:30,021] INFO ### Sink > > > Task [602953950] has processed [700] events > > > > > > [41:30,162] INFO ### [TopicsToFoldersSinkTask]:[518962883] received > > > [0] records [41:30,162] INFO ### Writing [0] events [41:30,162] INFO > > > ### Sink Task [518962883] has processed [0] events > > > > > > [41:30,162] INFO WorkerSinkTask{id=topicsToFolders-0} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > > > > [41:30,225] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [0] records [41:30,225] INFO ### Writing [0] events [41:30,225] INFO > > > ### Sink Task [602953950] has processed [700] events > > > > > > [41:30,225] INFO WorkerSinkTask{id=topicsToFolders-1} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > [41:30,225] INFO Finished WorkerSourceTask{id=foldersToTopics-0} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > [41:30,350] INFO Finished WorkerSourceTask{id=foldersToTopics-1} > > > commitOffsets successfully in 0 ms (org.apache.kafka.connect. > > > runtime.WorkerSourceTask:358) > > > > > > [41:30,475] INFO ### Task [2134126231] retrieved [100] records > > > [41:30,490] INFO ### Task [2134126231] has read all of its files > > > (c.b.b.pp.kafkaConnect.tasks.RecordRetriever:65) > > > > > > [41:30,490] INFO ### Task [1714504927] retrieved [100] records > > > > > > [41:30,490] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:30,490] INFO ### Topic [topic11] Kafka > > > Offset/Partition [90621/31] [41:30,490] INFO ### Writing [1] events > > > [41:30,490] INFO ### Sink Task [602953950] has processed [701] events > > > > > > [41:30,506] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [101] records [41:30,506] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90514/31] ... > > > [41:30,506] INFO ### Topic [topic3] Kafka Offset/Partition [90522/31] > > > [41:30,506] INFO ### Topic [topic4] Kafka Offset/Partition [91716/31] > > > ... > > > [41:30,506] INFO ### Topic [topic4] Kafka Offset/Partition [91728/31] > > > [41:30,506] INFO ### Topic [topic6] Kafka Offset/Partition [91178/31] > > > ... > > > [41:30,506] INFO ### Topic [topic6] Kafka Offset/Partition [91186/31] > > > [41:30,506] INFO ### Topic [topic2] Kafka Offset/Partition [90628/31] > > > ... > > > [41:30,521] INFO ### Topic [topic2] Kafka Offset/Partition [90634/31] > > > [41:30,521] INFO ### Topic [topic9] Kafka Offset/Partition [90995/31] > > > ... > > > [41:30,521] INFO ### Topic [topic9] Kafka Offset/Partition [91003/31] > > > [41:30,521] INFO ### Topic [topic10] Kafka Offset/Partition [91508/31] > > > ... > > > [41:30,521] INFO ### Topic [topic10] Kafka Offset/Partition [91516/31] > > > [41:30,521] INFO ### Topic [topic8] Kafka Offset/Partition [91094/31] > > > ... > > > [41:30,521] INFO ### Topic [topic8] Kafka Offset/Partition [91105/31] > > > [41:30,521] INFO ### Topic [topic5] Kafka Offset/Partition [91237/31] > > > ... > > > [41:30,521] INFO ### Topic [topic5] Kafka Offset/Partition [91242/31] > > > [41:30,521] INFO ### Topic [topic1] Kafka Offset/Partition [91209/31] > > > ... > > > [41:30,521] INFO ### Topic [topic1] Kafka Offset/Partition [91212/31] > > > ... > > > [41:30,521] INFO ### Topic [topic11] Kafka Offset/Partition [90633/31] > > > [41:30,521] INFO ### Topic [topic7] Kafka Offset/Partition [91000/31] > > > ... > > > [41:30,521] INFO ### Topic [topic7] Kafka Offset/Partition [91010/31] > > > [41:30,521] INFO ### Writing [101] events [41:30,584] INFO ### Sink > > > Task [602953950] has processed [802] events > > > > > > [41:30,584] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [98] records [41:30,584] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90523/31] ... > > > [41:30,584] INFO ### Topic [topic3] Kafka Offset/Partition [90530/31] > > > [41:30,584] INFO ### Topic [topic4] Kafka Offset/Partition [91729/31] > > > ... > > > [41:30,584] INFO ### Topic [topic4] Kafka Offset/Partition [91731/31] > > > [41:30,584] INFO ### Topic [topic6] Kafka Offset/Partition [91187/31] > > > ... > > > [41:30,584] INFO ### Topic [topic6] Kafka Offset/Partition [91200/31] > > > [41:30,584] INFO ### Topic [topic2] Kafka Offset/Partition [90635/31] > > > ... > > > [41:30,584] INFO ### Topic [topic2] Kafka Offset/Partition [90647/31] > > > [41:30,584] INFO ### Topic [topic9] Kafka Offset/Partition [91004/31] > > > ... > > > [41:30,584] INFO ### Topic [topic9] Kafka Offset/Partition [91010/31] > > > [41:30,584] INFO ### Topic [topic10] Kafka Offset/Partition [91517/31] > > > ... > > > [41:30,584] INFO ### Topic [topic10] Kafka Offset/Partition [91525/31] > > > [41:30,584] INFO ### Topic [topic8] Kafka Offset/Partition [91106/31] > > > ... > > > [41:30,584] INFO ### Topic [topic8] Kafka Offset/Partition [91116/31] > > > [41:30,584] INFO ### Topic [topic5] Kafka Offset/Partition [91243/31] > > > ... > > > [41:30,584] INFO ### Topic [topic5] Kafka Offset/Partition [91250/31] > > > [41:30,584] INFO ### Topic [topic1] Kafka Offset/Partition [91213/31] > > > ... > > > [41:30,584] INFO ### Topic [topic1] Kafka Offset/Partition [91222/31] > > > [41:30,584] INFO ### Topic [topic11] Kafka Offset/Partition [90634/31] > > > ... > > > [41:30,584] INFO ### Topic [topic11] Kafka Offset/Partition [90639/31] > > > [41:30,584] INFO ### Topic [topic7] Kafka Offset/Partition [91011/31] > > > ... > > > [41:30,584] INFO ### Topic [topic7] Kafka Offset/Partition [91019/31] > > > [41:30,584] INFO ### Writing [98] events [41:30,646] INFO ### Sink > > > Task [602953950] has processed [900] events > > > > > > [41:30,881] INFO ### Task [1714504927] retrieved [100] records > > > [41:30,881] INFO ### Task [1714504927] has read all of its files > > > (c.b.b.pp.kafkaConnect.tasks.RecordRetriever:65) > > > > > > [41:30,881] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [1] records [41:30,881] INFO ### Topic [topic8] Kafka Offset/Partition > > > [91117/31] [41:30,881] INFO ### Writing [1] events [41:30,896] INFO > > > ### Sink Task [602953950] has processed [901] events > > > > > > [41:30,896] INFO ### [TopicsToFoldersSinkTask]:[602953950] received > > > [99] records [41:30,896] INFO ### Topic [topic3] Kafka > > > Offset/Partition [90531/31] ... > > > [41:30,896] INFO ### Topic [topic3] Kafka Offset/Partition [90539/31] > > > [41:30,896] INFO ### Topic [topic4] Kafka Offset/Partition [91732/31] > > > ... > > > [41:30,896] INFO ### Topic [topic4] Kafka Offset/Partition [91737/31] > > > [41:30,896] INFO ### Topic [topic6] Kafka Offset/Partition [91201/31] > > > ... > > > [41:30,896] INFO ### Topic [topic6] Kafka Offset/Partition [91208/31] > > > [41:30,896] INFO ### Topic [topic2] Kafka Offset/Partition [90648/31] > > > ... > > > [41:30,896] INFO ### Topic [topic2] Kafka Offset/Partition [90657/31] > > > [41:30,896] INFO ### Topic [topic9] Kafka Offset/Partition [91011/31] > > > ... > > > [41:30,896] INFO ### Topic [topic9] Kafka Offset/Partition [91020/31] > > > [41:30,896] INFO ### Topic [topic10] Kafka Offset/Partition [91526/31] > > > ... > > > [41:30,896] INFO ### Topic [topic10] Kafka Offset/Partition [91534/31] > > > [41:30,896] INFO ### Topic [topic8] Kafka Offset/Partition [91118/31] > > > ... > > > [41:30,896] INFO ### Topic [topic8] Kafka Offset/Partition [91122/31] > > > [41:30,896] INFO ### Topic [topic5] Kafka Offset/Partition [91251/31] > > > ... > > > [41:30,896] INFO ### Topic [topic5] Kafka Offset/Partition [91262/31] > > > [41:30,896] INFO ### Topic [topic1] Kafka Offset/Partition [91223/31] > > > ... > > > [41:30,896] INFO ### Topic [topic1] Kafka Offset/Partition [91231/31] > > > [41:30,896] INFO ### Topic [topic11] Kafka Offset/Partition [90640/31] > > > ... > > > [41:30,896] INFO ### Topic [topic11] Kafka Offset/Partition [90650/31] > > > [41:30,896] INFO ### Topic [topic7] Kafka Offset/Partition [91020/31] > > > ... > > > [41:30,896] INFO ### Topic [topic7] Kafka Offset/Partition [91029/31] > > > [41:30,896] INFO ### Writing [99] events [41:30,943] INFO ### Sink > > > Task [602953950] has processed [1000] events > > > > > > [41:31,178] INFO ### [TopicsToFoldersSinkTask]:[518962883] received > > > [0] records [41:31,178] INFO ### Writing [0] events [41:31,178] INFO > > > ### Sink Task [518962883] has processed [0] events > > > > > > [41:31,178] INFO WorkerSinkTask{id=topicsToFolders-0} Committing > > > offsets > > > (org.apache.kafka.connect.runtime.WorkerSinkTask:261) > > > > > > -----Original Message----- > > > From: Gwen Shapira [mailto:g...@confluent.io] > > > Sent: 26 April 2017 18:29 > > > To: dev@kafka.apache.org > > > Subject: Re: Kafka Connect and Partitions > > > > > > Hi, > > > > > > I'll need a bit more detail to help :) Are you writing a connector or > > > trying to use an existing one? If existing, which connector? Is it > > > source or sink? > > > > > > Here are few things I'd look at when debugging: > > > > > > * Is the connector reading from the topic you think you are reading? > > > * Do you actually have 4 tasks? Are they all running? Are there errors? > > > What happens if you stop the only task doing the work? > > > * Is the one task subscribed to all partitions? How did you check that? > > > * Do you have data in all 50 partitions? > > > * Anything interesting in the log? > > > > > > I hope this helps you get started :) > > > In general, if all 50 partitions have data and all 4 tasks are running > > > but only one is actually subscribed to partitions, it sounds like a > > > bug in consumer rebalance - but this also seems highly unlikely, so > > > I'm searching for other causes. > > > > > > Gwen > > > > > > On Wed, Apr 26, 2017 at 8:57 AM, <david.frank...@bt.com> wrote: > > > > > > > I've defined several Kafka Connect tasks via the tasks.max property > > > > to process a set of topics. > > > > Initially I set the partitions on the topics to 1 and partitioned > > > > the topics across the tasks programmatically so that each task > > > > processed a subset of the topics (or so I thought ...). > > > > I then noticed that only 1 of the tasks ever read any Kafka messages > > > > and concluded that the topics property defined in > > > > connector.properties cannot be split across the tasks in this way. > > > > > > > > It then dawned on me that perhaps I ought to be partitioning the > > > > topic at creation time so that each task would be assigned a set of > > > > partitions across the entire set of topics. > > > > > > > > However, that seems not to work either - again only 1 task does any > > > > work - and this task reads from the same partition for every topic > > > > (I have defined > > > > 50 partitions and 4 tasks so would expect (naively perhaps) each > > > > task to get a dozen or so partitions for each topic). > > > > > > > > Could some kind soul point out the error of my ways please and tell > > > > me how to achieve this properly. > > > > > > > > Thanks in advance, > > > > David > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > *Gwen Shapira* > > > Product Manager | Confluent > > > 650.450.2760 | @gwenshap > > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog < > > > http://www.confluent.io/blog> > > > > > > > > > > > -- > > *Gwen Shapira* > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog < > > http://www.confluent.io/blog> > > >
