xvrl commented on a change in pull request #10730:
URL: https://github.com/apache/druid/pull/10730#discussion_r552341496
##########
File path:
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaRecordSupplier.java
##########
@@ -119,15 +120,16 @@ public void seekToLatest(Set<StreamPartition<Integer>>
partitions)
@Nonnull
@Override
- public List<OrderedPartitionableRecord<Integer, Long>> poll(long timeout)
+ public List<OrderedPartitionableRecord<Integer, Long, KafkaRecordEntity>>
poll(long timeout)
{
- List<OrderedPartitionableRecord<Integer, Long>> polledRecords = new
ArrayList<>();
+ List<OrderedPartitionableRecord<Integer, Long, KafkaRecordEntity>>
polledRecords = new ArrayList<>();
for (ConsumerRecord<byte[], byte[]> record :
consumer.poll(Duration.ofMillis(timeout))) {
+
polledRecords.add(new OrderedPartitionableRecord<>(
record.topic(),
record.partition(),
record.offset(),
- record.value() == null ? null : ImmutableList.of(record.value())
+ ImmutableList.of(new KafkaRecordEntity(record))
Review comment:
note, today we skip records with null values, so we should decide if/how
we want to allow passing those through. I'm thinking we could add a config
option if needed that would expose them as empty byte arrays for backwards
compatibility in ByteEntity, but allow the full record in KafkaRecordEntity.
However, I have not found many use-cases for null values, so we can decide to
punt on this and add this option later if needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
