[
https://issues.apache.org/jira/browse/DRILL-7388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976933#comment-16976933
]
ASF GitHub Bot commented on DRILL-7388:
---------------------------------------
paul-rogers commented on issue #1901: DRILL-7388: Kafka improvements
URL: https://github.com/apache/drill/pull/1901#issuecomment-555231426
@arina-ielchiieva, we do now have multiple plugins using EVF: text (which I
did) and several which Charles has done including Pcap, HDF5 and regex.
You are correct, since the Kafka reader uses the JSON reader, it makes sense
to get that merged as part of the JSON format reader. Once that is stable, we
can apply it here (and in Charles' HTTP storage plugin.)
The JSON format plugin will get merged once I finish resolving the
batch/container record count issues. Those were causing unexpected test
failures when I enabled the new JSON reader.
So, nothing yet to do on Kafka for EVF; but your changes will make the
eventual upgrade easier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Apache Drill Kafka Storage module fails to return results for partitions
> containing single offset record
> --------------------------------------------------------------------------------------------------------
>
> Key: DRILL-7388
> URL: https://issues.apache.org/jira/browse/DRILL-7388
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.16.0
> Reporter: daniel kelly
> Assignee: Arina Ielchiieva
> Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> If a partition only contains one record - e.g.
> [topicName=myTopic, partitionId=117, startOffset=0, endOffset=1]
> no data is returned.
> I fixed this locally with the following code change in contrib/storage-kafka
> :-
> {code:java}
> git diff
> src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java
> @@ -109,7 +109,7 @@ public class KafkaRecordReader extends
> AbstractRecordReader {
> currentMessageCount = 0;
>
> try {
> - while (currentOffset < subScanSpec.getEndOffset() - 1 &&
> msgItr.hasNext()) {
> + while (currentOffset < subScanSpec.getEndOffset() && msgItr.hasNext())
> {
> ConsumerRecord<byte[], byte[]> consumerRecord = msgItr.next();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
