[
https://issues.apache.org/jira/browse/BEAM-4038?focusedWorklogId=99104&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-99104
]
ASF GitHub Bot logged work on BEAM-4038:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/May/18 18:33
Start Date: 07/May/18 18:33
Worklog Time Spent: 10m
Work Description: rangadi commented on a change in pull request #5287:
[BEAM-4038] Support writing ProducerRecords to Kafka
URL: https://github.com/apache/beam/pull/5287#discussion_r186504911
##########
File path:
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
##########
@@ -1130,6 +1134,20 @@ public T decode(InputStream inStream) {
}
}
+ private static class ProducerRecordWrite<K, V> extends
+ PTransform<PCollection<ProducerRecord<K, V>>, PDone> {
+
+ private final Write<K, V> kvWriteTransform;
+
+ ProducerRecordWrite(Write<K, V> kvWriteTransform){
+ this.kvWriteTransform = kvWriteTransform;
+ }
+
+ @Override public PDone expand(PCollection<ProducerRecord<K, V>> input) {
+ input.apply(ParDo.of(new ProducerRecordWriter<>(kvWriteTransform)));
Review comment:
The main issue with this approach is that `kvWriteTransform`'s `expand()` is
not invoked. None of the validation checks are performed. I am thinking too how
best to organize this.
One option is to have the main `Write` transform primarily work with
ProducerRecord.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 99104)
Time Spent: 9.5h (was: 9h 20m)
> Support Kafka Headers in KafkaIO
> --------------------------------
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka
> Reporter: Geet Kumar
> Assignee: Geet Kumar
> Priority: Major
> Fix For: 2.5.0
>
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The
> purpose of this JIRA is to support this feature in KafkaIO.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
