imaffe commented on a change in pull request #18406:
URL: https://github.com/apache/flink/pull/18406#discussion_r810451071
##########
File path:
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/reader/split/PulsarPartitionSplitReaderBase.java
##########
@@ -104,24 +104,31 @@ protected PulsarPartitionSplitReaderBase(
PulsarMessageCollector<OUT> collector = new
PulsarMessageCollector<>(splitId, builder);
Deadline deadline =
Deadline.fromNow(sourceConfiguration.getMaxFetchTime());
- // Consume message from pulsar until it was woke up by flink reader.
+ // Consume message from pulsar until it was woken up by flink reader.
for (int messageNum = 0;
messageNum < sourceConfiguration.getMaxFetchRecords()
&& deadline.hasTimeLeft()
&& isNotWakeup();
messageNum++) {
try {
Duration timeout = deadline.timeLeftIfAny();
- Message<byte[]> message = pollMessage(timeout);
+ Message<?> message = pollMessage(timeout);
if (message == null) {
break;
}
- // Deserialize message.
collector.setMessage(message);
- deserializationSchema.deserialize(message, collector);
- // Acknowledge message if need.
+ // Deserialize message by DeserializationSchema or Pulsar
Schema.
+ if (sourceConfiguration.isEnableSchemaEvolution()) {
Review comment:
If we implement that way, we can use a different approach to support
schema evolution, in that case, users uses nativePulsarSchema(...) and pass
this `PulsarDeserializationSchema` impl to pulsar source build, and
ENABLE_SCHEMA_EVOLUTION related configs is not necessary in that case. The good
part is that the schema evolution semantics are more fine-grained.
The downside is our api turns from 3 to 4: originally only support
TypeInformation, flink and pulsar, now we have a nativePulsar (which is the
only schema that supports schema evolution)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
