danielfordfc commented on code in PR #9059:
URL: https://github.com/apache/hudi/pull/9059#discussion_r1244998980
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java:
##########
@@ -639,7 +639,13 @@ private Pair<SchemaProvider, Pair<String,
JavaRDD<HoodieRecord>>> fetchFromSourc
BuiltinKeyGenerator builtinKeyGenerator = (BuiltinKeyGenerator)
HoodieSparkKeyGeneratorFactory.createKeyGenerator(props);
List<HoodieRecord> avroRecords = new ArrayList<>();
while (genericRecordIterator.hasNext()) {
- GenericRecord genRec = genericRecordIterator.next();
+ GenericRecord genRec = null;
+ try {
+ genRec = genericRecordIterator.next();
+ } catch (IllegalArgumentException e) {
+ LOG.warn("Handling exception for transaction topic - " +
e.getMessage());
+ break;
Review Comment:
@ad1happy2go I've just pulled this and had tested it out, and it works fine
if there's only 1 batch of write.. but If I produce more records
transactionally, and then re-run the job (i.e. the ingestion starts from one of
these offset markers), it fails out..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
