Remco2376 edited a comment on issue #9186:
URL: https://github.com/apache/pulsar/issues/9186#issuecomment-759499499
I can confirm that Pulsar sink connectors occasionally receive records that
do not contain eventTime and/or message. This behavior seems random. These Java
calls fail randomly:
long eventTime = record.getEventTime().orElseThrow(() -> new
RuntimeException("EventTime does not exist in record"));
Message<GenericRecord> message = record.getMessage().orElseThrow(() -> new
RuntimeException("Message does not exist in record"));
All records do actually contain an eventTime and message.
This happened with Pulsar 2.6.1. Recently upgraded to 2.7.0 and haven't seen
the same issue so far. But I've tested only for a very limited time. This issue
was observed in Pulsar/Docker
(https://pulsar.apache.org/docs/en/standalone-docker/).
I'm not sure, but I got the impression that this issue could be related to
high memory usage
(https://github.com/streamnative/pulsar-io-cloud-storage/blob/master/src/main/java/org/apache/pulsar/io/jcloud/util/AvroRecordUtil.java
contains a memory leak), so I'm currently logging memory usage to confirm that:
long totalMemory = Runtime.getRuntime().totalMemory();
long freeMemory = Runtime.getRuntime().freeMemory();
long memoryUsage = totalMemory - freeMemory;
System.out.println(totalMemory + " " + freeMemory + " " + memoryUsage);
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
