Ethan Guo created HUDI-2865:
-------------------------------
Summary: Enable timeline-server-based markers for Spark structured
streaming
Key: HUDI-2865
URL: https://issues.apache.org/jira/browse/HUDI-2865
Project: Apache Hudi
Issue Type: Sub-task
Reporter: Ethan Guo
For Spark structured streaming, after the end of first micro-batch, the write
client is closed and hence triggers closure of timeline service. But subsequent
micro-batches do succeed though.
Currently, we explicitly override the marker type to be DIRECT in Spark
structured streaming. We need to revisit the behavior of timeline service
before we can enable timeline-server-based markers for Spark structured
streaming.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
