Raviteja created SPARK-27833:
--------------------------------
Summary: Structured Streaming Custom Sink --
Key: SPARK-27833
URL: https://issues.apache.org/jira/browse/SPARK-27833
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.3.0
Environment: spark 2.3.0
java 1.8
kafka version 0.10.
Reporter: Raviteja
Hi ,
We have a requirement to read data from kafka, apply some transformation and
store data to database .For this we are implementing watermarking feature along
with aggregate function and for storing we are writing our own sink
(Structured streaming) .we are using spark 2.3.0, java 1.8 and kafka version
0.10.
We are getting the below error.
"*java.lang.AssertionError: assertion failed: No plan for EventTimeWatermark
timestamp#39: timestamp, interval 2 minutes*"
works perfectly fine when we use Console as sink instead custom sink. For
Debugging the issue, we are performing "dataframe.show()" in our custom sink
and nothing else.
Please find the attachment for the Error log and the code. Please look into
this issue as this a blocker and we are not able to proceed further or find any
alternatives as we need watermarking feature.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
