holdenk created SPARK-16454:
-------------------------------
Summary: Consider adding a per-batch transform for structured
streaming
Key: SPARK-16454
URL: https://issues.apache.org/jira/browse/SPARK-16454
Project: Spark
Issue Type: Improvement
Components: SQL, Streaming
Reporter: holdenk
The new structured streaming API lacks the DStream functionality of transform
(which allowed one to mix in existing RDD transformation logic). It would be
useful to be able to do per-batch (even without any specific gaurantees about
the batch being complete provided you eventually get called with the "catch up"
records) processing as was done in the DStream API.
This might be useful for implementing Streaming Machine Learning on Structured
Streaming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
