[
https://issues.apache.org/jira/browse/SPARK-27975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27975:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> ConsoleSink should display alias and options for streaming progress
> -------------------------------------------------------------------
>
> Key: SPARK-27975
> URL: https://issues.apache.org/jira/browse/SPARK-27975
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.1.0
> Reporter: Jacek Laskowski
> Priority: Minor
>
> {{console}} sink shows itself in progress with this internal instance
> representation as follows:
> {code:json}
> "sink" : {
> "description" :
> "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
> }
> {code}
> That is not very user-friendly and would be much better for debugging if it
> included the alias and options as {{socket}} does:
> {code}
> "sources" : [ {
> "description" : "TextSocketV2[host: localhost, port: 8888]",
> ...
> } ],
> {code}
> The entire sample progress looks as follows:
> {code}
> 19/06/07 11:47:18 INFO MicroBatchExecution: Streaming query made progress: {
> "id" : "26bedc9f-076f-4b15-8e17-f09609aaecac",
> "runId" : "8c365e74-7ac9-4fad-bf1b-397eb086661e",
> "name" : "socket-console",
> "timestamp" : "2019-06-07T09:47:18.969Z",
> "batchId" : 2,
> "numInputRows" : 0,
> "inputRowsPerSecond" : 0.0,
> "durationMs" : {
> "getEndOffset" : 0,
> "setOffsetRange" : 0,
> "triggerExecution" : 0
> },
> "stateOperators" : [ ],
> "sources" : [ {
> "description" : "TextSocketV2[host: localhost, port: 8888]",
> "startOffset" : 0,
> "endOffset" : 0,
> "numInputRows" : 0,
> "inputRowsPerSecond" : 0.0
> } ],
> "sink" : {
> "description" :
> "org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
