[
https://issues.apache.org/jira/browse/SPARK-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit Baghel updated SPARK-21249:
--------------------------------
Description:
I am working with 2.2.0-SNAPSHOT and Structured Streaming. Is it possible to
use File Sink with mapGroupsWithState? I verified below cases and getting
exception in all.
* mapGroupsWithState - Aggregation without watermark and File Sink (Append mode)
Exception in thread "main" org.apache.spark.sql.AnalysisException: Append
output mode not supported when there are streaming aggregations on streaming
DataFrames/DataSets without watermark;;
mapGroupsWithState - Aggregation with watermark and File Sink (Append mode)
Exception in thread "main" org.apache.spark.sql.AnalysisException:
mapGroupsWithState is not supported with aggregation on a streaming
DataFrame/Dataset;;
mapGroupsWithState - No Aggregation and File Sink (Append mode)
Exception in thread "main" org.apache.spark.sql.AnalysisException:
mapGroupsWithState is not supported with Append output mode on a streaming
DataFrame/Dataset;;
mapGroupsWithState - No Aggregation and File Sink (Update mode)
Exception in thread "main" org.apache.spark.sql.AnalysisException: Data source
parquet does not support Update output mode;
mapGroupsWithState - No Aggregation and File Sink (Complete mode)
Exception in thread "main" org.apache.spark.sql.AnalysisException: Data source
parquet does not support Complete output mode;
was:
I am working with 2.2.0-SNAPSHOT and Structured Streaming. Is it possible to
use File Sink with mapGroupsWithState? With append output mode I am getting
below exception.
Exception in thread "main" org.apache.spark.sql.AnalysisException:
mapGroupsWithState is not supported with Append output mode on a streaming
DataFrame/Dataset;;
> Is it possible to use File Sink with mapGroupsWithState in Structured
> Streaming?
> --------------------------------------------------------------------------------
>
> Key: SPARK-21249
> URL: https://issues.apache.org/jira/browse/SPARK-21249
> Project: Spark
> Issue Type: Question
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Amit Baghel
> Priority: Minor
>
> I am working with 2.2.0-SNAPSHOT and Structured Streaming. Is it possible to
> use File Sink with mapGroupsWithState? I verified below cases and getting
> exception in all.
> * mapGroupsWithState - Aggregation without watermark and File Sink (Append
> mode)
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Append
> output mode not supported when there are streaming aggregations on streaming
> DataFrames/DataSets without watermark;;
> mapGroupsWithState - Aggregation with watermark and File Sink (Append mode)
> Exception in thread "main" org.apache.spark.sql.AnalysisException:
> mapGroupsWithState is not supported with aggregation on a streaming
> DataFrame/Dataset;;
> mapGroupsWithState - No Aggregation and File Sink (Append mode)
> Exception in thread "main" org.apache.spark.sql.AnalysisException:
> mapGroupsWithState is not supported with Append output mode on a streaming
> DataFrame/Dataset;;
> mapGroupsWithState - No Aggregation and File Sink (Update mode)
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Data
> source parquet does not support Update output mode;
> mapGroupsWithState - No Aggregation and File Sink (Complete mode)
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Data
> source parquet does not support Complete output mode;
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
