[
https://issues.apache.org/jira/browse/SPARK-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-24791:
----------------------------------
Comment: was deleted
(was: User 'zhangmeng0426' has created a pull request for this issue:
https://github.com/apache/spark/pull/24791)
> Spark Structured Streaming randomly does not process batch
> ----------------------------------------------------------
>
> Key: SPARK-24791
> URL: https://issues.apache.org/jira/browse/SPARK-24791
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Arvind Ramachandran
> Priority: Major
>
> I have developed an application that writes small CSV files to a specific
> HDFS folder . Spark Structured Streaming reads the HDFS folder . On a random
> basis i see that it does not process a CSV File , the only case this occurs
> is the batch size is one CSV file again random in nature not consistent.I
> cannot guarantee the size of the batch will be greater than one because the
> requirement is low latency processing but volume is low.
> I can see that the commits , offset and source folders has the batch
> information but the csv file is not processed when i look at the logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
