Arvind Ramachandran created SPARK-24791:
-------------------------------------------
Summary: Spark Structured Streaming randomly does not process batch
Key: SPARK-24791
URL: https://issues.apache.org/jira/browse/SPARK-24791
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.1
Reporter: Arvind Ramachandran
I have developed an application that writes small CSV files to a specific HDFS
folder . Spark Structured Streaming reads the HDFS folder . On a random basis i
see that it does not process a CSV File , the only case this occurs is the
batch size is one CSV file again random in nature not consistent.I cannot
guarantee the size of the batch will be greater than one because the
requirement is low latency processing but volume is low.
I can see that the commits , offset and source folders has the batch
information but the csv file is not processed when i look at the logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
