[
https://issues.apache.org/jira/browse/SPARK-10555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-10555:
---------------------------------
Labels: bulk-closed (was: )
> Add INotifyDStream to Spark Streaming
> -------------------------------------
>
> Key: SPARK-10555
> URL: https://issues.apache.org/jira/browse/SPARK-10555
> Project: Spark
> Issue Type: New Feature
> Components: DStreams
> Reporter: Vinoth Chandar
> Priority: Major
> Labels: bulk-closed
>
> Currently, spark streaming has support for fileStreams, and while this is
> super useful in general, it has its limitations - such as only being able to
> process new files under each folder.
> There are certain use cases (such as monitoring a root folder for incoming
> data, and registering the files into HIVE & performing file level replication
> across HDFS clusters) where taking actions based on multi level nested
> uploads is useful.
> We have a POC version of INotifyDStream that we are currently using in
> Staging environment at Uber. Would love to contribute that back, if it makes
> sense for Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
