I couldn't spot it anywhere on the web so it doesn't look to be contributed yet, but note that the HDFS APIs are already available per https://issues.apache.org/jira/browse/HDFS-6634 (you can see the test case for an implementation guideline in Java: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSInotifyEventInputStream.java#L107 )
On Wed, Dec 9, 2015 at 2:55 AM octagon blue <octagonb...@fastmail.com> wrote: > Hi All, > > I am using pyspark streaming to ETL raw data files as they land on HDFS. > While researching this topic I found this great presentation about Spark > and Spark Streaming at Uber > (http://www.slideshare.net/databricks/spark-meetup-at-uber), where they > mention this INotifyDStream that sounds very interesting and like it may > suit my use case well. > > Does anyone know if this code has been submitted to apache, or how I > might otherwise come upon it? > > Reference: https://issues.apache.org/jira/browse/SPARK-10555 - Add > INotifyDStream to Spark Streaming > > Thanks! > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
