GitHub user wangxiaojing reopened a pull request:
https://github.com/apache/spark/pull/2765
[SPARK-3586][streaming]Support nested directories in Spark Streaming
For text files, the method streamingContext.textFileStream(dataDirectory).
The improvement of the streaming to Support subdirectories,spark streaming
can monitor the subdirectories dataDirectory and process any files created in
that directory.
eg:
streamingContext.textFileStream(/test).
Look at the direction contents:
/test/file1
/test/file2
/test/dr/file1
if the directory "/test/dr/" have new file "file2" ,spark streaming can
process the file
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wangxiaojing/spark spark-3586
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2765.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2765
----
commit 843f905cdf4ecb45aaa2edb9b34dc213e59e65c0
Author: wangxiaojing <[email protected]>
Date: 2014-10-11T08:22:31Z
Support nested directories in Spark Streaming
commit 6d30f6373fe06176043364c6bf4f7da81a37cf01
Author: wangxiaojing <[email protected]>
Date: 2014-10-12T05:27:22Z
change Nit
commit f00f2822dfbf21501458dd6f59e24eb4e7aac9c9
Author: wangxiaojing <[email protected]>
Date: 2014-10-17T03:46:01Z
support depth
commit 703754517d7077b891346b83e821c081215621db
Author: wangxiaojing <[email protected]>
Date: 2014-10-17T06:22:12Z
Change space before brace
commit 3d9bb2a7ef7ebea3f12866381af4201f4ddc7d60
Author: wangxiaojing <[email protected]>
Date: 2014-10-17T07:24:38Z
change process any files created in nested directories
commit 27dd88425b91471fcf9202364ddd9abb970e8223
Author: wangxiaojing <[email protected]>
Date: 2014-10-24T07:12:17Z
reformat code
commit 70d1b1fba5bb09636f4f3655771a98287c73b9ee
Author: wangxiaojing <[email protected]>
Date: 2014-10-24T07:54:09Z
add a require(depth >= 0)
commit 03489f28f4d8cae05564b41c98a839cf88bfba2f
Author: wangxiaojing <[email protected]>
Date: 2014-10-24T08:54:03Z
reformat code
commit 113c6d4e61e8c7e3fe4dfa4ed65cfe228575508f
Author: wangxiaojing <[email protected]>
Date: 2014-10-28T02:52:01Z
change performance
commit 2cc32fa187bc371f033da5bb2b67bbc7694964ed
Author: wangxiaojing <[email protected]>
Date: 2014-10-28T08:48:37Z
change filter name
commit 0ea8eda9831fcb3796720b0bfe95e51c8f1c3ab0
Author: wangxiaojing <[email protected]>
Date: 2014-11-03T09:55:46Z
change line exceeds 100 columns
commit 997ae5151d13a56ce0f97078ba4dacf454d43edd
Author: wangxiaojing <[email protected]>
Date: 2014-11-03T10:09:15Z
no braces for case clauses
commit 2bb9e9a148747c68d780ccbeaa8206e3b55cecfa
Author: wangxiaojing <[email protected]>
Date: 2014-11-10T03:46:00Z
Performance optimizationï¼directory records have judgment
commit 8bc22af117ca73f10b7f642ccc566493ba6c4a1a
Author: wangxiaojing <[email protected]>
Date: 2014-11-10T05:47:09Z
line over 100
commit 15c389371d645a7fa6f13f3909034fb57ea7360c
Author: wangxiaojing <[email protected]>
Date: 2014-12-04T09:27:12Z
remove line
commit 21f0d82153f2f7d8967256251fe1ef02c84ffa71
Author: wangxiaojing <[email protected]>
Date: 2014-12-04T10:21:19Z
style
commit e488919eb3ffe3b4d6509995720f4e33c48c0762
Author: wangxiaojing <[email protected]>
Date: 2014-12-17T04:49:36Z
change get depth
commit ce86bcc5be8a790245787f75dfd2cba51ab50f55
Author: wangxiaojing <[email protected]>
Date: 2014-12-24T06:07:43Z
Use 'isDir' to modify the compatibility
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
