Github user jpiper commented on the issue:
https://github.com/apache/spark/pull/16578
I've just tested this patch and in my use case I've gone from 2 mins down
to about 6 seconds - absolutely fantastic, great work!
Given the benefits of storing data in nested/complex schema
Github user jpiper closed the pull request at:
https://github.com/apache/spark/pull/14861
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user jpiper commented on the issue:
https://github.com/apache/spark/pull/14861
Looks like this was actually added in #15140, so we can close this :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user jpiper commented on the issue:
https://github.com/apache/spark/pull/14861
@holdenk sorry I've been on vacation! I'll fix the typo and merge in the
latest master for you later today or tomorrow.
Cheers!
---
If your project is set up for it, you can reply
Github user jpiper commented on a diff in the pull request:
https://github.com/apache/spark/pull/14861#discussion_r77109772
--- Diff: python/test_support/test_folder/test_folder2/hello.txt ---
@@ -0,0 +1 @@
+Hello World!
--- End diff --
I wanted to ensure
Github user jpiper commented on the issue:
https://github.com/apache/spark/pull/14861
Is anyone available to review this small change? What's the proper process
here, @MLnick?
It seems @JoshRosen committed the original `addFile` implementation, maybe
he's the most
Github user jpiper commented on the issue:
https://github.com/apache/spark/pull/14861
I fixed the Python style issue and amended the last commit - tests should
be good now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
GitHub user jpiper opened a pull request:
https://github.com/apache/spark/pull/14861
[SPARK-17287] [PySpark] Add `recursive` kwarg to Java Python
`SparkContext.addFile`
## What changes were proposed in this pull request?
Add the ability to add entire directories using
