GitHub user janewangfb reopened a pull request:
https://github.com/apache/spark/pull/18975
[SPARK-4131] Support "Writing data into the filesystem from queries"
## What changes were proposed in this pull request?
This PR implements the sql feature:
INSERT OVERWRITE [LOCAL] DIRECTORY directory1
[ROW FORMAT row_format] [STORED AS file_format]
SELECT ... FROM ...
## How was this patch tested?
Added new unittests and also pulled the code to fb-spark so that we could
test writing to hdfs directory.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/janewangfb/spark port_local_directory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18975.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18975
----
commit 6ca7771a39fe7c6ca5f82acc7d76dff00238de68
Author: Jane Wang <[email protected]>
Date: 2017-08-17T01:32:23Z
add insert overwrite local directory
commit a975536ea3fed0674410ebc65538e134992779b1
Author: Jane Wang <[email protected]>
Date: 2017-08-17T03:59:25Z
Add Unittests
commit a15bf4e0c12a37127788528f1e69b0462a0e287b
Author: Jane Wang <[email protected]>
Date: 2017-08-17T05:23:34Z
fix local path
commit 9f596fda9dccdb0d5541340e8d9f2bf83573066a
Author: Jane Wang <[email protected]>
Date: 2017-08-17T18:25:04Z
Merge branch 'master' into port_local_directory
commit b9db02ecf130a8524fc9e864e1758117094f04cf
Author: Jane Wang <[email protected]>
Date: 2017-08-17T18:50:50Z
fix style
commit e516bec41bf877070023affbb79cc7823e2d6af8
Author: Jane Wang <[email protected]>
Date: 2017-08-18T04:06:38Z
Merge branch 'master' into port_local_directory
commit e05624ff15092773c329182e85cd2f942151def0
Author: Jane Wang <[email protected]>
Date: 2017-08-18T06:53:27Z
condense storage
commit 7f5664dcf5e55f3cc2ad91a000bb2c8ab58f596a
Author: Jane Wang <[email protected]>
Date: 2017-08-18T07:13:00Z
change InsertInto to InsertIntoTable
commit d50b3a2a722783e4bf971382946fce3e61ddee3b
Author: Jane Wang <[email protected]>
Date: 2017-08-19T16:25:28Z
add InsertIntoDirectory
commit 61a18a29955e71a89eecfeab6a01ba725be157ce
Author: Jane Wang <[email protected]>
Date: 2017-08-19T16:36:32Z
update insertInto
commit 4c19aafff3167206eaf9895f66ae5bf5301725cf
Author: Jane Wang <[email protected]>
Date: 2017-08-19T18:16:57Z
SQLQuerySuite passed
commit 47fde8a6591cbdad5656b51c3b55e3647d7b8d1f
Author: Jane Wang <[email protected]>
Date: 2017-08-19T18:37:39Z
fix comments
commit 068662a5abaaa693529320bb855b7a3323915bf8
Author: Jane Wang <[email protected]>
Date: 2017-08-21T03:55:09Z
Merge branch 'master' into port_local_directory
commit da7065ba5b30a777878ee1f8cb6c5fb4cfa42212
Author: Jane Wang <[email protected]>
Date: 2017-08-21T16:35:14Z
Add tableProdier
commit 7f4b488a3aff850949c4d414eea8c071265a7d4c
Author: Jane Wang <[email protected]>
Date: 2017-08-21T20:54:46Z
Add InsertIntoDataSourceDirCommand
commit 051018ef776e8634295e9e60b353694feefd72cb
Author: Jane Wang <[email protected]>
Date: 2017-08-21T20:55:48Z
Merge branch 'master' into port_local_directory
commit 73f605ed955657e6db4d29503ea1df403c304ba4
Author: Jane Wang <[email protected]>
Date: 2017-08-21T20:59:12Z
fix style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
