Github user salilsurendran commented on the issue:
https://github.com/apache/spark/pull/16664
@cloud-fan From what I understand we need to modify InsertXXX command to
carry all the write options instead of the change suggested in this PR. Right
now the QueryExecution object doesn't
Github user salilsurendran commented on the issue:
https://github.com/apache/spark/pull/16664
When I was working on this PR the output path wasn't there but if you are
confident that it is there then it might have been added recently. I can check
and get back to you.
---
If your
Github user salilsurendran commented on the issue:
https://github.com/apache/spark/pull/16664
The QueryExecution object doesn't have details related to the output
metadata. Like for eg. if I call df.write.parquet("/my/path"). The path to
which the DataFrame is written i.e.
Github user salilsurendran commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r99979817
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -218,7 +247,14 @@ final class DataFrameWriter[T] private[sql](ds
Github user salilsurendran commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r99979749
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -218,7 +247,14 @@ final class DataFrameWriter[T] private[sql](ds
Github user salilsurendran commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r99415030
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -660,12 +660,21 @@ object SQLConf {
.booleanConf
Github user salilsurendran commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r99414739
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -428,8 +481,14 @@ final class DataFrameWriter[T] private[sql](ds
Github user salilsurendran commented on a diff in the pull request:
https://github.com/apache/spark/pull/16664#discussion_r99386276
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala ---
@@ -190,6 +192,32 @@ final class DataFrameWriter[T] private[sql](ds
Github user salilsurendran commented on the issue:
https://github.com/apache/spark/pull/16664
@yhuai @marmbrus @liancheng Can someone review my PR please. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user salilsurendran opened a pull request:
https://github.com/apache/spark/pull/16664
[SPARK-18120 ][SQL] Call QueryExecutionListener callback methods for â¦
â¦DataFrameWriter methods
## What changes were proposed in this pull request?
QueryExecutionListener
10 matches
Mail list logo
