HyukjinKwon commented on a change in pull request #27199: [SPARK-30508][SQL]
Add DataFrameReader.executeCommand API for external datasource
URL: https://github.com/apache/spark/pull/27199#discussion_r366709534
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -160,6 +161,30 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
this
}
+ /**
+ * Execute a random DDL/DML command inside an external execution engine
rather than Spark,
+ * especially for JDBC data source. This could be useful when user has some
custom commands
+ * which Spark doesn't support, need to be executed. Please note that this
is not appropriate
+ * for query which returns lots of data.
+ *
+ * Datasource should implement <code>ExternalCommandRunnableProvider</code>
to perform its
+ * own logic of command execution.
+ *
+ * @since 3.0.0
+ */
+ def executeCommand(command: String): DataFrame = {
+ DataSource.lookupDataSource(source, sparkSession.sessionState.conf) match {
+ case provider if
classOf[ExternalCommandRunnableProvider].isAssignableFrom(provider) =>
+ Dataset.ofRows(sparkSession,
+ ExternalCommandExecutor(command, extraOptions.asJava,
+
provider.newInstance().asInstanceOf[ExternalCommandRunnableProvider]))
+
+ case _ =>
+ throw new AnalysisException(s"`executeCommand` is not allowed for
source: $source, " +
+ s"because it doesn't implement `ExternalCommandRunnableProvider`")
Review comment:
I would just say "Command execution is not supported in source $source"
because that's what implies by not inheriting `ExternalCommandRunnableProvider`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
