viirya commented on a change in pull request #31296:
URL: https://github.com/apache/spark/pull/31296#discussion_r563011361
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
##########
@@ -2889,6 +2889,24 @@ class Dataset[T] private[sql](
flatMap(func)(encoder)
}
+ /**
+ * Return a new Dataset of string created by piping elements to a forked
external process.
+ * The resulting Dataset is computed by executing the given process once per
partition.
+ * All elements of each input partition are written to a process's stdin as
lines of input
+ * separated by a newline. The resulting partition consists of the process's
stdout output, with
+ * each line of stdout resulting in one element of the output partition. A
process is invoked
+ * even for empty partitions.
+ *
+ * @param command command to run in forked process.
+ *
+ * @group typedrel
+ * @since 3.2.0
+ */
+ def pipe(command: String): Dataset[String] = {
Review comment:
Yea, for now I think a simple command parameter is enough. If there is
some needs for other parameters, we can add them later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
