[
https://issues.apache.org/jira/browse/SPARK-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096738#comment-15096738
]
Antonio Piccolboni commented on SPARK-6817:
-------------------------------------------
So I am not sure row-orientation means anything anymore. Could you please point
me to any examples or documentation for "projected rows" and "batching" UDFs?
Sorry I am writing my first few UDFs, I may just need more education. The
mechanism I am using is to extend the org.apache.hadoop.hive.ql.exec.UDF and
provide an evaluate method. This is called once for each row. I don't have to
invoke R at every call, but I have to return something, which is used directly
or indirectly to create elements in a new column. Plus, and that may just be
ignorance on my part, I don't know of any method that is invoked when all the
data has been seen or any other way to detect I am on the last record from
evaluate. I don't see how I can batch say 1000 rows and compute without this
information. What happens when I batched the last incomplete batch and the last
call to evaluate happens with no knowledge it's the last one?
> DataFrame UDFs in R
> -------------------
>
> Key: SPARK-6817
> URL: https://issues.apache.org/jira/browse/SPARK-6817
> Project: Spark
> Issue Type: New Feature
> Components: SparkR, SQL
> Reporter: Shivaram Venkataraman
> Attachments: SparkR UDF Design Documentation v1.pdf
>
>
> This depends on some internal interface of Spark SQL, should be done after
> merging into Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
