[ https://issues.apache.org/jira/browse/SPARK-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096738#comment-15096738 ]
Antonio Piccolboni commented on SPARK-6817: ------------------------------------------- So I am not sure row-orientation means anything anymore. Could you please point me to any examples or documentation for "projected rows" and "batching" UDFs? Sorry I am writing my first few UDFs, I may just need more education. The mechanism I am using is to extend the org.apache.hadoop.hive.ql.exec.UDF and provide an evaluate method. This is called once for each row. I don't have to invoke R at every call, but I have to return something, which is used directly or indirectly to create elements in a new column. Plus, and that may just be ignorance on my part, I don't know of any method that is invoked when all the data has been seen or any other way to detect I am on the last record from evaluate. I don't see how I can batch say 1000 rows and compute without this information. What happens when I batched the last incomplete batch and the last call to evaluate happens with no knowledge it's the last one? > DataFrame UDFs in R > ------------------- > > Key: SPARK-6817 > URL: https://issues.apache.org/jira/browse/SPARK-6817 > Project: Spark > Issue Type: New Feature > Components: SparkR, SQL > Reporter: Shivaram Venkataraman > Attachments: SparkR UDF Design Documentation v1.pdf > > > This depends on some internal interface of Spark SQL, should be done after > merging into Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org