[
https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822521#comment-16822521
]
Xiangrui Meng edited comment on SPARK-26412 at 4/20/19 5:13 PM:
----------------------------------------------------------------
[~bryanc] It handles the data exchange for DL model inference use cases, not
much for training. I also updated the description and mentioned data pipelining
benefits.
Also +1 on exposing Arrow format directly (only in Python).
was (Author: mengxr):
[~bryanc] It handles the data exchange for DL model inference use cases, not
much for training. I also updated the description and mentioned data pipelining
benefits.
> Allow Pandas UDF to take an iterator of pd.DataFrames for the entire partition
> ------------------------------------------------------------------------------
>
> Key: SPARK-26412
> URL: https://issues.apache.org/jira/browse/SPARK-26412
> Project: Spark
> Issue Type: New Feature
> Components: PySpark
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Priority: Major
>
> Pandas UDF is the ideal connection between PySpark and DL model inference
> workload. However, user needs to load the model file first to make
> predictions. It is common to see models of size ~100MB or bigger. If the
> Pandas UDF execution is limited to batch scope, user need to repeatedly load
> the same model for every batch in the same python worker process, which is
> inefficient. I created this JIRA to discuss possible solutions.
> Essentially we need to support "start()" and "finish()" besides "apply". We
> can either provide those interfaces or simply provide users the iterator of
> batches in pd.DataFrame or Arrow table and let user code handle it.
> Another benefit is with iterator interface and asyncio from Python, it is
> flexible for users to implement data pipelining.
> cc: [~icexelloss] [~bryanc] [~holdenk] [~hyukjin.kwon] [~ueshin] [~smilegator]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
