[
https://issues.apache.org/jira/browse/HIVE-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-8834:
-------------------------
Attachment: HIVE-8834.4-spark.patch
The new patch wraps requesting job/stage info into a {{Job}} so we can use the
existing asynchronous API in {{SparkClient}}.
I was trying to use {{KryoSerializer}} in the {{Job}}, but it turned out
{{KryoSerializer}} can't properly ser/de {{SparkJobInfo}} and
{{SparkStageInfo}}. I guess this is because the two classes are immutable,
without zero-arg constructor. Others may help me verify this.
Therefore, I added two wrappers for {{SparkJobInfo}} and {{SparkStageInfo}}.
This may seem a little ugly but we don't have to do all the refactors.
Hopefully this is a cleaner solution.
> enable job progress monitoring of Remote Spark Context [Spark Branch]
> ---------------------------------------------------------------------
>
> Key: HIVE-8834
> URL: https://issues.apache.org/jira/browse/HIVE-8834
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Rui Li
> Labels: Spark-M3
> Attachments: HIVE-8834.1-spark.patch, HIVE-8834.2-spark.patch,
> HIVE-8834.3-spark.patch, HIVE-8834.4-spark.patch
>
>
> We should enable job progress monitor in Remote Spark Context, the spark job
> progress info should fit into SparkJobStatus. SPARK-2321 supply new spark
> progress API, which should make this task easier.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
