[
https://issues.apache.org/jira/browse/PIG-5080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091473#comment-16091473
]
Nandor Kollar commented on PIG-5080:
------------------------------------
[~jeffzhang]
Could you please apply these modifications:
- Minor thing, but could you please delete two unused imports (HashMap and
LinkedHashMap) in JobGraphBuilder? Sorry, I hate unused imports. :)
- In StoreConverter please don't use wildcard import. Also, it seems there are
unused imports in this file too.
- In StoreConverter#ToRowFunction please wrap the Exception into
RuntimeException: instead of {{throw new RuntimeException();}} use {{throw new
RuntimeException(e);}}
- In SparkTableStorage, please don't use wildcard imports, and cleanup unused
imports.
After Spark 2.x upgrade this patch no longer applies. Can we wait with this
until PIG-5157 is resolved, try to apply the patch on top of it and make the
the necessary changes? For example in Ivy spark configuration no longer exist,
but instead we have spark1 and spark2 configs.
> Support store alias as spark table
> ----------------------------------
>
> Key: PIG-5080
> URL: https://issues.apache.org/jira/browse/PIG-5080
> Project: Pig
> Issue Type: New Feature
> Components: spark
> Affects Versions: 0.16.0
> Reporter: Jeff Zhang
> Assignee: Jeff Zhang
> Fix For: 0.17.1
>
> Attachments: PIG-5080-1.patch, PIG-5080-2.patch
>
>
> The purpose is that I'd like to take advantage of both pig and hive.
> Pig-latin has powerful data flow expression ability which is useful for ETL
> while hive is good at query.
> The scenario is that I'd like to store pig alias as spark temporary table
> (cache can be optional). And I have an another spark engine which share the
> same SparkContext (in the same JVM) to query the table.
> Please close this ticket if it is already supported. I didn't go through all
> the features of pig-spark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
