[
https://issues.apache.org/jira/browse/PIG-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang_intel updated PIG-4871:
----------------------------------
Description:
In current code base, we use OperatorPlan#forceConnect() while merge the
physical plan of spliter and splittee in MultiQueryOptimizationSpark.
{code}
/**
* connect from and to and ignore some judgements: like ignoring judge
whether from operator supports multiOutputs
* and whether to operator supports multiInputs
*
* @param from Operator data will flow from.
* @param to Operator data will flow to.
* @throws PlanException if connect from or to which is not in the plan
*/
public void forceConnect(E from, E to) throws PlanException {
markDirty();
// Check that both nodes are in the plan.
checkInPlan(from);
checkInPlan(to);
mFromEdges.put(from, to);
mToEdges.put(to, from);
}
{code}
was:
It's better to configure some properties of spark like
“spark.eventLog.enabled”,"pig.spark.eventLog.dir","spark.driver.memory" in
SparkLauncher.java.
We can configure these kind of properties in conf/pig.properties
> Not use OperatorPlan#forceConnect in MultiQueryOptimizationSpark
> -----------------------------------------------------------------
>
> Key: PIG-4871
> URL: https://issues.apache.org/jira/browse/PIG-4871
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
>
> In current code base, we use OperatorPlan#forceConnect() while merge the
> physical plan of spliter and splittee in MultiQueryOptimizationSpark.
> {code}
> /**
> * connect from and to and ignore some judgements: like ignoring judge
> whether from operator supports multiOutputs
> * and whether to operator supports multiInputs
> *
> * @param from Operator data will flow from.
> * @param to Operator data will flow to.
> * @throws PlanException if connect from or to which is not in the plan
> */
> public void forceConnect(E from, E to) throws PlanException {
> markDirty();
> // Check that both nodes are in the plan.
> checkInPlan(from);
> checkInPlan(to);
> mFromEdges.put(from, to);
> mToEdges.put(to, from);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
