[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode
[
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172741#comment-16172741
]
liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:48 AM:
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
Compiling against Spark 2
Compiling against Spark 1
{code}
If my understanding is right, why TestEvalPipeline unit test passes in tez mode
before this jira? Seem there are no unit test failures in tez mode before this
jira.
was (Author: kellyzly):
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
Compiling against Spark 2
Compiling against Spark 1
{code}
If my understanding is right, why TestEvalPipeline unit test passes in tez mode
before this jira?
> Enable yarn-client mode execution of tests in Spark (1) mode
>
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
> Issue Type: Sub-task
> Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode
[
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172741#comment-16172741
]
liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:47 AM:
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
Compiling against Spark 2
Compiling against Spark 1
{code}
If my understanding is right, why TestEvalPipeline unit test passes in tez mode
before this jira?
was (Author: kellyzly):
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
Compiling against Spark 2
Compiling against Spark 1
{code}
> Enable yarn-client mode execution of tests in Spark (1) mode
>
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
> Issue Type: Sub-task
> Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode
[
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167284#comment-16167284
]
liyunzhang_intel edited comment on PIG-5305 at 9/15/17 3:49 AM:
[~szita]: several suggestions
1. can we only add {{pigtest-jar}} to {{test-spark}} target in build.xml? I
guess for {{test-tez}}, there is no need for {{pigtest-jar}}
meanwhile there is no need to add {{jar-simple}} in the dependency of
{{test-tez}} as the dependency of {{compile-test}} includes {{jar-simple}}. If
my understanding is not right, tell me.
2.please add comment {{added feature to re-initialize SparkContext when
switching between cluster and local mode PigServers}} on related code.
Besides, is there any unit test failures if convert {{SPARK_MASTER}} from
"local" to "yarn-client"?
was (Author: kellyzly):
[~szita]: several suggestions
1. can we only add {{pigtest-jar}} to {{test-spark}} target in build.xml? I
guess for {{test-tez}}, there is no need for {{pigtest-jar}}
meanwhile there is no need to add {{jar-simple}} in the dependency of
{{test-tez}} as the dependency of {{compile-test}} includes {{jar-simple}}. If
my understanding is not right, tell me.
2.please add comment {{added feature to re-initialize SparkContext when
switching between cluster and local mode PigServers}} on related code.
Besides, is there any unit test failures if convert {{SPARK_MASTER}} from
"local" to "yarn-client"?
> Enable yarn-client mode execution of tests in Spark (1) mode
>
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
> Issue Type: Sub-task
> Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
