[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172741#comment-16172741
 ] 

liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:47 AM:
----------------------------------------------------------------

{quote}
I also checked, test-tez was not running properly since the Spark 2 support 
commit, because setTezEnv was clearing the excluded sources property. I fixed 
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in  the dependency of 
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset 
{{src.exclude.dir}} , this will influence {{jar}} which use the property 
{{src.exclude.dir}}?
{code}
 <target name="jar" description="Create pig jar with Spark 1 and 2">
        <echo>Compiling against Spark 2</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="2"/>
        <propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
        <antcall target="jar-core" inheritRefs="true" inheritall="true"/>
        <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>

        <echo>Compiling against Spark 1</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="1"/>
        <propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
        <antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
        <jar update="yes" jarfile="${output.jarfile.core}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <delete file="${basedir}/_pig-shims.jar"/>
    </target>
{code}

If my understanding is right, why TestEvalPipeline unit test passes in tez mode 
before this jira?



was (Author: kellyzly):
{quote}
I also checked, test-tez was not running properly since the Spark 2 support 
commit, because setTezEnv was clearing the excluded sources property. I fixed 
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in  the dependency of 
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset 
{{src.exclude.dir}} , this will influence {{jar}} which use the property 
{{src.exclude.dir}}?
{code}
 <target name="jar" description="Create pig jar with Spark 1 and 2">
        <echo>Compiling against Spark 2</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="2"/>
        <propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
        <antcall target="jar-core" inheritRefs="true" inheritall="true"/>
        <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>

        <echo>Compiling against Spark 1</echo>
        <antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
        <propertyreset name="sparkversion" value="1"/>
        <propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
        <antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
        <jar update="yes" jarfile="${output.jarfile.core}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
            <zipfileset src="${basedir}/_pig-shims.jar" 
includes="**/Spark2*.class"/>
        </jar>
        <delete file="${basedir}/_pig-shims.jar"/>
    </target>


{code}


> Enable yarn-client mode execution of tests in Spark (1) mode
> ------------------------------------------------------------
>
>                 Key: PIG-5305
>                 URL: https://issues.apache.org/jira/browse/PIG-5305
>             Project: Pig
>          Issue Type: Sub-task
>          Components: spark
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>         Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to