[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333410#comment-15333410
]
Srikanth Sundarrajan commented on PIG-4903:
---
For spark mode, we should support yarn-client, mesos
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1585#comment-1585
]
Srikanth Sundarrajan commented on PIG-4903:
---
Sorry to nitpick [~kellyzly], would this print a 0/1
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1532#comment-1532
]
Srikanth Sundarrajan commented on PIG-4903:
---
Looks largely ok, except the following:
{code
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325646#comment-15325646
]
Srikanth Sundarrajan commented on PIG-4903:
---
Makes sense to keep it consistent. Should we
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15324640#comment-15324640
]
Srikanth Sundarrajan commented on PIG-4903:
---
Yes [~rohini], By default (particularly
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321850#comment-15321850
]
Srikanth Sundarrajan commented on PIG-4903:
---
Looked at the latest patch, it still seems like
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313486#comment-15313486
]
Srikanth Sundarrajan commented on PIG-4903:
---
Have two followup questions on the patch.
1
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309216#comment-15309216
]
Srikanth Sundarrajan commented on PIG-4903:
---
Two things to consider.
1. Since Spark Assembly
[
https://issues.apache.org/jira/browse/PIG-4903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303400#comment-15303400
]
Srikanth Sundarrajan commented on PIG-4903:
---
If spark-assembly is present, then nothing else
Congratulations Xuefu
Regards,
Srikanth Sundarrajan
On Thu, Feb 25, 2016 at 10:07 AM Xuefu Zhang <xzh...@cloudera.com> wrote:
> Thank you, Liyun! You did the hard work. I think you well deserve a
> committership once we merge the branch to trunk.
>
> --Xuefu
>
> On Wed,
[
https://issues.apache.org/jira/browse/PIG-4746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan reassigned PIG-4746:
-
Assignee: Srikanth Sundarrajan
> Ensure spork can be run as PIG action in Oo
[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021754#comment-15021754
]
Srikanth Sundarrajan commented on PIG-4741:
---
[~kellyzly], We did attempt to support in
[yarn
[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021488#comment-15021488
]
Srikanth Sundarrajan commented on PIG-4741:
---
SPARK_DIST_CLASSPATH is used by AM launcher via Node
[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021021#comment-15021021
]
Srikanth Sundarrajan commented on PIG-4741:
---
If there is an issue, you can consider replacing
[
https://issues.apache.org/jira/browse/PIG-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020964#comment-15020964
]
Srikanth Sundarrajan commented on PIG-4741:
---
"\" before ${PWD} is required to escape su
[
https://issues.apache.org/jira/browse/PIG-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14980088#comment-14980088
]
Srikanth Sundarrajan commented on PIG-4720:
---
Patch looks good. Verified build as well. +1
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974015#comment-14974015
]
Srikanth Sundarrajan commented on PIG-4698:
---
FYKA [~xuefuz] / [~mohitsabharwal]
> Enable dyna
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4698:
--
Status: Patch Available (was: Open)
> Enable dynamic resource allocation/de-allocat
service need to be enabled at the node manager
and application level for this to work correctly.
Thanks,
Srikanth Sundarrajan
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966876#comment-14966876
]
Srikanth Sundarrajan commented on PIG-4698:
---
Yes. We need to enable auxillary shuffle service
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966590#comment-14966590
]
Srikanth Sundarrajan commented on PIG-4698:
---
Or more tersely
{code}
set
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966587#comment-14966587
]
Srikanth Sundarrajan commented on PIG-4698:
---
Am looking at propagating conf set in pig script
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4693:
--
Attachment: PIG-4693.patch
Verified that OrcStorage works well with kryo-2.21. Used
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4693:
--
Status: Patch Available (was: Open)
> Class conflicts: Kryo bundled in spark vs k
[
https://issues.apache.org/jira/browse/PIG-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951708#comment-14951708
]
Srikanth Sundarrajan commented on PIG-4680:
---
[~rohini], [~daijy], Would really appreciate some
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948889#comment-14948889
]
Srikanth Sundarrajan commented on PIG-4693:
---
[~xuefuz], From what I can tell, switching
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948991#comment-14948991
]
Srikanth Sundarrajan commented on PIG-4693:
---
Thanks [~vanzin]. The trunk version of chill still
Srikanth Sundarrajan created PIG-4698:
-
Summary: Enable dynamic resource allocation/de-allocation on Yarn
backends
Key: PIG-4698
URL: https://issues.apache.org/jira/browse/PIG-4698
Project: Pig
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4698:
--
Description: Resource elasticity needs to be enabled on Yarn backend to
allow jobs
[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949904#comment-14949904
]
Srikanth Sundarrajan commented on PIG-4698:
---
There are a couple of options on how we can go about
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14946525#comment-14946525
]
Srikanth Sundarrajan commented on PIG-4693:
---
Issue persists even with usage with spark-core
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948068#comment-14948068
]
Srikanth Sundarrajan commented on PIG-4693:
---
That might not help. As this is a direct dependency
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14947062#comment-14947062
]
Srikanth Sundarrajan commented on PIG-4693:
---
Yes. After the AM is launched, client makes a request
Srikanth Sundarrajan created PIG-4693:
-
Summary: Class conflicts Kryo bundled in spark vs kryo bundled
with pig
Key: PIG-4693
URL: https://issues.apache.org/jira/browse/PIG-4693
Project: Pig
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4693:
--
Summary: Class conflicts: Kryo bundled in spark vs kryo bundled with pig
(was: Class
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944499#comment-14944499
]
Srikanth Sundarrajan commented on PIG-4693:
---
[~xuefuz], [~mohitz], [~kexianda], Thoughts welcome
[
https://issues.apache.org/jira/browse/PIG-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14944494#comment-14944494
]
Srikanth Sundarrajan commented on PIG-4693:
---
Running the following simple pig script
{code
[
https://issues.apache.org/jira/browse/PIG-4236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909169#comment-14909169
]
Srikanth Sundarrajan commented on PIG-4236:
---
Am assuming PIG-4667 has already addressed
[
https://issues.apache.org/jira/browse/PIG-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909168#comment-14909168
]
Srikanth Sundarrajan commented on PIG-4681:
---
In the cluster mode the Spark Context will be created
[
https://issues.apache.org/jira/browse/PIG-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14802876#comment-14802876
]
Srikanth Sundarrajan commented on PIG-4680:
---
This can be quite handy, particularly when pig
[
https://issues.apache.org/jira/browse/PIG-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4680:
--
Assignee: Abhishek Agarwal
> Enable pig job graphs to resume from last successful st
Srikanth Sundarrajan created PIG-4681:
-
Summary: Enable Pig on Spark to run on Yarn Cluster mode
Key: PIG-4681
URL: https://issues.apache.org/jira/browse/PIG-4681
Project: Pig
Issue Type
[
https://issues.apache.org/jira/browse/PIG-4681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4681:
--
Labels: spork (was: spark)
> Enable Pig on Spark to run on Yarn Cluster m
HADOOP_CONF_DIR="/opt/hadoop-2.6.0.2.2.0.0-2041/etc/hadoop/"
declare -x HADOOP_HOME="/opt/hadoop-2.6.0.2.2.0.0-2041/"
declare -x SPARK_MASTER="yarn-client"
Thanks,
Srikanth Sundarrajan
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4667:
--
Attachment: PIG-4667-v1.patch
Hi [~xuefuz], Have attached a revised patch which removes
1/etc/hadoop/"
declare -x HADOOP_HOME="/opt/hadoop-2.6.0.2.2.0.0-2041/"
declare -x SPARK_MASTER="yarn-client"
Thanks,
Srikanth Sundarrajan
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4667:
--
Summary: Enable Pig on Spark to run on Yarn Client mode (was: Enable Pig
on Spark to run
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4667:
--
Status: Patch Available (was: Open)
> Enable Pig on Spark to run on Yarn Client m
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srikanth Sundarrajan updated PIG-4667:
--
Attachment: PIG-4667.patch
Was able to use SPARK_DIST_CLASSPATH env variable to set
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14734656#comment-14734656
]
Srikanth Sundarrajan commented on PIG-4667:
---
[~xuefuz], Jars that are added to the spark context
[
https://issues.apache.org/jira/browse/PIG-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733564#comment-14733564
]
Srikanth Sundarrajan commented on PIG-4667:
---
Here is a quick update on this
It seems a bit tricky
51 matches
Mail list logo
