[jira] [Commented] (SPARK-1395) Cannot launch jobs on Yarn cluster with local: scheme in SPARK_JAR

2014-04-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958165#comment-13958165 ] Marcelo Vanzin commented on SPARK-1395: --- https://github.com/apache/spark/pull/303

[jira] [Commented] (SPARK-1021) sortByKey() launches a cluster job when it shouldn't

2014-04-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964658#comment-13964658 ] Marcelo Vanzin commented on SPARK-1021: --- I actually played with the idea and just

[jira] [Created] (SPARK-1459) EventLoggingListener does not work with file:// target dir

2014-04-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-1459: - Summary: EventLoggingListener does not work with file:// target dir Key: SPARK-1459 URL: https://issues.apache.org/jira/browse/SPARK-1459 Project: Spark

[jira] [Commented] (SPARK-1459) EventLoggingListener does not work with file:// target dir

2014-04-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13964783#comment-13964783 ] Marcelo Vanzin commented on SPARK-1459: --- PR #375 EventLoggingListener does not

[jira] [Created] (SPARK-1518) Spark master doesn't compile against hadoop-common trunk

2014-04-16 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-1518: - Summary: Spark master doesn't compile against hadoop-common trunk Key: SPARK-1518 URL: https://issues.apache.org/jira/browse/SPARK-1518 Project: Spark

[jira] [Resolved] (SPARK-1459) EventLoggingListener does not work with file:// target dir

2014-04-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-1459. --- Resolution: Fixed This was commit 69047506. (If someone with permissions could set me as the

[jira] [Reopened] (SPARK-1459) EventLoggingListener does not work with file:// target dir

2014-04-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-1459: --- Sorry, got confused. This PR is still pending. EventLoggingListener does not work with file://

[jira] [Commented] (SPARK-1631) App name set in SparkConf (not in JVM properties) not respected by Yarn backend

2014-04-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13981337#comment-13981337 ] Marcelo Vanzin commented on SPARK-1631: --- PR:

[jira] [Reopened] (SPARK-1395) Cannot launch jobs on Yarn cluster with local: scheme in SPARK_JAR

2014-04-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-1395: --- This is broken again. Cannot launch jobs on Yarn cluster with local: scheme in SPARK_JAR

[jira] [Created] (SPARK-1768) History Server enhancements

2014-05-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-1768: - Summary: History Server enhancements Key: SPARK-1768 URL: https://issues.apache.org/jira/browse/SPARK-1768 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-1768) History Server enhancements

2014-05-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993185#comment-13993185 ] Marcelo Vanzin commented on SPARK-1768: --- BTW I'm working on this (patch isn't ready

[jira] [Commented] (SPARK-1616) input file not found issue

2014-05-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994044#comment-13994044 ] Marcelo Vanzin commented on SPARK-1616: --- Hi Prasad, This doesn't really sound like

[jira] [Commented] (SPARK-1768) History Server enhancements

2014-05-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995375#comment-13995375 ] Marcelo Vanzin commented on SPARK-1768: --- https://github.com/apache/spark/pull/718

[jira] [Commented] (SPARK-1518) Spark master doesn't compile against hadoop-common trunk

2014-05-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010305#comment-14010305 ] Marcelo Vanzin commented on SPARK-1518: --- Hmm, may I suggest a different approach?

[jira] [Created] (SPARK-2080) Yarn: history UI link missing, wrong reported user

2014-06-09 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2080: - Summary: Yarn: history UI link missing, wrong reported user Key: SPARK-2080 URL: https://issues.apache.org/jira/browse/SPARK-2080 Project: Spark Issue

[jira] [Commented] (SPARK-2080) Yarn: history UI link missing, wrong reported user

2014-06-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14025346#comment-14025346 ] Marcelo Vanzin commented on SPARK-2080: --- Patrick / someone, I can't seem to be able

[jira] [Commented] (SPARK-2007) Spark on YARN picks up hadoop log4j.properties even if SPARK_LOG4J_CONF is set

2014-06-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14026952#comment-14026952 ] Marcelo Vanzin commented on SPARK-2007: --- I believe

[jira] [Created] (SPARK-2098) All Spark processes should support spark-defaults.conf, config file

2014-06-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2098: - Summary: All Spark processes should support spark-defaults.conf, config file Key: SPARK-2098 URL: https://issues.apache.org/jira/browse/SPARK-2098 Project: Spark

[jira] [Updated] (SPARK-2098) All Spark processes should support spark-defaults.conf, config file

2014-06-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-2098: -- Issue Type: Improvement (was: Bug) All Spark processes should support spark-defaults.conf,

[jira] [Created] (SPARK-2169) SparkUI.setAppName() has no effect

2014-06-17 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2169: - Summary: SparkUI.setAppName() has no effect Key: SPARK-2169 URL: https://issues.apache.org/jira/browse/SPARK-2169 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-2261) Spark application event logs are not very NameNode-friendly

2014-06-24 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2261: - Summary: Spark application event logs are not very NameNode-friendly Key: SPARK-2261 URL: https://issues.apache.org/jira/browse/SPARK-2261 Project: Spark

[jira] [Created] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

2014-06-24 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2268: - Summary: Utils.createTempDir() creates race with HDFS at shutdown Key: SPARK-2268 URL: https://issues.apache.org/jira/browse/SPARK-2268 Project: Spark

[jira] [Commented] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

2014-06-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043079#comment-14043079 ] Marcelo Vanzin commented on SPARK-2268: --- I'll look more closely, but we have users

[jira] [Resolved] (SPARK-2268) Utils.createTempDir() creates race with HDFS at shutdown

2014-06-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-2268. --- Resolution: Invalid Looking at the logs from the user there are multiple exceptions being

[jira] [Commented] (SPARK-2167) spark-submit should return exit code based on failure/success

2014-06-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048175#comment-14048175 ] Marcelo Vanzin commented on SPARK-2167: --- Given that spark-submit doesn't really have

[jira] [Commented] (SPARK-2444) Make spark.yarn.executor.memoryOverhead a first class citizen

2014-07-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058987#comment-14058987 ] Marcelo Vanzin commented on SPARK-2444: --- As Tom mentions, the setting is documented.

[jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts

2014-07-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14070966#comment-14070966 ] Marcelo Vanzin commented on SPARK-2420: --- I'm all for sanitizing dependencies, but

[jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts

2014-07-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072259#comment-14072259 ] Marcelo Vanzin commented on SPARK-2420: --- Hi Sean, I agree in part about the

[jira] [Commented] (SPARK-2633) support register spark listener to listener bus with Java API

2014-07-23 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072579#comment-14072579 ] Marcelo Vanzin commented on SPARK-2633: --- So, being able to register listeners is

[jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts

2014-07-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14074596#comment-14074596 ] Marcelo Vanzin commented on SPARK-2420: --- After some brainstorming, the path of least

[jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts

2014-07-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076456#comment-14076456 ] Marcelo Vanzin commented on SPARK-2420: --- So let me see if I'm following things so

[jira] [Commented] (SPARK-2420) Change Spark build to minimize library conflicts

2014-07-28 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14076488#comment-14076488 ] Marcelo Vanzin commented on SPARK-2420: --- Forking {{Optional}} would make Option 2

[jira] [Commented] (SPARK-2744) The configuration spark.history.retainedApplications is invalid

2014-07-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14079497#comment-14079497 ] Marcelo Vanzin commented on SPARK-2744: --- Are you sure that option means what you

[jira] [Commented] (SPARK-2744) The configuration spark.history.retainedApplications is invalid

2014-07-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081191#comment-14081191 ] Marcelo Vanzin commented on SPARK-2744: --- Sort of. The docs say: {quote} The number

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2014-07-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081465#comment-14081465 ] Marcelo Vanzin commented on SPARK-1537: --- I'm working on this but this all sort of

[jira] [Commented] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line

2014-07-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081490#comment-14081490 ] Marcelo Vanzin commented on SPARK-1576: --- With Sandy's recent patch

[jira] [Commented] (SPARK-1131) Better document the --args option for yarn-standalone mode

2014-07-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081497#comment-14081497 ] Marcelo Vanzin commented on SPARK-1131: --- This is probably obsolete now with

[jira] [Created] (SPARK-2778) Add unit tests for Yarn integration

2014-07-31 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2778: - Summary: Add unit tests for Yarn integration Key: SPARK-2778 URL: https://issues.apache.org/jira/browse/SPARK-2778 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-2718) YARN does not handle spark configs with quotes or backslashes

2014-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-2718: -- Ran into this while working on some other stuff, so I'll work on a fix. YARN does not handle

[jira] [Created] (SPARK-2845) Add timestamp to BlockManager events

2014-08-04 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2845: - Summary: Add timestamp to BlockManager events Key: SPARK-2845 URL: https://issues.apache.org/jira/browse/SPARK-2845 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-2848) Shade Guava in Spark deliverables

2014-08-04 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2848: - Summary: Shade Guava in Spark deliverables Key: SPARK-2848 URL: https://issues.apache.org/jira/browse/SPARK-2848 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-2636) no where to get job identifier while submit spark job through spark API

2014-08-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086494#comment-14086494 ] Marcelo Vanzin commented on SPARK-2636: --- (BTW, just checked SPARK-2321, so if you

[jira] [Commented] (SPARK-2848) Shade Guava in Spark deliverables

2014-08-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086670#comment-14086670 ] Marcelo Vanzin commented on SPARK-2848: --- Question for others ([~pwendell], [~sowen],

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2014-08-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086817#comment-14086817 ] Marcelo Vanzin commented on SPARK-1537: --- Currently busy with other more urgent

[jira] [Commented] (SPARK-2848) Shade Guava in Spark deliverables

2014-08-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087179#comment-14087179 ] Marcelo Vanzin commented on SPARK-2848: --- Nevermind the question, I got code mostly

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2014-08-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088438#comment-14088438 ] Marcelo Vanzin commented on SPARK-1537: --- Current code is here:

[jira] [Created] (SPARK-2932) Move MasterFailureTest out of main source directory

2014-08-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2932: - Summary: Move MasterFailureTest out of main source directory Key: SPARK-2932 URL: https://issues.apache.org/jira/browse/SPARK-2932 Project: Spark Issue

[jira] [Created] (SPARK-2933) Cleanup unnecessary and duplicated code in Yarn module

2014-08-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2933: - Summary: Cleanup unnecessary and duplicated code in Yarn module Key: SPARK-2933 URL: https://issues.apache.org/jira/browse/SPARK-2933 Project: Spark Issue

[jira] [Updated] (SPARK-2933) Cleanup unnecessary and duplicated code in Yarn module

2014-08-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-2933: -- Component/s: YARN Cleanup unnecessary and duplicated code in Yarn module

[jira] [Updated] (SPARK-2932) Move MasterFailureTest out of main source directory

2014-08-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-2932: -- Component/s: Streaming Move MasterFailureTest out of main source directory

[jira] [Created] (SPARK-2996) Standalone and Yarn have different settings for adding the user classpath first

2014-08-12 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-2996: - Summary: Standalone and Yarn have different settings for adding the user classpath first Key: SPARK-2996 URL: https://issues.apache.org/jira/browse/SPARK-2996

[jira] [Commented] (SPARK-1537) Add integration with Yarn's Application Timeline Server

2014-08-21 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106164#comment-14106164 ] Marcelo Vanzin commented on SPARK-1537: --- No concrete timeline at the moment. I'm

[jira] [Commented] (SPARK-3107) Don't pass null jar to executor in yarn-client mode

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107476#comment-14107476 ] Marcelo Vanzin commented on SPARK-3107: --- The {{--jar}} issue is fixed in SPARK-2933.

[jira] [Commented] (SPARK-3101) Missing volatile annotation in ApplicationMaster

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107478#comment-14107478 ] Marcelo Vanzin commented on SPARK-3101: --- I covered this in the PR for SPARK-2933

[jira] [Commented] (SPARK-3102) Add tests for yarn-client mode

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107480#comment-14107480 ] Marcelo Vanzin commented on SPARK-3102: --- SPARK-2778? Add tests for yarn-client

[jira] [Commented] (SPARK-3099) Staging Directory is never deleted when we run job with YARN Client Mode

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107481#comment-14107481 ] Marcelo Vanzin commented on SPARK-3099: --- Pretty sure I covered this in the PR for

[jira] [Commented] (SPARK-3090) Avoid not stopping SparkContext with YARN Client mode

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107488#comment-14107488 ] Marcelo Vanzin commented on SPARK-3090: --- I think that if we want to add this, it

[jira] [Commented] (SPARK-2140) yarn stable client doesn't properly handle MEMORY_OVERHEAD for AM

2014-08-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107499#comment-14107499 ] Marcelo Vanzin commented on SPARK-2140: --- Same as SPARK-1287? yarn stable client

[jira] [Created] (SPARK-3187) Refactor and cleanup Yarn allocator code

2014-08-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3187: - Summary: Refactor and cleanup Yarn allocator code Key: SPARK-3187 URL: https://issues.apache.org/jira/browse/SPARK-3187 Project: Spark Issue Type:

[jira] [Created] (SPARK-3215) Add remote interface for SparkContext

2014-08-25 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3215: - Summary: Add remote interface for SparkContext Key: SPARK-3215 URL: https://issues.apache.org/jira/browse/SPARK-3215 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-3215) Add remote interface for SparkContext

2014-08-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-3215: -- Attachment: RemoteSparkContext.pdf Initial proposal for a remote context interface. Note that

[jira] [Commented] (SPARK-3217) Shaded Guava jar doesn't play well with Maven build

2014-08-25 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14110035#comment-14110035 ] Marcelo Vanzin commented on SPARK-3217: --- Just did a git clean -dfx on master and

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112780#comment-14112780 ] Marcelo Vanzin commented on SPARK-3215: --- Hi Reynold, thanks for the comments. This

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112803#comment-14112803 ] Marcelo Vanzin commented on SPARK-3215: --- Hi Matei, Both suggestions came up during

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112958#comment-14112958 ] Marcelo Vanzin commented on SPARK-3215: --- I think the what RPC to use discussion is

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113062#comment-14113062 ] Marcelo Vanzin commented on SPARK-3215: --- Hi Matei, sorry if I'm missing what you're

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-08-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113206#comment-14113206 ] Marcelo Vanzin commented on SPARK-3215: --- Matei, yes, all those things exist, but

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-09-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118960#comment-14118960 ] Marcelo Vanzin commented on SPARK-3215: --- For those who'd prefer to see some code,

[jira] [Created] (SPARK-3388) Expose cluster applicationId, use it to serve history information

2014-09-03 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3388: - Summary: Expose cluster applicationId, use it to serve history information Key: SPARK-3388 URL: https://issues.apache.org/jira/browse/SPARK-3388 Project: Spark

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-09-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120708#comment-14120708 ] Marcelo Vanzin commented on SPARK-3215: --- Thanks Matei. Looking at a Java API is next

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-09-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123749#comment-14123749 ] Marcelo Vanzin commented on SPARK-3215: --- I updated the prototype to include a Java

[jira] [Created] (SPARK-3446) FutureAction should expose the job ID

2014-09-08 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3446: - Summary: FutureAction should expose the job ID Key: SPARK-3446 URL: https://issues.apache.org/jira/browse/SPARK-3446 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-3450) Enable specifiying the --jars CLI option multiple times

2014-09-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127533#comment-14127533 ] Marcelo Vanzin commented on SPARK-3450: --- My only concern is that adding this would

[jira] [Commented] (SPARK-3215) Add remote interface for SparkContext

2014-09-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127564#comment-14127564 ] Marcelo Vanzin commented on SPARK-3215: --- For those following, I moved the prototype

[jira] [Commented] (SPARK-3492) Clean up Yarn integration code

2014-09-11 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14130736#comment-14130736 ] Marcelo Vanzin commented on SPARK-3492: --- If you want to keep track, SPARK-3187 is

[jira] [Resolved] (SPARK-1576) Passing of JAVA_OPTS to YARN on command line

2014-09-18 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-1576. --- Resolution: Not a Problem spark-submit already supports this with existing options. Passing

[jira] [Created] (SPARK-3606) Spark-on-Yarn AmIpFilter does not work with Yarn HA.

2014-09-19 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3606: - Summary: Spark-on-Yarn AmIpFilter does not work with Yarn HA. Key: SPARK-3606 URL: https://issues.apache.org/jira/browse/SPARK-3606 Project: Spark Issue

[jira] [Created] (SPARK-3647) Shaded Guava patch causes access issues with package private classes

2014-09-22 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3647: - Summary: Shaded Guava patch causes access issues with package private classes Key: SPARK-3647 URL: https://issues.apache.org/jira/browse/SPARK-3647 Project: Spark

[jira] [Commented] (SPARK-3647) Shaded Guava patch causes access issues with package private classes

2014-09-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143796#comment-14143796 ] Marcelo Vanzin commented on SPARK-3647: --- There are two options I see here: - extend

[jira] [Commented] (SPARK-2848) Shade Guava in Spark deliverables

2014-09-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143885#comment-14143885 ] Marcelo Vanzin commented on SPARK-2848: --- Yes, that's right, this was pushed onto

[jira] [Updated] (SPARK-2848) Shade Guava in Spark deliverables

2014-09-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-2848: -- Fix Version/s: (was: 1.1.0) 1.2.0 Shade Guava in Spark deliverables

[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()

2014-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152294#comment-14152294 ] Marcelo Vanzin commented on SPARK-3732: --- I think that explicit System.exit() could

[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()

2014-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152309#comment-14152309 ] Marcelo Vanzin commented on SPARK-3732: --- Removing the call should work regardless;

[jira] [Commented] (SPARK-3732) Yarn Client: Add option to NOT System.exit() at end of main()

2014-09-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152318#comment-14152318 ] Marcelo Vanzin commented on SPARK-3732: --- BTW, if the call is removed, it should be

[jira] [Commented] (SPARK-2778) Add unit tests for Yarn integration

2014-09-30 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153429#comment-14153429 ] Marcelo Vanzin commented on SPARK-2778: --- Patrick, would it be possible to get all

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157351#comment-14157351 ] Marcelo Vanzin commented on SPARK-3633: --- Hey [~pwendell] [~matei], is anyone

[jira] [Commented] (SPARK-3710) YARN integration test is flaky

2014-10-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158364#comment-14158364 ] Marcelo Vanzin commented on SPARK-3710: --- Hmm. For some the e-mail for this bug ended

[jira] [Commented] (SPARK-3710) YARN integration test is flaky

2014-10-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158376#comment-14158376 ] Marcelo Vanzin commented on SPARK-3710: --- I filed a Yarn bug (YARN-2642), although we

[jira] [Created] (SPARK-3788) Yarn dist cache code is not friendly to HDFS HA, Federation

2014-10-03 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-3788: - Summary: Yarn dist cache code is not friendly to HDFS HA, Federation Key: SPARK-3788 URL: https://issues.apache.org/jira/browse/SPARK-3788 Project: Spark

[jira] [Commented] (SPARK-3788) Yarn dist cache code is not friendly to HDFS HA, Federation

2014-10-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158657#comment-14158657 ] Marcelo Vanzin commented on SPARK-3788: --- Note: 2 above only applies to branch-1.1.

[jira] [Commented] (SPARK-3788) Yarn dist cache code is not friendly to HDFS HA, Federation

2014-10-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158710#comment-14158710 ] Marcelo Vanzin commented on SPARK-3788: --- Ah, 2 was fixed in branch-1.1 as part of

[jira] [Resolved] (SPARK-3718) FsHistoryProvider should consider spark.eventLog.dir not only spark.history.fs.logDirectory

2014-10-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-3718. --- Resolution: Not a Problem I think it was decided that this is not a feature we want (see

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

2014-10-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161120#comment-14161120 ] Marcelo Vanzin commented on SPARK-3633: --- [~matei], as Nishkam mentions above, we

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-07 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162514#comment-14162514 ] Marcelo Vanzin commented on SPARK-3174: --- Hi Andrew, thanks for writing this up. My

[jira] [Commented] (SPARK-3873) Scala style: check import ordering

2014-10-09 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165916#comment-14165916 ] Marcelo Vanzin commented on SPARK-3873: --- Actually looking at this, since I've been

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171580#comment-14171580 ] Marcelo Vanzin commented on SPARK-3174: --- [~andrewor14] still regarding the shuffle

[jira] [Commented] (SPARK-3873) Scala style: check import ordering

2014-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172552#comment-14172552 ] Marcelo Vanzin commented on SPARK-3873: --- It doesn't seem like I can unassing bugs...

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-15 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172861#comment-14172861 ] Marcelo Vanzin commented on SPARK-3174: --- [~praveenseluka] I took a quick look at

[jira] [Commented] (SPARK-3174) Provide elastic scaling within a Spark application

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173951#comment-14173951 ] Marcelo Vanzin commented on SPARK-3174: --- bq. Lets say we start the Spark

[jira] [Commented] (SPARK-3883) Provide SSL support for Akka and HttpServer based connections

2014-10-16 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14174005#comment-14174005 ] Marcelo Vanzin commented on SPARK-3883: --- FYI, any PR here should make sure the

  1   2   3   4   5   6   7   8   9   10   >