[jira] [Created] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

Saisai Shao created SPARK-12977: --- Summary: Factoring out StreamingListener and UI to support history UI Key: SPARK-12977 URL: https://issues.apache.org/jira/browse/SPARK-12977 Project: Spark

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111243#comment-15111243 ] Saisai Shao commented on SPARK-11045: - Hi [~dibbhatt], I'm afraid I could not agree with your comment

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao commented on SPARK-12140: - Hi guys, I though a bit on this feature, besides this one big

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -- Hi guys, I

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -- Hi guys, I

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation document for removing executors with cached data differs in different sections

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15107085#comment-15107085 ] Saisai Shao commented on SPARK-12883: - I get your point now. But I think these two descriptions are

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105671#comment-15105671 ] Saisai Shao commented on SPARK-12864: - What's Spark version are you using? I remember I fixed a

[jira] [Comment Edited] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15105671#comment-15105671 ] Saisai Shao edited comment on SPARK-12864 at 1/18/16 7:03 PM: -- What Spark

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Attachment: Screen Shot 2016-01-18 at 3.47.24 PM.png > RM redirects to incorrect URL in Spark

[jira] [Created] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

Saisai Shao created SPARK-12893: --- Summary: RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode Key: SPARK-12893 URL: https://issues.apache.org/jira/browse/SPARK-12893 Project:

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Description: This will cause application not found error, the screenshot shows below:

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation doc still refers to 1.2

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106005#comment-15106005 ] Saisai Shao commented on SPARK-12883: - I think this doc is still valid, current way of setting a

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106214#comment-15106214 ] Saisai Shao commented on SPARK-12864: - So the problem should be that: {{BlockManager}} should be

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Attachment: screenshot-1.png > Prepending base URI of job description is missing >

[jira] [Created] (SPARK-12673) Prepending base URI of job description is missing

Saisai Shao created SPARK-12673: --- Summary: Prepending base URI of job description is missing Key: SPARK-12673 URL: https://issues.apache.org/jira/browse/SPARK-12673 Project: Spark Issue Type:

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15084497#comment-15084497 ] Saisai Shao commented on SPARK-12650: - [~vines], what is your meaning of "SparkSubmit does not Xmx

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082085#comment-15082085 ] Saisai Shao commented on SPARK-12516: - Thanks a lot [~vanzin] for your reply. Looks like

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15075693#comment-15075693 ] Saisai Shao commented on SPARK-12516: - Hi [~vanzin], what is your suggestion of this issue? I'm

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15074588#comment-15074588 ] Saisai Shao commented on SPARK-12554: - For case 2, I think it is really a misconfiguration problem,

[jira] [Created] (SPARK-12552) Recovered driver's resource is not counted in the Master

Saisai Shao created SPARK-12552: --- Summary: Recovered driver's resource is not counted in the Master Key: SPARK-12552 URL: https://issues.apache.org/jira/browse/SPARK-12552 Project: Spark Issue

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073954#comment-15073954 ] Saisai Shao commented on SPARK-12554: - First from my understanding tasks can be scheduled after

[jira] [Commented] (SPARK-11782) Master Web UI should link to correct Application UI in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073430#comment-15073430 ] Saisai Shao commented on SPARK-11782: - I just verified with latest master branch, seems no such

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Updated] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

[ https://issues.apache.org/jira/browse/SPARK-12447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12447: Description: Currently {{YarnAllocator}} will update its managed states like

[jira] [Commented] (SPARK-12514) Spark MetricsSystem can fill disks/cause OOMs when using GangliaSink

[ https://issues.apache.org/jira/browse/SPARK-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15070717#comment-15070717 ] Saisai Shao commented on SPARK-12514: - Since we need to differentiate the metrics between

[jira] [Created] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

Saisai Shao created SPARK-12516: --- Summary: Properly handle NM failure situation for Spark on Yarn Key: SPARK-12516 URL: https://issues.apache.org/jira/browse/SPARK-12516 Project: Spark Issue

[jira] [Created] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

Saisai Shao created SPARK-12447: --- Summary: Only update AM's internal state when executor is successfully launched by NM Key: SPARK-12447 URL: https://issues.apache.org/jira/browse/SPARK-12447 Project:

[jira] [Commented] (SPARK-10500) sparkr.zip cannot be created if $SPARK_HOME/R/lib is unwritable

[ https://issues.apache.org/jira/browse/SPARK-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061730#comment-15061730 ] Saisai Shao commented on SPARK-10500: - [~sunrui] It would be better to back port to 1.5 if possible,

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063359#comment-15063359 ] Saisai Shao commented on SPARK-12400: - So from my understanding this will only be existed when

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15063372#comment-15063372 ] Saisai Shao commented on SPARK-12400: - [~rxin], would you mind me taking a crack at this issue,

[jira] [Commented] (SPARK-12384) Allow -Xms to be set differently then -Xmx

[ https://issues.apache.org/jira/browse/SPARK-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061639#comment-15061639 ] Saisai Shao commented on SPARK-12384: - IIUC, there's also another limitation in container level. For

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059937#comment-15059937 ] Saisai Shao commented on SPARK-12345: - I think by default Spark Mesos implementation will ship all

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059954#comment-15059954 ] Saisai Shao commented on SPARK-12345: - Having a quick test by not exporting {{SPARK_HOME}}, the

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15060035#comment-15060035 ] Saisai Shao commented on SPARK-12345: - Here is the one solution

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059778#comment-15059778 ] Saisai Shao commented on SPARK-12345: - A simple solution is to change the scripts to not expose

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059569#comment-15059569 ] Saisai Shao commented on SPARK-12345: - It is OK in my local test when I followed the step one by one

[jira] [Commented] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055508#comment-15055508 ] Saisai Shao commented on SPARK-12176: - It is OK in my local test against latest master branch, seems

[jira] [Commented] (SPARK-9059) Update Python Direct Kafka Word count examples to show the use of HasOffsetRanges

[ https://issues.apache.org/jira/browse/SPARK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049804#comment-15049804 ] Saisai Shao commented on SPARK-9059: HasOffsetRanges also has python version, which is added in

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049958#comment-15049958 ] Saisai Shao commented on SPARK-6735: I've submitted a patch to continue this work

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046602#comment-15046602 ] Saisai Shao commented on SPARK-12178: - This is a good idea to make it generic if there's more direct

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041260#comment-15041260 ] Saisai Shao commented on SPARK-10123: - Hi [~vanzin], would you mind letting me take a crack on this?

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15041301#comment-15041301 ] Saisai Shao commented on SPARK-12103: - I think I had a proposal of message handler (receiver

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042598#comment-15042598 ] Saisai Shao commented on SPARK-10123: - Just confirm if it is on your plan, in case duplicated :). >

[jira] [Comment Edited] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034997#comment-15034997 ] Saisai Shao edited comment on SPARK-12059 at 12/2/15 12:47 AM: --- A simple

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034997#comment-15034997 ] Saisai Shao commented on SPARK-12059: - A simple solution is to loose the condition or remove the

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035350#comment-15035350 ] Saisai Shao commented on SPARK-12059: - I see, so I will relax the condition to avoid exception from

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032806#comment-15032806 ] Saisai Shao commented on SPARK-12059: - Thanks a lot [~andrewor14], I will look into this issue. >

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033268#comment-15033268 ] Saisai Shao commented on SPARK-12059: - Hi [~andrewor14], when will this be happened? I suppose state

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031308#comment-15031308 ] Saisai Shao commented on SPARK-12009: - OK, I see. So how about setting

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029425#comment-15029425 ] Saisai Shao commented on SPARK-12009: - So I guess your problem is that after you call {{sc.stop()}}

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029416#comment-15029416 ] Saisai Shao commented on SPARK-12009: - So what actually version of Spark you're running? 1.4.0 or

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029416#comment-15029416 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:30 AM: --- So what

[jira] [Commented] (SPARK-12002) offsetRanges attribute missing in Kafka RDD when resuming from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029421#comment-15029421 ] Saisai Shao commented on SPARK-12002: - Looks like because Python `KafkaTransformDStream` specific

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029431#comment-15029431 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:50 AM: --- Alright, my

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029431#comment-15029431 ] Saisai Shao commented on SPARK-12009: - Alright, my code is master branch. Anyway I understood your

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028340#comment-15028340 ] Saisai Shao commented on SPARK-12009: - Looking at the code again, {{onDisconnected}} in

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028370#comment-15028370 ] Saisai Shao commented on SPARK-12009: - If I understood correctly, this log should come from

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028375#comment-15028375 ] Saisai Shao commented on SPARK-12009: - Looks like you're running on yarn-cluster mode, but this log

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028325#comment-15028325 ] Saisai Shao commented on SPARK-12009: - A interesting thing is that AM is shutting down at time

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028325#comment-15028325 ] Saisai Shao edited comment on SPARK-12009 at 11/26/15 7:58 AM: --- A

[jira] [Commented] (SPARK-11909) Spark Standalone's master URL accepts URLs without port (assuming default 7077)

[ https://issues.apache.org/jira/browse/SPARK-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021669#comment-15021669 ] Saisai Shao commented on SPARK-11909: - The master will print the master url in web UI and log. Since

[jira] [Commented] (SPARK-11744) bin/pyspark --version doesn't return version and exit

[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005862#comment-15005862 ] Saisai Shao commented on SPARK-11744: - Yes, it is actually a small bug in Launcher, let me fix this.

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003996#comment-15003996 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves] I think I'm referring to your last post. From my

[jira] [Created] (SPARK-11718) Explicit killing executor dies silent without get response information

Saisai Shao created SPARK-11718: --- Summary: Explicit killing executor dies silent without get response information Key: SPARK-11718 URL: https://issues.apache.org/jira/browse/SPARK-11718 Project: Spark

[jira] [Commented] (SPARK-2960) Spark executables fail to start via symlinks

[ https://issues.apache.org/jira/browse/SPARK-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003598#comment-15003598 ] Saisai Shao commented on SPARK-2960: Hi [~yinxusen], from script's view point, it cannot detect which

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003601#comment-15003601 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves], I think I also hit this issue, it is caused by

[jira] [Created] (SPARK-11632) Filter out empty partition for KafkaRDD

Saisai Shao created SPARK-11632: --- Summary: Filter out empty partition for KafkaRDD Key: SPARK-11632 URL: https://issues.apache.org/jira/browse/SPARK-11632 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5934) DStreamGraph.clearMetadata attempts to unpersist the same RDD multiple times

[ https://issues.apache.org/jira/browse/SPARK-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14991353#comment-14991353 ] Saisai Shao commented on SPARK-5934: I think here in your code: {code} val output =

[jira] [Commented] (SPARK-11472) SparkContext creation error after sc.stop() when Spark is compiled for Hive

[ https://issues.apache.org/jira/browse/SPARK-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987314#comment-14987314 ] Saisai Shao commented on SPARK-11472: - Yes, agree with [~srowen], there might be a lot of problem

[jira] [Created] (SPARK-11457) Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint

Saisai Shao created SPARK-11457: --- Summary: Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint Key: SPARK-11457 URL: https://issues.apache.org/jira/browse/SPARK-11457

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977969#comment-14977969 ] Saisai Shao commented on SPARK-2089: Hi [~pwendell], [~mridulm80], [~sandyr] and [~lianhuiwang], I'm

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979586#comment-14979586 ] Saisai Shao commented on SPARK-2089: Hi [~sandyr] and [~tgraves], thank a lot for your comments. I'm

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968648#comment-14968648 ] Saisai Shao commented on SPARK-11181: - OK, get it. > Spark Yarn : Spark reducing total executors

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968649#comment-14968649 ] Saisai Shao commented on SPARK-11181: - I think you'd better backport to branch 1.3, not for tag 1.3.1

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from HistoryServer web portal

[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Summary: Support importing and exporting event logs from HistoryServer web portal (was: Support

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from History web portal

[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Description: Many users who met the problems when running Spark applications will send the logs

[jira] [Created] (SPARK-11272) Support importing and exporting event logs from History web portal

Saisai Shao created SPARK-11272: --- Summary: Support importing and exporting event logs from History web portal Key: SPARK-11272 URL: https://issues.apache.org/jira/browse/SPARK-11272 Project: Spark

[jira] [Updated] (SPARK-11272) Support importing and exporting event logs from History web portal

[ https://issues.apache.org/jira/browse/SPARK-11272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-11272: Attachment: screenshot-1.png > Support importing and exporting event logs from History web portal

[jira] [Commented] (SPARK-11181) Spark Yarn : Spark reducing total executors count even when Dynamic Allocation is disabled.

[ https://issues.apache.org/jira/browse/SPARK-11181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968453#comment-14968453 ] Saisai Shao commented on SPARK-11181: - Hi [~prakhar088], As discussed in the mailist

[jira] [Commented] (SPARK-11175) Concurrent execution of JobSet within a batch in Spark streaming

[ https://issues.apache.org/jira/browse/SPARK-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14962708#comment-14962708 ] Saisai Shao commented on SPARK-11175: - There's a configuration "spark.streaming.concurrentJobs" that

[jira] [Created] (SPARK-11082) Cores per executor is wrong when response vcore number is less than requested number

Saisai Shao created SPARK-11082: --- Summary: Cores per executor is wrong when response vcore number is less than requested number Key: SPARK-11082 URL: https://issues.apache.org/jira/browse/SPARK-11082

[jira] [Closed] (SPARK-11082) Cores per executor is wrong when response vcore number is less than requested number

[ https://issues.apache.org/jira/browse/SPARK-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-11082. --- Resolution: Invalid > Cores per executor is wrong when response vcore number is less than requested

[jira] [Created] (SPARK-11060) Fix some potential NPEs in DStream transformation

Saisai Shao created SPARK-11060: --- Summary: Fix some potential NPEs in DStream transformation Key: SPARK-11060 URL: https://issues.apache.org/jira/browse/SPARK-11060 Project: Spark Issue Type:

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952594#comment-14952594 ] Saisai Shao commented on SPARK-11045: - Hi Dib, To get better performance and fix the issues as you

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952560#comment-14952560 ] Saisai Shao commented on SPARK-11045: - {quote} Unfortunately, when customer fall back to

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934724#comment-14934724 ] Saisai Shao commented on SPARK-10858: - [~tgraves], I assume you're using yarn-cluster mode to submit

[jira] [Comment Edited] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936317#comment-14936317 ] Saisai Shao edited comment on SPARK-10858 at 9/30/15 5:49 AM: -- So basically

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936317#comment-14936317 ] Saisai Shao commented on SPARK-10858: - So basically I think the problem is do we need to treat this

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936313#comment-14936313 ] Saisai Shao commented on SPARK-10858: - Hi [~tgraves], I tested again with Mac and Linux (centos),

[jira] [Issue Comment Deleted] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10858: Comment: was deleted (was: The error I got in the failed case is the same as you mentioned above,

[jira] [Comment Edited] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936313#comment-14936313 ] Saisai Shao edited comment on SPARK-10858 at 9/30/15 3:50 AM: -- Hi

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936176#comment-14936176 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same as you

[jira] [Issue Comment Deleted] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10858: Comment: was deleted (was: The error I got in the failed case is the same as you mentioned above,

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936174#comment-14936174 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same as you

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936175#comment-14936175 ] Saisai Shao commented on SPARK-10858: - The error I got in the failed case is the same as you

[jira] [Commented] (SPARK-10790) Dynamic Allocation does not request any executors if first stage needs less than or equal to spark.dynamicAllocation.initialExecutors

[ https://issues.apache.org/jira/browse/SPARK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906696#comment-14906696 ] Saisai Shao commented on SPARK-10790: - Thanks [~srowen], let me check it. > Dynamic Allocation does

<    4   5   6   7   8   9   10   11   12   >