[jira] [Commented] (SPARK-12384) Allow -Xms to be set differently then -Xmx

[ https://issues.apache.org/jira/browse/SPARK-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061639#comment-15061639 ] Saisai Shao commented on SPARK-12384: - IIUC, there's also another limitation in conta

[jira] [Commented] (SPARK-10500) sparkr.zip cannot be created if $SPARK_HOME/R/lib is unwritable

[ https://issues.apache.org/jira/browse/SPARK-10500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061730#comment-15061730 ] Saisai Shao commented on SPARK-10500: - [~sunrui] It would be better to back port to 1

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063359#comment-15063359 ] Saisai Shao commented on SPARK-12400: - So from my understanding this will only be exi

[jira] [Commented] (SPARK-12400) Avoid writing a shuffle file if a partition has no output (empty)

[ https://issues.apache.org/jira/browse/SPARK-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063372#comment-15063372 ] Saisai Shao commented on SPARK-12400: - [~rxin], would you mind me taking a crack at t

[jira] [Created] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

Saisai Shao created SPARK-12447: --- Summary: Only update AM's internal state when executor is successfully launched by NM Key: SPARK-12447 URL: https://issues.apache.org/jira/browse/SPARK-12447 Project: S

[jira] [Commented] (SPARK-12514) Spark MetricsSystem can fill disks/cause OOMs when using GangliaSink

[ https://issues.apache.org/jira/browse/SPARK-12514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070717#comment-15070717 ] Saisai Shao commented on SPARK-12514: - Since we need to differentiate the metrics bet

[jira] [Created] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

Saisai Shao created SPARK-12516: --- Summary: Properly handle NM failure situation for Spark on Yarn Key: SPARK-12516 URL: https://issues.apache.org/jira/browse/SPARK-12516 Project: Spark Issue Ty

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Updated] (SPARK-12447) Only update AM's internal state when executor is successfully launched by NM

[ https://issues.apache.org/jira/browse/SPARK-12447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12447: Description: Currently {{YarnAllocator}} will update its managed states like {{numExecutorsRunning

[jira] [Updated] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12516: Description: Failure of NodeManager will make all the executors belong to that NM exit silently.

[jira] [Commented] (SPARK-11782) Master Web UI should link to correct Application UI in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073430#comment-15073430 ] Saisai Shao commented on SPARK-11782: - I just verified with latest master branch, see

[jira] [Created] (SPARK-12552) Recovered driver's resource is not counted in the Master

Saisai Shao created SPARK-12552: --- Summary: Recovered driver's resource is not counted in the Master Key: SPARK-12552 URL: https://issues.apache.org/jira/browse/SPARK-12552 Project: Spark Issue

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073954#comment-15073954 ] Saisai Shao commented on SPARK-12554: - First from my understanding tasks can be sched

[jira] [Commented] (SPARK-12554) Standalone app scheduler will hang when app.coreToAssign < minCoresPerExecutor

[ https://issues.apache.org/jira/browse/SPARK-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15074588#comment-15074588 ] Saisai Shao commented on SPARK-12554: - For case 2, I think it is really a misconfigur

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075693#comment-15075693 ] Saisai Shao commented on SPARK-12516: - Hi [~vanzin], what is your suggestion of this

[jira] [Commented] (SPARK-12516) Properly handle NM failure situation for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-12516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082085#comment-15082085 ] Saisai Shao commented on SPARK-12516: - Thanks a lot [~vanzin] for your reply. Looks l

[jira] [Commented] (SPARK-12650) No means to specify Xmx settings for SparkSubmit in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15084497#comment-15084497 ] Saisai Shao commented on SPARK-12650: - [~vines], what is your meaning of "SparkSubmit

[jira] [Created] (SPARK-12673) Prepending base URI of job description is missing

Saisai Shao created SPARK-12673: --- Summary: Prepending base URI of job description is missing Key: SPARK-12673 URL: https://issues.apache.org/jira/browse/SPARK-12673 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Attachment: screenshot-1.png > Prepending base URI of job description is missing >

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes acc

[jira] [Updated] (SPARK-12673) Prepending base URI of job description is missing

[ https://issues.apache.org/jira/browse/SPARK-12673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12673: Description: The base URI of job description is not prepending in the current code, which makes ac

[jira] [Commented] (SPARK-2044) Pluggable interface for shuffles

[ https://issues.apache.org/jira/browse/SPARK-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019511#comment-14019511 ] Saisai Shao commented on SPARK-2044: Hi Matei, it's great to see you guys have plan on

[jira] [Commented] (SPARK-2044) Pluggable interface for shuffles

[ https://issues.apache.org/jira/browse/SPARK-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019617#comment-14019617 ] Saisai Shao commented on SPARK-2044: Hi Matei, thanks for your reply. I will carefully

[jira] [Created] (SPARK-2122) Move aggregation into shuffle implementation

Saisai Shao created SPARK-2122: -- Summary: Move aggregation into shuffle implementation Key: SPARK-2122 URL: https://issues.apache.org/jira/browse/SPARK-2122 Project: Spark Issue Type: Improvemen

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028833#comment-14028833 ] Saisai Shao commented on SPARK-2125: Hi Matei, for moving sort into hash shuffle imple

[jira] [Commented] (SPARK-2124) Move aggregation into ShuffleManager implementations

[ https://issues.apache.org/jira/browse/SPARK-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028922#comment-14028922 ] Saisai Shao commented on SPARK-2124: PR submitted https://github.com/apache/spark/pull

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041809#comment-14041809 ] Saisai Shao commented on SPARK-2125: Hi Matei, I have some basic implementations about

[jira] [Comment Edited] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041809#comment-14041809 ] Saisai Shao edited comment on SPARK-2125 at 6/24/14 7:26 AM: -

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044285#comment-14044285 ] Saisai Shao commented on SPARK-2125: PR submitted: https://github.com/apache/spark/pul

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045413#comment-14045413 ] Saisai Shao commented on SPARK-2104: Hi Reynold, is that annoying message you mentione

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045430#comment-14045430 ] Saisai Shao commented on SPARK-2104: Ok, got it. I will try to fix this issue :) > Ra

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045519#comment-14045519 ] Saisai Shao commented on SPARK-2104: Hi Reynold, thanks a lot for your code. At first

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045524#comment-14045524 ] Saisai Shao commented on SPARK-2104: OK, got it. Thanks a lot > RangePartitioner shou

[jira] [Created] (SPARK-2402) DiskBlockObjectWriter should update the initial position when reusing this object

Saisai Shao created SPARK-2402: -- Summary: DiskBlockObjectWriter should update the initial position when reusing this object Key: SPARK-2402 URL: https://issues.apache.org/jira/browse/SPARK-2402 Project:

[jira] [Commented] (SPARK-2402) DiskBlockObjectWriter should update the initial position when reusing this object

[ https://issues.apache.org/jira/browse/SPARK-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054686#comment-14054686 ] Saisai Shao commented on SPARK-2402: PR submitted, https://github.com/apache/spark/pul

[jira] [Created] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

Saisai Shao created SPARK-2492: -- Summary: KafkaReceiver minor changes to align with Kafka 0.8 Key: SPARK-2492 URL: https://issues.apache.org/jira/browse/SPARK-2492 Project: Spark Issue Type: Im

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062053#comment-14062053 ] Saisai Shao commented on SPARK-2492: PR submitted: https://github.com/apache/spark/pul

[jira] [Commented] (SPARK-2045) Sort-based shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063006#comment-14063006 ] Saisai Shao commented on SPARK-2045: Hi Matei, great to see your design doc, a simple

[jira] [Commented] (SPARK-2045) Sort-based shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063037#comment-14063037 ] Saisai Shao commented on SPARK-2045: IMHO I think in this situation we should manage t

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063063#comment-14063063 ] Saisai Shao commented on SPARK-2492: Hi TD, The parameter "auto.offset.reset" is trig

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064636#comment-14064636 ] Saisai Shao commented on SPARK-2492: Hi TD, I revisit the Kafka's ConsoleConsumer ca

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064721#comment-14064721 ] Saisai Shao commented on SPARK-2492: Hi TD, Also I did some experiments on the previ

[jira] [Commented] (SPARK-2383) With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper

[ https://issues.apache.org/jira/browse/SPARK-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069692#comment-14069692 ] Saisai Shao commented on SPARK-2383: Hi Tobias, I've also noticed this problem, seem

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069727#comment-14069727 ] Saisai Shao commented on SPARK-2492: Hi Tobias, I agree with you. Though I do not kn

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072749#comment-14072749 ] Saisai Shao commented on SPARK-2492: Hi Tobias, I've updated the code, mind taking a

[jira] [Comment Edited] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14072749#comment-14072749 ] Saisai Shao edited comment on SPARK-2492 at 7/24/14 3:27 AM: -

[jira] [Commented] (SPARK-2780) Create a StreamingContext.setLocalProperty for setting local property of jobs launched by streaming

[ https://issues.apache.org/jira/browse/SPARK-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14081947#comment-14081947 ] Saisai Shao commented on SPARK-2780: Hi TD, I think the fair scheduler setting can be

[jira] [Created] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

Saisai Shao created SPARK-2926: -- Summary: Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle Key: SPARK-2926 URL: https://issues.apache.org/jira/browse/SPARK-2926 Project: Spark

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: SortBasedShuffleRead.pdf A rough design doc is uploaded. Any comments would be greatly ap

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Description: Currently Spark has already integrated sort-based shuffle write, which greatly improve

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091684#comment-14091684 ] Saisai Shao commented on SPARK-2926: Hi Sandy, Thanks a lot for your comments, basic

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091753#comment-14091753 ] Saisai Shao commented on SPARK-2926: Hi Matei, thanks a lot for your comments. The or

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091753#comment-14091753 ] Saisai Shao edited comment on SPARK-2926 at 8/9/14 2:09 PM: Hi

[jira] [Created] (SPARK-2967) Several SQL unit test failed when sort-based shuffle is enabled

Saisai Shao created SPARK-2967: -- Summary: Several SQL unit test failed when sort-based shuffle is enabled Key: SPARK-2967 URL: https://issues.apache.org/jira/browse/SPARK-2967 Project: Spark Is

[jira] [Commented] (SPARK-2978) Provide an MR-style shuffle transformation

[ https://issues.apache.org/jira/browse/SPARK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093866#comment-14093866 ] Saisai Shao commented on SPARK-2978: Hi Sandy, A simple question: do you mean to add

[jira] [Commented] (SPARK-2967) Several SQL unit test failed when sort-based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093874#comment-14093874 ] Saisai Shao commented on SPARK-2967: Hi Matei and Michael, thanks a lot for looking in

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: Spark Shuffle Test Report.pdf > Add MR-style (merge-sort) SortShuffleReader for sort-base

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096608#comment-14096608 ] Saisai Shao commented on SPARK-2926: Hi Matei, I just uploaded a Spark shuffle perfo

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096608#comment-14096608 ] Saisai Shao edited comment on SPARK-2926 at 8/14/14 7:12 AM: -

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14096688#comment-14096688 ] Saisai Shao commented on SPARK-2926: I think this prototype can easily offer the funct

[jira] [Created] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

Saisai Shao created SPARK-3032: -- Summary: Potential bug when running sort-based shuffle with sorting using TimSort Key: SPARK-3032 URL: https://issues.apache.org/jira/browse/SPARK-3032 Project: Spark

[jira] [Created] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

Saisai Shao created SPARK-3146: -- Summary: Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM Key: SPARK-3146 URL: https://issues.apache.org/jira/b

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103486#comment-14103486 ] Saisai Shao commented on SPARK-3146: This issue can actually solve the problem mention

[jira] [Updated] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3146: --- Description: Currently Spark Streaming Kafka API stores the key and value of each message into BM fo

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14105086#comment-14105086 ] Saisai Shao commented on SPARK-3129: Hi Hari, I have some high level questions about t

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105671#comment-15105671 ] Saisai Shao commented on SPARK-12864: - What's Spark version are you using? I remember

[jira] [Comment Edited] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105671#comment-15105671 ] Saisai Shao edited comment on SPARK-12864 at 1/18/16 7:03 PM: -

[jira] [Created] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

Saisai Shao created SPARK-12893: --- Summary: RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode Key: SPARK-12893 URL: https://issues.apache.org/jira/browse/SPARK-12893 Project: Spa

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Attachment: Screen Shot 2016-01-18 at 3.47.24 PM.png > RM redirects to incorrect URL in Spark Histo

[jira] [Updated] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12893: Description: This will cause application not found error, the screenshot shows below: !https://iss

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation doc still refers to 1.2

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106005#comment-15106005 ] Saisai Shao commented on SPARK-12883: - I think this doc is still valid, current way o

[jira] [Commented] (SPARK-12864) initialize executorIdCounter after ApplicationMaster killed for max number of executor failures reached

[ https://issues.apache.org/jira/browse/SPARK-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106214#comment-15106214 ] Saisai Shao commented on SPARK-12864: - So the problem should be that: {{BlockManager}

[jira] [Commented] (SPARK-12883) 1.6 Dynamic allocation document for removing executors with cached data differs in different sections

[ https://issues.apache.org/jira/browse/SPARK-12883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15107085#comment-15107085 ] Saisai Shao commented on SPARK-12883: - I get your point now. But I think these two de

[jira] [Commented] (SPARK-11045) Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to Apache Spark Project

[ https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111243#comment-15111243 ] Saisai Shao commented on SPARK-11045: - Hi [~dibbhatt], I'm afraid I could not agree w

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111656#comment-15111656 ] Saisai Shao commented on SPARK-12140: - Hi guys, I though a bit on this feature, besid

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -

[jira] [Comment Edited] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111656#comment-15111656 ] Saisai Shao edited comment on SPARK-12140 at 1/22/16 1:29 AM: -

[jira] [Commented] (SPARK-12973) Support to set priority when submit spark application to YARN

[ https://issues.apache.org/jira/browse/SPARK-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114807#comment-15114807 ] Saisai Shao commented on SPARK-12973: - I think there's a similar JIRA SPARK-10879 abo

[jira] [Created] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

Saisai Shao created SPARK-12977: --- Summary: Factoring out StreamingListener and UI to support history UI Key: SPARK-12977 URL: https://issues.apache.org/jira/browse/SPARK-12977 Project: Spark I

[jira] [Updated] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12977: Attachment: screenshot-1.png > Factoring out StreamingListener and UI to support history UI > -

[jira] [Commented] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116844#comment-15116844 ] Saisai Shao commented on SPARK-12977: - Attach the current working progress, still som

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123043#comment-15123043 ] Saisai Shao commented on SPARK-3374: +1 to address this in the upcoming 2.0 release.

[jira] [Commented] (SPARK-13106) KafkaUtils.createDirectStream method with messageHandler and topics

[ https://issues.apache.org/jira/browse/SPARK-13106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125598#comment-15125598 ] Saisai Shao commented on SPARK-13106: - IIUC, creating direct stream also supports pas

[jira] [Commented] (SPARK-13104) Spark Metrics currently does not return executors hostname

[ https://issues.apache.org/jira/browse/SPARK-13104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125655#comment-15125655 ] Saisai Shao commented on SPARK-13104: - I think it should be Codahale metrics. > Spar

[jira] [Commented] (SPARK-13106) KafkaUtils.createDirectStream method with messageHandler and topics

[ https://issues.apache.org/jira/browse/SPARK-13106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125911#comment-15125911 ] Saisai Shao commented on SPARK-13106: - I think this {{fromOffsets}} refers to the par

[jira] [Created] (SPARK-13109) SBT publishLocal failed to publish to local ivy repo

Saisai Shao created SPARK-13109: --- Summary: SBT publishLocal failed to publish to local ivy repo Key: SPARK-13109 URL: https://issues.apache.org/jira/browse/SPARK-13109 Project: Spark Issue Type

[jira] [Commented] (SPARK-13220) Deprecate "yarn-client" and "yarn-cluster"

[ https://issues.apache.org/jira/browse/SPARK-13220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15147022#comment-15147022 ] Saisai Shao commented on SPARK-13220: - [~andrewor14] mind me taking a crack on this?

[jira] [Commented] (SPARK-13275) With dynamic allocation, executors appear to be added before job starts

[ https://issues.apache.org/jira/browse/SPARK-13275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149939#comment-15149939 ] Saisai Shao commented on SPARK-13275: - would you please clarify the specific problem

[jira] [Created] (SPARK-9969) Remove old Yarn MR classpath api support for Spark Yarn client

Saisai Shao created SPARK-9969: -- Summary: Remove old Yarn MR classpath api support for Spark Yarn client Key: SPARK-9969 URL: https://issues.apache.org/jira/browse/SPARK-9969 Project: Spark Iss

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704832#comment-14704832 ] Saisai Shao commented on SPARK-10122: - Thanks [~am...@hemtech.co.za] and [~srowen], a

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706186#comment-14706186 ] Saisai Shao commented on SPARK-10122: - Hi [~aramesh], thanks a lot for pointing this

[jira] [Commented] (SPARK-10122) AttributeError: 'RDD' object has no attribute 'offsetRanges'

[ https://issues.apache.org/jira/browse/SPARK-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706306#comment-14706306 ] Saisai Shao commented on SPARK-10122: - Thanks :). > AttributeError: 'RDD' object has

[jira] [Created] (SPARK-10288) Add a rest client for Spark on Yarn

Saisai Shao created SPARK-10288: --- Summary: Add a rest client for Spark on Yarn Key: SPARK-10288 URL: https://issues.apache.org/jira/browse/SPARK-10288 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726814#comment-14726814 ] Saisai Shao commented on SPARK-10288: - Hi [~vanzin], thanks a lot for your comments.

[jira] [Updated] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-10288: Description: This is a proposal to add rest client for Spark on Yarn. Rest API offers a convenient

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726886#comment-14726886 ] Saisai Shao commented on SPARK-10399: - Hi [~paulweiss], a simple question about C++ l

[jira] [Commented] (SPARK-7173) Support YARN node label expressions for the application master

[ https://issues.apache.org/jira/browse/SPARK-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730358#comment-14730358 ] Saisai Shao commented on SPARK-7173: Hi guys, is it still meaningful to support label

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733223#comment-14733223 ] Saisai Shao commented on SPARK-10288: - Thanks a lot Marcelo for your comments, indeed

[jira] [Commented] (SPARK-10288) Add a rest client for Spark on Yarn

[ https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14734161#comment-14734161 ] Saisai Shao commented on SPARK-10288: - Hi Marcelo, Yes I agree with you that rest cl

<    2   3   4   5   6   7   8   9   10   11   >