[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

2016-06-09 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322788#comment-15322788 ] Saisai Shao commented on SPARK-15828: - I think this issue is not related to dynamic allocation, if

[jira] [Created] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog

2016-06-01 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15698: --- Summary: Ability to remove old metadata for structure streaming MetadataLog Key: SPARK-15698 URL: https://issues.apache.org/jira/browse/SPARK-15698 Project: Spark

[jira] [Commented] (SPARK-15683) spark sql local FS spark.sql.warehouse.dir throws on YARN

2016-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15309139#comment-15309139 ] Saisai Shao commented on SPARK-15683: - Hi [~tgraves], please see this JIRA SPARK-15659, I submitted a

[jira] [Commented] (SPARK-15479) Spark job doesn't shut gracefully in yarn mode.

2016-05-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15307387#comment-15307387 ] Saisai Shao commented on SPARK-15479: - I guess using "yarn kill" to stop the application has a

[jira] [Updated] (SPARK-15659) Ensure FileSystem is gotten from path in InMemoryCatalog

2016-05-30 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15659: Component/s: SQL > Ensure FileSystem is gotten from path in InMemoryCatalog >

[jira] [Created] (SPARK-15659) Ensure FileSystem is gotten from path in InMemoryCatalog

2016-05-30 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15659: --- Summary: Ensure FileSystem is gotten from path in InMemoryCatalog Key: SPARK-15659 URL: https://issues.apache.org/jira/browse/SPARK-15659 Project: Spark Issue

[jira] [Commented] (SPARK-15564) App name is the main class name in Spark streaming jobs

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303442#comment-15303442 ] Saisai Shao commented on SPARK-15564: - According to the your description, I guess you're running

[jira] [Comment Edited] (SPARK-15564) App name is the main class name in Spark streaming jobs

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303442#comment-15303442 ] Saisai Shao edited comment on SPARK-15564 at 5/27/16 4:04 AM: -- According to

[jira] [Commented] (SPARK-15556) Add new Dataset API to get the codegen info for Dataset

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301738#comment-15301738 ] Saisai Shao commented on SPARK-15556: - I see, thanks a lot for your comment. sorry to bring in noise.

[jira] [Commented] (SPARK-15548) Shuffle file remains with dynamic allocation on YARN

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301734#comment-15301734 ] Saisai Shao commented on SPARK-15548: - I did a quick test on the shuffle file removing, actually

[jira] [Created] (SPARK-15556) Add new Dataset API to get the codegen info for Dataset

2016-05-26 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15556: --- Summary: Add new Dataset API to get the codegen info for Dataset Key: SPARK-15556 URL: https://issues.apache.org/jira/browse/SPARK-15556 Project: Spark Issue

[jira] [Comment Edited] (SPARK-15548) Shuffle file remains with dynamic allocation on YARN

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301670#comment-15301670 ] Saisai Shao edited comment on SPARK-15548 at 5/26/16 7:45 AM: -- There will be

[jira] [Commented] (SPARK-15548) Shuffle file remains with dynamic allocation on YARN

2016-05-26 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301670#comment-15301670 ] Saisai Shao commented on SPARK-15548: - >From my understanding this problem is not only a yarn

[jira] [Closed] (SPARK-12893) RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster mode

2016-05-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-12893. --- Resolution: Not A Problem > RM redirects to incorrect URL in Spark HistoryServer for yarn-cluster

[jira] [Commented] (SPARK-15541) SparkContext.stop throws error

2016-05-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301352#comment-15301352 ] Saisai Shao commented on SPARK-15541: - This method which returns {{KeySetView}} only exists in jdk

[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

2016-05-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15300071#comment-15300071 ] Saisai Shao commented on SPARK-13955: - You will have 3 options to upload jars to distributed cache:

[jira] [Commented] (SPARK-15368) Spark History Server does not pick up extraClasspath

2016-05-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297827#comment-15297827 ] Saisai Shao commented on SPARK-15368: - Maybe you could try {{SPARK_DIST_CLASSPATH}}, I'm not sure it

[jira] [Commented] (SPARK-15329) When start spark with yarn: spark.SparkContext: Error initializing SparkContext.

2016-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294653#comment-15294653 ] Saisai Shao commented on SPARK-15329: - {code} 2016-05-15 00:06:08,368 WARN

[jira] [Updated] (SPARK-15443) Properly explain the streaming queries

2016-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15443: Description: Currently when called `explain()` on streaming dataset, it will only get the parsed

[jira] [Updated] (SPARK-15443) Properly explain the streaming queries

2016-05-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15443: Description: Currently when called `explain()` on streaming dataset, it will only get the parsed

[jira] [Created] (SPARK-15443) Properly explain the streaming queries

2016-05-20 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15443: --- Summary: Properly explain the streaming queries Key: SPARK-15443 URL: https://issues.apache.org/jira/browse/SPARK-15443 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15375) Add ConsoleSink for structure streaming to display the dataframe on the fly

2016-05-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15375: Summary: Add ConsoleSink for structure streaming to display the dataframe on the fly (was: Add

[jira] [Created] (SPARK-15375) Add ConsoleSink for structure sink to display the dataframe on the fly

2016-05-17 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15375: --- Summary: Add ConsoleSink for structure sink to display the dataframe on the fly Key: SPARK-15375 URL: https://issues.apache.org/jira/browse/SPARK-15375 Project: Spark

[jira] [Commented] (SPARK-15368) Spark History Server does not pick up extraClasspath

2016-05-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288049#comment-15288049 ] Saisai Shao commented on SPARK-15368: - Maybe you could try {{SPARK_CLASSPATH}}, though deprecated

[jira] [Commented] (SPARK-15371) YARNShuffleService doesn't get current local-dirs from NodeManager

2016-05-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288013#comment-15288013 ] Saisai Shao commented on SPARK-15371: - I think what you mentioned above is relate to this JIRA

[jira] [Commented] (SPARK-15329) When start spark with yarn: spark.SparkContext: Error initializing SparkContext.

2016-05-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15284292#comment-15284292 ] Saisai Shao commented on SPARK-15329: - Check the container log is an effective way, using {{yarn

[jira] [Updated] (SPARK-15283) YARN Application should get FAILED state if driver is failed in yarn-client mode

2016-05-11 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15283: Component/s: YARN > YARN Application should get FAILED state if driver is failed in yarn-client >

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273412#comment-15273412 ] Saisai Shao commented on SPARK-14963: - OK, I will do it. > YarnShuffleService should use YARN

[jira] [Commented] (SPARK-15146) Allow specifying kafka parameters through configurations

2016-05-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272058#comment-15272058 ] Saisai Shao commented on SPARK-15146: - Oh, I see, that is valid solution for it. Seems no need to

[jira] [Closed] (SPARK-15146) Allow specifying kafka parameters through configurations

2016-05-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-15146. --- Resolution: Not A Problem > Allow specifying kafka parameters through configurations >

[jira] [Commented] (SPARK-15146) Allow specifying kafka parameters through configurations

2016-05-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15271962#comment-15271962 ] Saisai Shao commented on SPARK-15146: - [~c...@koeninger.org], what is your opinion on the JIRA?

[jira] [Created] (SPARK-15146) Allow specifying kafka parameters through configurations

2016-05-05 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-15146: --- Summary: Allow specifying kafka parameters through configurations Key: SPARK-15146 URL: https://issues.apache.org/jira/browse/SPARK-15146 Project: Spark Issue

[jira] [Commented] (SPARK-15039) Kinesis reciever does not work in Yarn

2016-05-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267857#comment-15267857 ] Saisai Shao commented on SPARK-15039: - [~ltsai], do you have a exception stack about the failure you

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-04-29 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263706#comment-15263706 ] Saisai Shao commented on SPARK-14963: - Hi [~tgraves], would you mind me taking a crack at this if

[jira] [Created] (SPARK-14836) Zip local jars before uploading to distributed cache

2016-04-21 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14836: --- Summary: Zip local jars before uploading to distributed cache Key: SPARK-14836 URL: https://issues.apache.org/jira/browse/SPARK-14836 Project: Spark Issue

[jira] [Created] (SPARK-14788) Initial number of executors should honor min number if streaming dynamic allocation is enabled

2016-04-20 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14788: --- Summary: Initial number of executors should honor min number if streaming dynamic allocation is enabled Key: SPARK-14788 URL: https://issues.apache.org/jira/browse/SPARK-14788

[jira] [Commented] (SPARK-14725) Remove HttpServer

2016-04-20 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249557#comment-15249557 ] Saisai Shao commented on SPARK-14725: - I think now it forces to use RPC and there's no other option

[jira] [Commented] (SPARK-14723) A new way to support dynamic allocation in Spark Streaming

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247384#comment-15247384 ] Saisai Shao commented on SPARK-14723: - It would be better not to set fix version and target version.

[jira] [Comment Edited] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247358#comment-15247358 ] Saisai Shao edited comment on SPARK-14725 at 4/19/16 8:13 AM: -- I just search

[jira] [Commented] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247358#comment-15247358 ] Saisai Shao commented on SPARK-14725: - I just search the repl code, from my understanding seems

[jira] [Updated] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14725: Priority: Minor (was: Major) > Remove HttpServer > - > > Key:

[jira] [Updated] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14725: Issue Type: Bug (was: Sub-task) Parent: (was: SPARK-11806) > Remove HttpServer >

[jira] [Commented] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247318#comment-15247318 ] Saisai Shao commented on SPARK-14725: - OK, let me check first :). > Remove HttpServer >

[jira] [Commented] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247316#comment-15247316 ] Saisai Shao commented on SPARK-14725: - I think now it changes to use RPC instead of Http, please see

[jira] [Commented] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247293#comment-15247293 ] Saisai Shao commented on SPARK-14725: - What's your opinion [~rxin] [~andrewor14]? > Remove

[jira] [Created] (SPARK-14725) Remove HttpServer

2016-04-19 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14725: --- Summary: Remove HttpServer Key: SPARK-14725 URL: https://issues.apache.org/jira/browse/SPARK-14725 Project: Spark Issue Type: Sub-task Components:

[jira] [Commented] (SPARK-14453) Consider removing SPARK_JAVA_OPTS env variable

2016-04-17 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245048#comment-15245048 ] Saisai Shao commented on SPARK-14453: - Yes, this should be a part of SPARK-12344. Since SPARK-12344

[jira] [Created] (SPARK-14455) ReceiverTracker#allocatedExecutors throw NPE for receiver-less streaming application

2016-04-07 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14455: --- Summary: ReceiverTracker#allocatedExecutors throw NPE for receiver-less streaming application Key: SPARK-14455 URL: https://issues.apache.org/jira/browse/SPARK-14455

[jira] [Updated] (SPARK-14453) Consider removing SPARK_JAVA_OPTS env variable

2016-04-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14453: Priority: Minor (was: Major) > Consider removing SPARK_JAVA_OPTS env variable >

[jira] [Created] (SPARK-14453) Consider removing SPARK_JAVA_OPTS env variable

2016-04-07 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14453: --- Summary: Consider removing SPARK_JAVA_OPTS env variable Key: SPARK-14453 URL: https://issues.apache.org/jira/browse/SPARK-14453 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-14423) Handle jar conflict issue when uploading to distributed cache

2016-04-05 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15227664#comment-15227664 ] Saisai Shao commented on SPARK-14423: - I will fix it soon. > Handle jar conflict issue when

[jira] [Created] (SPARK-14423) Handle jar conflict issue when uploading to distributed cache

2016-04-05 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14423: --- Summary: Handle jar conflict issue when uploading to distributed cache Key: SPARK-14423 URL: https://issues.apache.org/jira/browse/SPARK-14423 Project: Spark

[jira] [Updated] (SPARK-14062) Put metrics.properties to distributed cache for Spark running on Yarn. Also Fix log4j propagation issue

2016-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14062: Summary: Put metrics.properties to distributed cache for Spark running on Yarn. Also Fix log4j

[jira] [Updated] (SPARK-14062) Put metrics.properties to distributed cache for Spark running on Yarn. Also Fix log3j propagation issue

2016-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14062: Summary: Put metrics.properties to distributed cache for Spark running on Yarn. Also Fix log3j

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

2016-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211640#comment-15211640 ] Saisai Shao commented on SPARK-14151: - Yes, sure I will add the doc. > Propose to refactor and

[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

2016-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211634#comment-15211634 ] Saisai Shao commented on SPARK-14151: - I'm currently working on the second choice, to offer external

[jira] [Updated] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface

2016-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14151: Summary: Propose to refactor and expose Metrics Sink and Source interface (was: Propose to expose

[jira] [Commented] (SPARK-14151) Propose to expose Metrics Sink and Source interface

2016-03-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211562#comment-15211562 ] Saisai Shao commented on SPARK-14151: - Yes, that's what I'm thinking about, from the code level, I

[jira] [Created] (SPARK-14151) Propose to expose Metrics Sink and Source interface

2016-03-25 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14151: --- Summary: Propose to expose Metrics Sink and Source interface Key: SPARK-14151 URL: https://issues.apache.org/jira/browse/SPARK-14151 Project: Spark Issue

[jira] [Created] (SPARK-14062) Put metrics.properties to distributed cache for Spark running on Yarn

2016-03-22 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-14062: --- Summary: Put metrics.properties to distributed cache for Spark running on Yarn Key: SPARK-14062 URL: https://issues.apache.org/jira/browse/SPARK-14062 Project: Spark

[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

2016-03-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203774#comment-15203774 ] Saisai Shao commented on SPARK-13955: - [~zjffdu], I think way to fix this is to specify assembly jar

[jira] [Commented] (SPARK-13915) Allow bin/spark-submit to be called via symbolic link

2016-03-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196590#comment-15196590 ] Saisai Shao commented on SPARK-13915: - I suppose it should be worked if you set {{SPARK_HOME}} in

[jira] [Commented] (SPARK-13914) Add functionality to back up spark event logs

2016-03-15 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196580#comment-15196580 ] Saisai Shao commented on SPARK-13914: - Agree with [~srowen], this feature is quite user specific,

[jira] [Created] (SPARK-13885) Spark On Yarn attempt id representation regression

2016-03-14 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13885: --- Summary: Spark On Yarn attempt id representation regression Key: SPARK-13885 URL: https://issues.apache.org/jira/browse/SPARK-13885 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13877) Consider removing Kafka modules from Spark / Spark Streaming

2016-03-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194653#comment-15194653 ] Saisai Shao commented on SPARK-13877: - I agreed to move this out, so it could easily support

[jira] [Commented] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true

2016-03-07 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15184177#comment-15184177 ] Saisai Shao commented on SPARK-13723: - Yes, I agree with what [~tgraves] described above. Normally

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

2016-03-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Attachment: Screen Shot 2016-02-29 at 3.57.32 PM.png > The url link in historypage is not correct

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

2016-03-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Description: Current URL for each application to access history UI is like:

[jira] [Updated] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

2016-03-04 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13675: Description: Current URL for each application to access history UI is like:

[jira] [Created] (SPARK-13675) The url link in historypage is not correct for application running in yarn cluster mode

2016-03-04 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13675: --- Summary: The url link in historypage is not correct for application running in yarn cluster mode Key: SPARK-13675 URL: https://issues.apache.org/jira/browse/SPARK-13675

[jira] [Commented] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179485#comment-15179485 ] Saisai Shao commented on SPARK-13669: - Hi [~imranr], I think you fixed this issue (SPARK-9439) and

[jira] [Created] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

2016-03-03 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13669: --- Summary: Job will always fail in the external shuffle service unavailable situation Key: SPARK-13669 URL: https://issues.apache.org/jira/browse/SPARK-13669 Project:

[jira] [Updated] (SPARK-13642) Properly handle signal kill of ApplicationMaster

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Summary: Properly handle signal kill of ApplicationMaster (was: Different finishing state between

[jira] [Comment Edited] (SPARK-13642) Different finishing state between Spark and YARN

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179102#comment-15179102 ] Saisai Shao edited comment on SPARK-13642 at 3/4/16 1:45 AM: - Hi [~tgraves],

[jira] [Updated] (SPARK-13642) Different finishing state between Spark and YARN

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Summary: Different finishing state between Spark and YARN (was: Inconsistent finishing state

[jira] [Commented] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15179102#comment-15179102 ] Saisai Shao commented on SPARK-13642: - Hi [~tgraves], thanks a lot for your comments. I think here

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Commented] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15177347#comment-15177347 ] Saisai Shao commented on SPARK-13642: - [~tgraves] [~vanzin], would you please comment on this, why

[jira] [Updated] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13642: Description: Currently when running Spark on Yarn with yarn cluster mode, the default application

[jira] [Created] (SPARK-13642) Inconsistent finishing state between driver and AM

2016-03-02 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13642: --- Summary: Inconsistent finishing state between driver and AM Key: SPARK-13642 URL: https://issues.apache.org/jira/browse/SPARK-13642 Project: Spark Issue

[jira] [Commented] (SPARK-11157) Allow Spark to be built without assemblies

2016-03-02 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175276#comment-15175276 ] Saisai Shao commented on SPARK-11157: - Hi [~vanzin], for Spark Streaming PySpark related parts, how

[jira] [Closed] (SPARK-13608) py4j.Py4JException: Method createDirectStream([class org.apache.spark.streaming.api.java.JavaStreamingContext, class java.util.HashMap, class java.util.HashSet, class jav

2016-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-13608. --- Resolution: Not A Problem > py4j.Py4JException: Method createDirectStream([class >

[jira] [Commented] (SPARK-13608) py4j.Py4JException: Method createDirectStream([class org.apache.spark.streaming.api.java.JavaStreamingContext, class java.util.HashMap, class java.util.HashSet, class

2016-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15175144#comment-15175144 ] Saisai Shao commented on SPARK-13608: - Hi [~avatarzhang] , would you please elaborate your problem,

[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation disabled situation

2016-02-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation disabled situation (was: Fix AM failure

[jira] [Updated] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled situation

2016-02-22 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13447: Summary: Fix AM failure situation for dynamic allocation diabled situation (was: Fix AM failure

[jira] [Created] (SPARK-13447) Fix AM failure situation for dynamic allocation diabled sitation

2016-02-22 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13447: --- Summary: Fix AM failure situation for dynamic allocation diabled sitation Key: SPARK-13447 URL: https://issues.apache.org/jira/browse/SPARK-13447 Project: Spark

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

2016-02-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Description: Currently there's not so many users who will use SIMR to run Spark, especially in a

[jira] [Updated] (SPARK-13426) Remove the support of SIMR cluster manager

2016-02-21 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-13426: Issue Type: Sub-task (was: Bug) Parent: SPARK-11806 > Remove the support of SIMR cluster

[jira] [Created] (SPARK-13426) Remove the support of SIMR cluster manager

2016-02-21 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13426: --- Summary: Remove the support of SIMR cluster manager Key: SPARK-13426 URL: https://issues.apache.org/jira/browse/SPARK-13426 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12343) Remove YARN Client / ClientArguments

2016-02-19 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153910#comment-15153910 ] Saisai Shao commented on SPARK-12343: - Hi guys, Do we still want to support users who directly

[jira] [Commented] (SPARK-13275) With dynamic allocation, executors appear to be added before job starts

2016-02-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149939#comment-15149939 ] Saisai Shao commented on SPARK-13275: - would you please clarify the specific problem you mentioned,

[jira] [Commented] (SPARK-13220) Deprecate "yarn-client" and "yarn-cluster"

2016-02-14 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147022#comment-15147022 ] Saisai Shao commented on SPARK-13220: - [~andrewor14] mind me taking a crack on this? > Deprecate

[jira] [Created] (SPARK-13109) SBT publishLocal failed to publish to local ivy repo

2016-02-01 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-13109: --- Summary: SBT publishLocal failed to publish to local ivy repo Key: SPARK-13109 URL: https://issues.apache.org/jira/browse/SPARK-13109 Project: Spark Issue

[jira] [Commented] (SPARK-13104) Spark Metrics currently does not return executors hostname

2016-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125655#comment-15125655 ] Saisai Shao commented on SPARK-13104: - I think it should be Codahale metrics. > Spark Metrics

[jira] [Commented] (SPARK-13106) KafkaUtils.createDirectStream method with messageHandler and topics

2016-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125598#comment-15125598 ] Saisai Shao commented on SPARK-13106: - IIUC, creating direct stream also supports passing

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

2016-01-28 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123043#comment-15123043 ] Saisai Shao commented on SPARK-3374: +1 to address this in the upcoming 2.0 release. 1. Currently in

[jira] [Updated] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

2016-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-12977: Attachment: screenshot-1.png > Factoring out StreamingListener and UI to support history UI >

[jira] [Commented] (SPARK-12977) Factoring out StreamingListener and UI to support history UI

2016-01-25 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15116844#comment-15116844 ] Saisai Shao commented on SPARK-12977: - Attach the current working progress, still some problems

[jira] [Commented] (SPARK-12973) Support to set priority when submit spark application to YARN

2016-01-24 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114807#comment-15114807 ] Saisai Shao commented on SPARK-12973: - I think there's a similar JIRA SPARK-10879 about this issue,

<    3   4   5   6   7   8   9   10   11   12   >