[jira] [Commented] (SPARK-17624) Flaky test? StateStoreSuite maintenance

[ https://issues.apache.org/jira/browse/SPARK-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512237#comment-15512237 ] Saisai Shao commented on SPARK-17624: - I cannot reproduce locally on my > Flaky test?

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Issue Type: Sub-task (was: Improvement) Parent: SPARK-17267 > Support purging aged file

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17604: Description: Currently with SPARK-15698, FileStreamSource metadata log will be compacted

[jira] [Created] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

Saisai Shao created SPARK-17604: --- Summary: Support purging aged file entry for FileStreamSource metadata log Key: SPARK-17604 URL: https://issues.apache.org/jira/browse/SPARK-17604 Project: Spark

[jira] [Commented] (SPARK-15698) Ability to remove old metadata for structure streaming MetadataLog

[ https://issues.apache.org/jira/browse/SPARK-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505799#comment-15505799 ] Saisai Shao commented on SPARK-15698: - I think [~rxin] set this target version before. I'm OK to

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500522#comment-15500522 ] Saisai Shao commented on SPARK-17566: - I've already submitted a PR under SPARK-17512, since this JIRA

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500511#comment-15500511 ] Saisai Shao commented on SPARK-17566: - Shouldn't it be {{!isYarnCluster}}? Since we need to avoid

[jira] [Updated] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17512: Component/s: YARN > Specifying remote files for Python based Spark jobs in Yarn cluster mode not

[jira] [Commented] (SPARK-17512) Specifying remote files for Python based Spark jobs in Yarn cluster mode not working

[ https://issues.apache.org/jira/browse/SPARK-17512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500178#comment-15500178 ] Saisai Shao commented on SPARK-17512: - This is due to some behavior changes during submitting spark

[jira] [Closed] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-17566. --- Resolution: Duplicate > "--master yarn --deploy-mode cluster" gives "Launching Python applications

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500145#comment-15500145 ] Saisai Shao commented on SPARK-17566: - Sorry I misunderstood your point, looks like it should be

[jira] [Commented] (SPARK-17566) "--master yarn --deploy-mode cluster" gives "Launching Python applications through spark-submit is currently only supported for local files"

[ https://issues.apache.org/jira/browse/SPARK-17566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500109#comment-15500109 ] Saisai Shao commented on SPARK-17566: - Can you confirm the above command you mentioned can be run on

[jira] [Comment Edited] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495243#comment-15495243 ] Saisai Shao edited comment on SPARK-17522 at 9/16/16 3:19 AM: -- [~sunrui] I

[jira] [Commented] (SPARK-17522) [MESOS] More even distribution of executors on Mesos cluster

[ https://issues.apache.org/jira/browse/SPARK-17522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495243#comment-15495243 ] Saisai Shao commented on SPARK-17522: - [~sunrui] I think the performance is depended on different

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470553#comment-15470553 ] Saisai Shao commented on SPARK-17340: - I think what [~asukhenko] mentioned in the description is one

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455080#comment-15455080 ] Saisai Shao commented on SPARK-17340: - yarn-client and yarn-cluster has different way to handle

[jira] [Comment Edited] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao edited comment on SPARK-17340 at 9/1/16 11:02 AM: -- I think in

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455076#comment-15455076 ] Saisai Shao commented on SPARK-17340: - You can try not kill local {{yarn#client}} process after

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15455055#comment-15455055 ] Saisai Shao commented on SPARK-17340: - I'm saying yarn cluster mode, I think here in my comment

[jira] [Commented] (SPARK-17340) .sparkStaging not cleaned if application exited incorrectly

[ https://issues.apache.org/jira/browse/SPARK-17340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454777#comment-15454777 ] Saisai Shao commented on SPARK-17340: - I think in your scenario, it is because you killed local

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436222#comment-15436222 ] Saisai Shao commented on SPARK-17204: - Yes, I could reproduce this issue, but not constantly,

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436213#comment-15436213 ] Saisai Shao commented on SPARK-17204: - I think to reflect the issue {{sc.range(0, 0)}} should be

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436205#comment-15436205 ] Saisai Shao commented on SPARK-17204: - No, I tested in yarn cluster, not local mode. > Spark 2.0 off

[jira] [Commented] (SPARK-17204) Spark 2.0 off heap RDD persistence with replication factor 2 leads to in-memory data corruption

[ https://issues.apache.org/jira/browse/SPARK-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436179#comment-15436179 ] Saisai Shao commented on SPARK-17204: - It works OK in my local test with latest build: {code} val

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Summary: Support manual credential updating in the run-time for Spark on YARN (was: Support

[jira] [Updated] (SPARK-17209) Support manual credential updating in the run-time for Spark on YARN

[ https://issues.apache.org/jira/browse/SPARK-17209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-17209: Description: Current Spark on YARN supports time based credential renewal and updating, this

[jira] [Created] (SPARK-17209) Support manual credential updating in the run-time

Saisai Shao created SPARK-17209: --- Summary: Support manual credential updating in the run-time Key: SPARK-17209 URL: https://issues.apache.org/jira/browse/SPARK-17209 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430402#comment-15430402 ] Saisai Shao commented on SPARK-17148: - I manually verified this by explicitly throwing the

[jira] [Created] (SPARK-17019) Expose off-heap memory usage in various places

Saisai Shao created SPARK-17019: --- Summary: Expose off-heap memory usage in various places Key: SPARK-17019 URL: https://issues.apache.org/jira/browse/SPARK-17019 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414481#comment-15414481 ] Saisai Shao commented on SPARK-16966: - Here is the code in {{SparkSubmitArguments}} to handle the

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413201#comment-15413201 ] Saisai Shao commented on SPARK-16966: - Yes, agreed. A better way is to handle this app name thing in

[jira] [Commented] (SPARK-16944) [MESOS] Improve data locality when launching new executors when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-16944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411299#comment-15411299 ] Saisai Shao commented on SPARK-16944: - Does Mesos have the similar concept like Yarn container, also

[jira] [Comment Edited] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411159#comment-15411159 ] Saisai Shao edited comment on SPARK-16914 at 8/8/16 1:48 AM: - So from your

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411159#comment-15411159 ] Saisai Shao commented on SPARK-16914: - So from your description, is this exception mainly due to the

[jira] [Updated] (SPARK-16871) Support getting HBase tokens from multiple clusters dynamically

[ https://issues.apache.org/jira/browse/SPARK-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16871: Summary: Support getting HBase tokens from multiple clusters dynamically (was: Support getting

[jira] [Created] (SPARK-16871) Support getting HBase tokens from multiple clusters and dynamically

Saisai Shao created SPARK-16871: --- Summary: Support getting HBase tokens from multiple clusters and dynamically Key: SPARK-16871 URL: https://issues.apache.org/jira/browse/SPARK-16871 Project: Spark

[jira] [Commented] (SPARK-16864) Comprehensive version info

[ https://issues.apache.org/jira/browse/SPARK-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405487#comment-15405487 ] Saisai Shao commented on SPARK-16864: - A program way to get spark version is to call

[jira] [Commented] (SPARK-14453) Remove SPARK_JAVA_OPTS environment variable

[ https://issues.apache.org/jira/browse/SPARK-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405112#comment-15405112 ] Saisai Shao commented on SPARK-14453: - If you want to fix this issue, it would be better target to

[jira] [Comment Edited] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401566#comment-15401566 ] Saisai Shao edited comment on SPARK-16815 at 8/1/16 6:01 AM: - >From my

[jira] [Commented] (SPARK-16815) Dataset[List[T]] leads to ArrayStoreException

[ https://issues.apache.org/jira/browse/SPARK-16815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401566#comment-15401566 ] Saisai Shao commented on SPARK-16815: - >From my understanding you can use {code}

[jira] [Commented] (SPARK-16817) Enable storing of shuffle data in Alluxio

[ https://issues.apache.org/jira/browse/SPARK-16817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401432#comment-15401432 ] Saisai Shao commented on SPARK-16817: - What's difference compared to use ramdisk to store shuffle

[jira] [Commented] (SPARK-16085) Spark stand-alone ui redirects to RM application master UI for yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395705#comment-15395705 ] Saisai Shao commented on SPARK-16085: - Unfortunately, there's no such configuration for Spark to

[jira] [Commented] (SPARK-16708) ExecutorAllocationManager.numRunningTasks can be negative when stage retry

[ https://issues.apache.org/jira/browse/SPARK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393377#comment-15393377 ] Saisai Shao commented on SPARK-16708: - Looks similar to SPARK-11334, and I have a patch on it, though

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393033#comment-15393033 ] Saisai Shao commented on SPARK-16723: - So maybe this application is not yet started in the yarn side,

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393003#comment-15393003 ] Saisai Shao commented on SPARK-16723: - Did you enable log aggregation in YARN, if not this command is

[jira] [Comment Edited] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393003#comment-15393003 ] Saisai Shao edited comment on SPARK-16723 at 7/26/16 1:36 AM: -- Did you

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392980#comment-15392980 ] Saisai Shao commented on SPARK-16723: - {{yarn logs -applicationId application_1467990031555_0089}} >

[jira] [Commented] (SPARK-16723) exception in thread main org.apache.spark.sparkexception application finished with failed status

[ https://issues.apache.org/jira/browse/SPARK-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15392965#comment-15392965 ] Saisai Shao commented on SPARK-16723: - I think you should check the AM and executor logs to see the

[jira] [Updated] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16540: Description: Currently when running spark on yarn, jars specified with \--jars, \--packages will

[jira] [Created] (SPARK-16540) Jars specified with --jars will added twice when running on YARN

Saisai Shao created SPARK-16540: --- Summary: Jars specified with --jars will added twice when running on YARN Key: SPARK-16540 URL: https://issues.apache.org/jira/browse/SPARK-16540 Project: Spark

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376449#comment-15376449 ] Saisai Shao commented on SPARK-16534: - Maybe I can take a try if no one is working on this :). BTW do

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376432#comment-15376432 ] Saisai Shao commented on SPARK-16534: - Is there anyone working on this? > Kafka 0.10 Python support

[jira] [Commented] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374612#comment-15374612 ] Saisai Shao commented on SPARK-16521: - I see, sorry about the duplication. > Add support of

[jira] [Closed] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16521. --- Resolution: Duplicate > Add support of parameterized configuration for SparkConf >

[jira] [Commented] (SPARK-16522) [MESOS] Spark application throws exception on exit

[ https://issues.apache.org/jira/browse/SPARK-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374583#comment-15374583 ] Saisai Shao commented on SPARK-16522: - Perhaps there's race condition when exiting the Spark

[jira] [Updated] (SPARK-16521) Add support of parameterized configuration for SparkConf

[ https://issues.apache.org/jira/browse/SPARK-16521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16521: Priority: Minor (was: Major) > Add support of parameterized configuration for SparkConf >

[jira] [Created] (SPARK-16521) Add support of parameterized configuration for SparkConf

Saisai Shao created SPARK-16521: --- Summary: Add support of parameterized configuration for SparkConf Key: SPARK-16521 URL: https://issues.apache.org/jira/browse/SPARK-16521 Project: Spark Issue

[jira] [Commented] (SPARK-16428) Spark file system watcher not working on Windows

[ https://issues.apache.org/jira/browse/SPARK-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372885#comment-15372885 ] Saisai Shao commented on SPARK-16428: - bq. Spark detected those files with the above terminal output

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371975#comment-15371975 ] Saisai Shao commented on SPARK-16435: - OK, I will file a small patch to add the warning log about

[jira] [Created] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

Saisai Shao created SPARK-16435: --- Summary: Behavior changes if initialExecutor is less than minExecutor for dynamic allocation Key: SPARK-16435 URL: https://issues.apache.org/jira/browse/SPARK-16435

[jira] [Updated] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-14743: Component/s: YARN > Improve delegation token handling in secure clusters >

[jira] [Closed] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16342. --- Resolution: Duplicate > Add a new Configurable Token Manager for Spark Running on YARN >

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365535#comment-15365535 ] Saisai Shao commented on SPARK-16342: - Close as JIRA as duplicated and move to SPARK-14743. > Add a

[jira] [Commented] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365534#comment-15365534 ] Saisai Shao commented on SPARK-14743: - Post design doc here and move SPARK-16342 to here. > Improve

[jira] [Comment Edited] (SPARK-14743) Improve delegation token handling in secure clusters

[ https://issues.apache.org/jira/browse/SPARK-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365534#comment-15365534 ] Saisai Shao edited comment on SPARK-14743 at 7/7/16 3:18 AM: - Post design doc

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365483#comment-15365483 ] Saisai Shao commented on SPARK-16342: - OK, I see. Sorry I didn't notice your JIRA, let me consolidate

[jira] [Issue Comment Deleted] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Comment: was deleted (was: Thanks [~vanzin] for pointing out the jira, looks like most part of the

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365477#comment-15365477 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, looks like most

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365478#comment-15365478 ] Saisai Shao commented on SPARK-16342: - Thanks [~vanzin] for pointing out the jira, looks like most

[jira] [Closed] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

[ https://issues.apache.org/jira/browse/SPARK-16393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16393. --- Resolution: Duplicate > Move and implement token obtaining logics for built-in token providers >

[jira] [Closed] (SPARK-16392) Build ConfigurableTokenManager framework

[ https://issues.apache.org/jira/browse/SPARK-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-16392. --- Resolution: Duplicate > Build ConfigurableTokenManager framework >

[jira] [Created] (SPARK-16393) Move and implement token obtaining logics for built-in token providers

Saisai Shao created SPARK-16393: --- Summary: Move and implement token obtaining logics for built-in token providers Key: SPARK-16393 URL: https://issues.apache.org/jira/browse/SPARK-16393 Project: Spark

[jira] [Created] (SPARK-16392) Build ConfigurableTokenManager framework

Saisai Shao created SPARK-16392: --- Summary: Build ConfigurableTokenManager framework Key: SPARK-16392 URL: https://issues.apache.org/jira/browse/SPARK-16392 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363592#comment-15363592 ] Saisai Shao commented on SPARK-16382: - I would suggest to fail and complain. Max usually specifies

[jira] [Commented] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361917#comment-15361917 ] Saisai Shao commented on SPARK-16342: - [~tgraves] [~vanzin] [~ste...@apache.org] would be great to

[jira] [Updated] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

[ https://issues.apache.org/jira/browse/SPARK-16342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-16342: Description: Current Spark on YARN token management has some problems: 1. Supported service is

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15361323#comment-15361323 ] Saisai Shao commented on SPARK-15923: - [~WeiqingYang] This is the correct behavior. In the client

[jira] [Created] (SPARK-16342) Add a new Configurable Token Manager for Spark Running on YARN

Saisai Shao created SPARK-16342: --- Summary: Add a new Configurable Token Manager for Spark Running on YARN Key: SPARK-16342 URL: https://issues.apache.org/jira/browse/SPARK-16342 Project: Spark

[jira] [Commented] (SPARK-16230) Executors self-killing after being assigned tasks while still in init

[ https://issues.apache.org/jira/browse/SPARK-16230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354280#comment-15354280 ] Saisai Shao commented on SPARK-16230: - I'm not sure which version Spark are you using? Here is a

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354207#comment-15354207 ] Saisai Shao commented on SPARK-16265: - If you want to run Spark on different JVM other than YARN, you

[jira] [Commented] (SPARK-16246) Too many block-manager-slave-async-thread opened (TIMED_WAITING) for spark Kafka streaming

[ https://issues.apache.org/jira/browse/SPARK-16246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352481#comment-15352481 ] Saisai Shao commented on SPARK-16246: - It would be better to have a thread dump about running

[jira] [Commented] (SPARK-16146) Spark application failed by Yarn preempting

[ https://issues.apache.org/jira/browse/SPARK-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345984#comment-15345984 ] Saisai Shao commented on SPARK-16146: - I see, that could explain why executor get lost so frequently

[jira] [Created] (SPARK-16166) Correctly honor off heap memory usage in web ui and log display

Saisai Shao created SPARK-16166: --- Summary: Correctly honor off heap memory usage in web ui and log display Key: SPARK-16166 URL: https://issues.apache.org/jira/browse/SPARK-16166 Project: Spark

[jira] [Commented] (SPARK-16146) Spark application failed by Yarn preempting

[ https://issues.apache.org/jira/browse/SPARK-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345942#comment-15345942 ] Saisai Shao commented on SPARK-16146: - If it is due to preemption, AM log will show the details of

[jira] [Commented] (SPARK-16085) Spark stand-alone ui redirects to RM application master UI for yarn-client mode

[ https://issues.apache.org/jira/browse/SPARK-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345445#comment-15345445 ] Saisai Shao commented on SPARK-16085: - [~yeshavora], it is the expected behavior, the spark UI will

[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN

[ https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15339136#comment-15339136 ] Saisai Shao commented on SPARK-15984: - I see. Here it is because user didn't specify this

[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN

[ https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337278#comment-15337278 ] Saisai Shao commented on SPARK-15984: - Is there any problem? I guess you might set max app attempt to

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335039#comment-15335039 ] Saisai Shao commented on SPARK-15343: - If timeline is enabled, YarnClient will also post some events

[jira] [Updated] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15343: Attachment: (was: jersey-client-2.22.2.jar) > NoClassDefFoundError when initializing Spark

[jira] [Updated] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15343: Attachment: jersey-client-2.22.2.jar > NoClassDefFoundError when initializing Spark with YARN >

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334719#comment-15334719 ] Saisai Shao commented on SPARK-15343: - The class ClientConfig is still existed but the package name

[jira] [Comment Edited] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334683#comment-15334683 ] Saisai Shao edited comment on SPARK-15343 at 6/16/16 9:06 PM: -- [~vanzin]

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334683#comment-15334683 ] Saisai Shao commented on SPARK-15343: - [~vanzin] [~srowen], I don't think it is a vendor specific

[jira] [Created] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

Saisai Shao created SPARK-15990: --- Summary: Support rolling log aggregation for Spark running on YARN Key: SPARK-15990 URL: https://issues.apache.org/jira/browse/SPARK-15990 Project: Spark

[jira] [Commented] (SPARK-15690) Fast single-node (single-process) in-memory shuffle

[ https://issues.apache.org/jira/browse/SPARK-15690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328378#comment-15328378 ] Saisai Shao commented on SPARK-15690: - I see. Since everything is in a single process, looks like

[jira] [Commented] (SPARK-15690) Fast single-node (single-process) in-memory shuffle

[ https://issues.apache.org/jira/browse/SPARK-15690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15328255#comment-15328255 ] Saisai Shao commented on SPARK-15690: - Hi [~rxin], what's the meaning of "single-process", is that

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322892#comment-15322892 ] Saisai Shao commented on SPARK-15828: - OK, I guess you're running on AWS or similar cloud

[jira] [Commented] (SPARK-15828) YARN is not aware of Spark's External Shuffle Service

[ https://issues.apache.org/jira/browse/SPARK-15828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322865#comment-15322865 ] Saisai Shao commented on SPARK-15828: - I see, but I don't clearly understand your scenario, are the

[jira] [Commented] (SPARK-15800) Accessing kerberised hdfs from Spark running with Resource Manager

[ https://issues.apache.org/jira/browse/SPARK-15800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322804#comment-15322804 ] Saisai Shao commented on SPARK-15800: - {quote} Spark is currently running using the Resource Manager,

[jira] [Commented] (SPARK-15801) spark-submit --num-executors switch also works without YARN

[ https://issues.apache.org/jira/browse/SPARK-15801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322800#comment-15322800 ] Saisai Shao commented on SPARK-15801: - It has already been mentioned in {{spark-submit --help}}:

<    2   3   4   5   6   7   8   9   10   11   >