[jira] [Created] (SPARK-23845) Continuous rate source uses different offset format

Saisai Shao created SPARK-23845: --- Summary: Continuous rate source uses different offset format Key: SPARK-23845 URL: https://issues.apache.org/jira/browse/SPARK-23845 Project: Spark Issue

[jira] [Created] (SPARK-23844) Socket Stream recovering from checkpoint will throw exception

Saisai Shao created SPARK-23844: --- Summary: Socket Stream recovering from checkpoint will throw exception Key: SPARK-23844 URL: https://issues.apache.org/jira/browse/SPARK-23844 Project: Spark

[jira] [Resolved] (SPARK-23767) DirectStream is producing the incorrect type of message

[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23767. - Resolution: Not A Problem > DirectStream is producing the incorrect type of message >

[jira] [Commented] (SPARK-23767) DirectStream is producing the incorrect type of message

[ https://issues.apache.org/jira/browse/SPARK-23767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16420291#comment-16420291 ] Saisai Shao commented on SPARK-23767: - This seems not a Spark issue, more like a microsoft event-hub

[jira] [Assigned] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23743: --- Assignee: Jongyoul Lee > IsolatedClientLoader.isSharedClass returns an unindented result

[jira] [Resolved] (SPARK-23743) IsolatedClientLoader.isSharedClass returns an unindented result against `slf4j` keyword

[ https://issues.apache.org/jira/browse/SPARK-23743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23743. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20860

[jira] [Commented] (SPARK-23807) Add Hadoop 3 profile with relevant POM fix ups, cloud-storage artifacts and binding

[ https://issues.apache.org/jira/browse/SPARK-23807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418298#comment-16418298 ] Saisai Shao commented on SPARK-23807: - Hi [~ste...@apache.org], this is dup of SPARK-23534, I think

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413412#comment-16413412 ] Saisai Shao commented on SPARK-22876: - Yes, that's the key problem. Current YARN doesn't support

[jira] [Resolved] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23787. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20895

[jira] [Assigned] (SPARK-23787) SparkSubmitSuite::"download remote resource if it is not supported by yarn" fails on Hadoop 2.9

[ https://issues.apache.org/jira/browse/SPARK-23787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23787: --- Assignee: Marcelo Vanzin > SparkSubmitSuite::"download remote resource if it is not

[jira] [Resolved] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23361. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20657

[jira] [Assigned] (SPARK-23361) Driver restart fails if it happens after 7 days from app submission

[ https://issues.apache.org/jira/browse/SPARK-23361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23361: --- Assignee: Marcelo Vanzin > Driver restart fails if it happens after 7 days from app

[jira] [Updated] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23644: Fix Version/s: 2.3.1 > SHS with proxy doesn't show applications >

[jira] [Issue Comment Deleted] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Comment: was deleted (was: I'm curious why there's a library which depends on a such new version

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405659#comment-16405659 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a library which depends on a such new

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405658#comment-16405658 ] Saisai Shao commented on SPARK-23534: - I'm curious why there's a library which depends on a such new

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405649#comment-16405649 ] Saisai Shao commented on SPARK-23534: - I'm trying to move forward this in the community. But due to

[jira] [Resolved] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23708. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20845

[jira] [Assigned] (SPARK-23708) Comment of ShutdownHookManager.addShutdownHook is error

[ https://issues.apache.org/jira/browse/SPARK-23708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23708: --- Assignee: zhoukang > Comment of ShutdownHookManager.addShutdownHook is error >

[jira] [Resolved] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23635. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20799

[jira] [Assigned] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23635: --- Assignee: Saisai Shao > Spark executor env variable is overwritten by same name AM env

[jira] [Commented] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16401538#comment-16401538 ] Saisai Shao commented on SPARK-23644: - Fixed in https://github.com/apache/spark/pull/20794. > SHS

[jira] [Resolved] (SPARK-23644) SHS with proxy doesn't show applications

[ https://issues.apache.org/jira/browse/SPARK-23644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23644. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.4.0 > SHS with proxy

[jira] [Created] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

Saisai Shao created SPARK-23635: --- Summary: Spark executor env variable is overwritten by same name AM env variable Key: SPARK-23635 URL: https://issues.apache.org/jira/browse/SPARK-23635 Project: Spark

[jira] [Updated] (SPARK-23635) Spark executor env variable is overwritten by same name AM env variable

[ https://issues.apache.org/jira/browse/SPARK-23635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23635: Description: In the current Spark on YARN code, AM always will copy and overwrite its env

[jira] [Comment Edited] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388761#comment-16388761 ] Saisai Shao edited comment on SPARK-23534 at 3/7/18 12:37 AM: -- I don't think

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388761#comment-16388761 ] Saisai Shao commented on SPARK-23534: - I don't think so. Spark uses its own fork hive version

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16385474#comment-16385474 ] Saisai Shao commented on SPARK-18673: - Thanks for the info [~aihuaxu]. > Dataframes doesn't work on

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382932#comment-16382932 ] Saisai Shao commented on SPARK-23534: - One issue is Hive 1.2.1.spark2 rejects Hadoop 3 (SPARK-18673)

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382929#comment-16382929 ] Saisai Shao commented on SPARK-18673: - Spark itself uses it own hive (1.2.1.spark2), I think we need

[jira] [Updated] (SPARK-23534) Spark run on Hadoop 3.0.0

[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23534: Description: Major Hadoop vendors already/will step in Hadoop 3.0. So we should also make sure

[jira] [Created] (SPARK-23534) Spark run on Hadoop 3.0.0

Saisai Shao created SPARK-23534: --- Summary: Spark run on Hadoop 3.0.0 Key: SPARK-23534 URL: https://issues.apache.org/jira/browse/SPARK-23534 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-12140) Support Streaming UI in HistoryServer

[ https://issues.apache.org/jira/browse/SPARK-12140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363300#comment-16363300 ] Saisai Shao commented on SPARK-12140: - [~gschiavon] the community has concern about supporting

[jira] [Commented] (SPARK-23096) Migrate rate source to v2

[ https://issues.apache.org/jira/browse/SPARK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358064#comment-16358064 ] Saisai Shao commented on SPARK-23096: - [~joseph.torres] [~tdas] can I take a crack on this if you're

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357824#comment-16357824 ] Saisai Shao commented on SPARK-23206: - [~cltlfcjin] from Ebay also plans to do similar things, I

[jira] [Commented] (SPARK-23235) Add executor Threaddump to api

[ https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349604#comment-16349604 ] Saisai Shao commented on SPARK-23235: - There's a new similar Jira (SPARK-23206) about adding and

[jira] [Updated] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23200: Fix Version/s: (was: 2.4.0) > Reset configuration when restarting from checkpoints >

[jira] [Reopened] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened SPARK-23200: - Assignee: (was: Santiago Saavedra) > Reset configuration when restarting from checkpoints

[jira] [Resolved] (SPARK-23279) Avoid triggering distributed job for Console sink

[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23279. - Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.3.0 > Avoid triggering

[jira] [Commented] (SPARK-23279) Avoid triggering distributed job for Console sink

[ https://issues.apache.org/jira/browse/SPARK-23279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346301#comment-16346301 ] Saisai Shao commented on SPARK-23279: - Issue resolved by pull request 20447

[jira] [Created] (SPARK-23279) Avoid triggering distributed job for Console sink

Saisai Shao created SPARK-23279: --- Summary: Avoid triggering distributed job for Console sink Key: SPARK-23279 URL: https://issues.apache.org/jira/browse/SPARK-23279 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-23088) History server not showing incomplete/running applications

[ https://issues.apache.org/jira/browse/SPARK-23088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23088. - Resolution: Fixed Assignee: paul mackles Fix Version/s: 2.4.0 Issue resolved by

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342824#comment-16342824 ] Saisai Shao commented on SPARK-23187: - I think I tried, {{accumUpdates}} can be received in driver

[jira] [Created] (SPARK-23228) Able to track Python create SparkSession in JVM

Saisai Shao created SPARK-23228: --- Summary: Able to track Python create SparkSession in JVM Key: SPARK-23228 URL: https://issues.apache.org/jira/browse/SPARK-23228 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-23200: --- Assignee: Anirudh Ramanathan > Reset configuration when restarting from checkpoints >

[jira] [Commented] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340648#comment-16340648 ] Saisai Shao commented on SPARK-23200: - Issue resolved by pull request 20383

[jira] [Resolved] (SPARK-23200) Reset configuration when restarting from checkpoints

[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23200. - Resolution: Fixed Fix Version/s: 2.4.0 > Reset configuration when restarting from

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340536#comment-16340536 ] Saisai Shao commented on SPARK-23206: - Would you please summarize what kind of metrics you want to

[jira] [Resolved] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23187. - Resolution: Not A Problem > Accumulator object can not be sent from Executor to Driver >

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340522#comment-16340522 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as there's no issue in reporting

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340521#comment-16340521 ] Saisai Shao commented on SPARK-23187: - I'm going to close this JIRA, as there's no issue in reporting

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338737#comment-16338737 ] Saisai Shao commented on SPARK-23187: - Actually heartbeat report is OK according to my investigation,

[jira] [Commented] (SPARK-23206) Additional Memory Tuning Metrics

[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338553#comment-16338553 ] Saisai Shao commented on SPARK-23206: - I think this Jira duplicates SPARK-9103. Also seems some

[jira] [Commented] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335428#comment-16335428 ] Saisai Shao commented on SPARK-23187: - I will try to investigate on it. > Accumulator object can not

[jira] [Updated] (SPARK-23187) Accumulator object can not be sent from Executor to Driver

[ https://issues.apache.org/jira/browse/SPARK-23187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23187: Affects Version/s: (was: 2.3.1) 2.3.0 > Accumulator object can not be

[jira] [Resolved] (SPARK-22838) Avoid unnecessary copying of data

[ https://issues.apache.org/jira/browse/SPARK-22838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22838. - Resolution: Invalid > Avoid unnecessary copying of data > - > >

[jira] [Assigned] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22976: --- Assignee: Russell Spitzer > Worker cleanup can remove running driver directories >

[jira] [Resolved] (SPARK-22976) Worker cleanup can remove running driver directories

[ https://issues.apache.org/jira/browse/SPARK-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22976. - Resolution: Fixed Fix Version/s: 2.3.0 > Worker cleanup can remove running driver

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331735#comment-16331735 ] Saisai Shao commented on SPARK-23097: - Thanks, will do. > Migrate text socket source >

[jira] [Commented] (SPARK-23097) Migrate text socket source

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331715#comment-16331715 ] Saisai Shao commented on SPARK-23097: - Hi [~joseph.torres], would you mind if I take a shot on this

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16331542#comment-16331542 ] Saisai Shao commented on SPARK-23128: - I'm wondering if we need a SPIP for such changes? > A new

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Description: Current Stage page's task table will throw an exception when there's no complete

[jira] [Created] (SPARK-23147) Stage page will throw exception when there's no complete tasks

Saisai Shao created SPARK-23147: --- Summary: Stage page will throw exception when there's no complete tasks Key: SPARK-23147 URL: https://issues.apache.org/jira/browse/SPARK-23147 Project: Spark

[jira] [Updated] (SPARK-23147) Stage page will throw exception when there's no complete tasks

[ https://issues.apache.org/jira/browse/SPARK-23147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23147: Attachment: Screen Shot 2018-01-18 at 8.50.08 PM.png > Stage page will throw exception when

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Description: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner, which will

[jira] [Updated] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

[ https://issues.apache.org/jira/browse/SPARK-23140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23140: Environment: (was: DataSourceV2Strategy is not added into HiveSessionStateBuilder's planner,

[jira] [Created] (SPARK-23140) DataSourceV2Strategy is missing in HiveSessionStateBuilder

Saisai Shao created SPARK-23140: --- Summary: DataSourceV2Strategy is missing in HiveSessionStateBuilder Key: SPARK-23140 URL: https://issues.apache.org/jira/browse/SPARK-23140 Project: Spark

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328364#comment-16328364 ] Saisai Shao commented on SPARK-23123: - [~ste...@apache.org], any thought on this issue? > Unable to

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328361#comment-16328361 ] Saisai Shao commented on SPARK-23123: - >From the stack, looks like the issue is from YARN to

[jira] [Commented] (SPARK-23123) Unable to run Spark Job with Hadoop NameNode Federation using ViewFS

[ https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328297#comment-16328297 ] Saisai Shao commented on SPARK-23123: - It looks like a Hadoop issue rather than a Spark issue from

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325986#comment-16325986 ] Saisai Shao commented on SPARK-22958: - Please see this line:

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323872#comment-16323872 ] Saisai Shao commented on SPARK-23029: - There should be no issue unless you want to fix doc issue...

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323871#comment-16323871 ] Saisai Shao commented on SPARK-23029: - Total buffer size for one executor is {{32MB * C (core) * R

[jira] [Commented] (SPARK-21475) Change to use NIO's Files API for external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323590#comment-16323590 ] Saisai Shao commented on SPARK-21475: - [~zsxwing] is 3.0.0 the valid fix version? > Change to use

[jira] [Commented] (SPARK-22958) Spark is stuck when the only one executor fails to register with driver

[ https://issues.apache.org/jira/browse/SPARK-22958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323569#comment-16323569 ] Saisai Shao commented on SPARK-22958: - If executor is failed to register itself to driver, it will

[jira] [Resolved] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-23029. - Resolution: Not A Problem > Setting spark.shuffle.file.buffer will make the shuffle fail >

[jira] [Commented] (SPARK-23029) Setting spark.shuffle.file.buffer will make the shuffle fail

[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323496#comment-16323496 ] Saisai Shao commented on SPARK-23029: - Please see the comments in the code in

[jira] [Commented] (SPARK-22921) Merge script should prompt for assigning jiras

[ https://issues.apache.org/jira/browse/SPARK-22921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321669#comment-16321669 ] Saisai Shao commented on SPARK-22921: - Hi [~irashid], looks like the changes will throw an exception

[jira] [Resolved] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22587. - Resolution: Fixed > Spark job fails if fs.defaultFS and application jar are different url >

[jira] [Updated] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22587: Fix Version/s: 2.3.0 > Spark job fails if fs.defaultFS and application jar are different url >

[jira] [Assigned] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22587: --- Assignee: Mingjie Tang > Spark job fails if fs.defaultFS and application jar are different

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309363#comment-16309363 ] Saisai Shao commented on SPARK-22876: - This looks like a bug, ideally we should get number of

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273985#comment-16273985 ] Saisai Shao commented on SPARK-22393: - Shall we wait for this before 2.2.1 is out? > spark-shell

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265035#comment-16265035 ] Saisai Shao commented on SPARK-2926: So your saying of 12x-30x boosting is only referring to reduce

[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265021#comment-16265021 ] Saisai Shao commented on SPARK-22579: - I think this issue should have already been fixed by

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264986#comment-16264986 ] Saisai Shao commented on SPARK-22587: - [~Prabhu Joseph], I think it is because the logic of comparing

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264904#comment-16264904 ] Saisai Shao commented on SPARK-2926: [~XuanYuan], would you please use spark-perf's micro benchmark

[jira] [Commented] (SPARK-22528) History service and non-HDFS filesystems

[ https://issues.apache.org/jira/browse/SPARK-22528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254669#comment-16254669 ] Saisai Shao commented on SPARK-22528: - Hi [~adobe_pmackles] I think this issue might be due to the

[jira] [Commented] (SPARK-9104) expose network layer memory usage

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250675#comment-16250675 ] Saisai Shao commented on SPARK-9104: [~vsr] I think SPARK-21934 already exposed Netty shuffle metrics

[jira] [Commented] (SPARK-22426) Spark AM launching containers on node where External spark shuffle service failed to initialize

[ https://issues.apache.org/jira/browse/SPARK-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236939#comment-16236939 ] Saisai Shao commented on SPARK-22426: - This kind of scenario was handled in SPARK-13669 regarding to

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235042#comment-16235042 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comment, let me propose a

[jira] [Assigned] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22172: --- Assignee: Devaraj K > Worker hangs when the external shuffle service port is already in use

[jira] [Resolved] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22172. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19396

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16233569#comment-16233569 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comments. bq. When implementing

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226724#comment-16226724 ] Saisai Shao commented on SPARK-22405: - Thanks [~cloud_fan] for your comments. Using fake

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226474#comment-16226474 ] Saisai Shao commented on SPARK-22405: - CC [~smilegator], what's your opinion about this proposal?

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226468#comment-16226468 ] Saisai Shao commented on SPARK-22405: - This is the WIP branch

[jira] [Created] (SPARK-22405) Enrich the event information of ExternalCatalogEvent

Saisai Shao created SPARK-22405: --- Summary: Enrich the event information of ExternalCatalogEvent Key: SPARK-22405 URL: https://issues.apache.org/jira/browse/SPARK-22405 Project: Spark Issue

[jira] [Updated] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22405: Summary: Enrich the event information and add new event of ExternalCatalogEvent (was: Enrich the

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220431#comment-16220431 ] Saisai Shao commented on SPARK-22357: - Yes, I know this parameter is ignored, but I'm not sure is it

<    1   2   3   4   5   6   7   8   9   10   >