[jira] [Commented] (SPARK-15371) YARNShuffleService doesn't get current local-dirs from NodeManager

[ https://issues.apache.org/jira/browse/SPARK-15371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288013#comment-15288013 ] Saisai Shao commented on SPARK-15371: - I think what you mentioned above is relate to

[jira] [Commented] (SPARK-15368) Spark History Server does not pick up extraClasspath

[ https://issues.apache.org/jira/browse/SPARK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15288049#comment-15288049 ] Saisai Shao commented on SPARK-15368: - Maybe you could try {{SPARK_CLASSPATH}}, thoug

[jira] [Created] (SPARK-15375) Add ConsoleSink for structure sink to display the dataframe on the fly

Saisai Shao created SPARK-15375: --- Summary: Add ConsoleSink for structure sink to display the dataframe on the fly Key: SPARK-15375 URL: https://issues.apache.org/jira/browse/SPARK-15375 Project: Spark

[jira] [Updated] (SPARK-15375) Add ConsoleSink for structure streaming to display the dataframe on the fly

[ https://issues.apache.org/jira/browse/SPARK-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15375: Summary: Add ConsoleSink for structure streaming to display the dataframe on the fly (was: Add Con

[jira] [Created] (SPARK-15443) Properly explain the streaming queries

Saisai Shao created SPARK-15443: --- Summary: Properly explain the streaming queries Key: SPARK-15443 URL: https://issues.apache.org/jira/browse/SPARK-15443 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-15443) Properly explain the streaming queries

[ https://issues.apache.org/jira/browse/SPARK-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15443: Description: Currently when called `explain()` on streaming dataset, it will only get the parsed a

[jira] [Updated] (SPARK-15443) Properly explain the streaming queries

[ https://issues.apache.org/jira/browse/SPARK-15443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-15443: Description: Currently when called `explain()` on streaming dataset, it will only get the parsed a

[jira] [Commented] (SPARK-15329) When start spark with yarn: spark.SparkContext: Error initializing SparkContext.

[ https://issues.apache.org/jira/browse/SPARK-15329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15294653#comment-15294653 ] Saisai Shao commented on SPARK-15329: - {code} 2016-05-15 00:06:08,368 WARN org.apach

[jira] [Commented] (SPARK-15368) Spark History Server does not pick up extraClasspath

[ https://issues.apache.org/jira/browse/SPARK-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297827#comment-15297827 ] Saisai Shao commented on SPARK-15368: - Maybe you could try {{SPARK_DIST_CLASSPATH}},

[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

[ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15300071#comment-15300071 ] Saisai Shao commented on SPARK-13955: - You will have 3 options to upload jars to dist

[jira] [Resolved] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22074. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19287 [https://githu

[jira] [Assigned] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22074: --- Assignee: Li Yuanjian > Task killed by other attempt task should not be resubmitted > --

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198204#comment-16198204 ] Saisai Shao commented on SPARK-21737: - Hi [~tgraves], I'm trying to understand the de

[jira] [Commented] (SPARK-22199) Spark Job on YARN fails with executors "Slave registration failed"

[ https://issues.apache.org/jira/browse/SPARK-22199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198258#comment-16198258 ] Saisai Shao commented on SPARK-22199: - Can you please list the steps to reproduce thi

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198674#comment-16198674 ] Saisai Shao commented on SPARK-21737: - I was trying to understand how Spark communica

[jira] [Commented] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200372#comment-16200372 ] Saisai Shao commented on SPARK-22243: - Yes, it is a related issue regarding to Spark

[jira] [Commented] (SPARK-22062) BlockManager does not account for memory consumed by remote fetches

[ https://issues.apache.org/jira/browse/SPARK-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201305#comment-16201305 ] Saisai Shao commented on SPARK-22062: - Yes, there potentially has OOM problem, but I

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16201950#comment-16201950 ] Saisai Shao commented on SPARK-9: - My concern is about how to maintain this code

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202900#comment-16202900 ] Saisai Shao commented on SPARK-9: - {quote} I don't think that limited familiarity

[jira] [Commented] (SPARK-22242) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205018#comment-16205018 ] Saisai Shao commented on SPARK-22242: - It is not resolved, but it is the same type of

[jira] [Assigned] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22290: --- Assignee: Marcelo Vanzin > Starting second context in same JVM fails to get new Hive delegat

[jira] [Resolved] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22290. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19509 [https://githu

[jira] [Commented] (SPARK-22312) Spark job stuck with no executor due to bug in Executor Allocation Manager

[ https://issues.apache.org/jira/browse/SPARK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16212071#comment-16212071 ] Saisai Shao commented on SPARK-22312: - I think it is duplicated to SPARK-11334 accord

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Affects Version/s: 2.2.0 > SparkSubmit calls getFileStatus before calling loginUserFromKeytab > ---

[jira] [Resolved] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22319. - Resolution: Fixed Fix Version/s: 2.3.0 > SparkSubmit calls getFileStatus before calling lo

[jira] [Assigned] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22319: --- Assignee: Steven Rand > SparkSubmit calls getFileStatus before calling loginUserFromKeytab >

[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214569#comment-16214569 ] Saisai Shao commented on SPARK-22272: - Setting "spark.file.transferTo" to "false" wil

[jira] [Resolved] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22272. - Resolution: Won't Fix > killing task may cause the executor progress hang because of the JVM bug

[jira] [Closed] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-22272. --- > killing task may cause the executor progress hang because of the JVM bug >

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Fix Version/s: 2.2.1 > SparkSubmit calls getFileStatus before calling loginUserFromKeytab > ---

[jira] [Resolved] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21840. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19519 [https://githu

[jira] [Assigned] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21840: --- Assignee: Marcelo Vanzin > Allow multiple SparkSubmit invocations in same JVM without pollut

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220160#comment-16220160 ] Saisai Shao commented on SPARK-22357: - [~WeichenXu123] would you please format the co

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220395#comment-16220395 ] Saisai Shao commented on SPARK-22357: - bq. In our specific case this was a problem, b

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220431#comment-16220431 ] Saisai Shao commented on SPARK-22357: - Yes, I know this parameter is ignored, but I'm

[jira] [Updated] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22405: Summary: Enrich the event information and add new event of ExternalCatalogEvent (was: Enrich the e

[jira] [Created] (SPARK-22405) Enrich the event information of ExternalCatalogEvent

Saisai Shao created SPARK-22405: --- Summary: Enrich the event information of ExternalCatalogEvent Key: SPARK-22405 URL: https://issues.apache.org/jira/browse/SPARK-22405 Project: Spark Issue Type

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226468#comment-16226468 ] Saisai Shao commented on SPARK-22405: - This is the WIP branch (https://github.com/je

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226474#comment-16226474 ] Saisai Shao commented on SPARK-22405: - CC [~smilegator], what's your opinion about th

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16226724#comment-16226724 ] Saisai Shao commented on SPARK-22405: - Thanks [~cloud_fan] for your comments. Using f

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233569#comment-16233569 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comments. bq. W

[jira] [Resolved] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22172. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19396 [https://githu

[jira] [Assigned] (SPARK-22172) Worker hangs when the external shuffle service port is already in use

[ https://issues.apache.org/jira/browse/SPARK-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22172: --- Assignee: Devaraj K > Worker hangs when the external shuffle service port is already in use

[jira] [Commented] (SPARK-22405) Enrich the event information and add new event of ExternalCatalogEvent

[ https://issues.apache.org/jira/browse/SPARK-22405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16235042#comment-16235042 ] Saisai Shao commented on SPARK-22405: - Thanks [~hvanhovell] for your comment, let me

[jira] [Commented] (SPARK-22426) Spark AM launching containers on node where External spark shuffle service failed to initialize

[ https://issues.apache.org/jira/browse/SPARK-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16236939#comment-16236939 ] Saisai Shao commented on SPARK-22426: - This kind of scenario was handled in SPARK-136

[jira] [Commented] (SPARK-9104) expose network layer memory usage

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250675#comment-16250675 ] Saisai Shao commented on SPARK-9104: [~vsr] I think SPARK-21934 already exposed Netty

[jira] [Commented] (SPARK-22528) History service and non-HDFS filesystems

[ https://issues.apache.org/jira/browse/SPARK-22528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254669#comment-16254669 ] Saisai Shao commented on SPARK-22528: - Hi [~adobe_pmackles] I think this issue might

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264904#comment-16264904 ] Saisai Shao commented on SPARK-2926: [~XuanYuan], would you please use spark-perf's mi

[jira] [Commented] (SPARK-22587) Spark job fails if fs.defaultFS and application jar are different url

[ https://issues.apache.org/jira/browse/SPARK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264986#comment-16264986 ] Saisai Shao commented on SPARK-22587: - [~Prabhu Joseph], I think it is because the lo

[jira] [Commented] (SPARK-22579) BlockManager.getRemoteValues and BlockManager.getRemoteBytes should be implemented using streaming

[ https://issues.apache.org/jira/browse/SPARK-22579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265021#comment-16265021 ] Saisai Shao commented on SPARK-22579: - I think this issue should have already been fi

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16265035#comment-16265035 ] Saisai Shao commented on SPARK-2926: So your saying of 12x-30x boosting is only referr

[jira] [Commented] (SPARK-22393) spark-shell can't find imported types in class constructors, extends clause

[ https://issues.apache.org/jira/browse/SPARK-22393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273985#comment-16273985 ] Saisai Shao commented on SPARK-22393: - Shall we wait for this before 2.2.1 is out?

[jira] [Commented] (SPARK-24739) PySpark does not work with Python 3.7.0

[ https://issues.apache.org/jira/browse/SPARK-24739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533189#comment-16533189 ] Saisai Shao commented on SPARK-24739: - Do we have to fix it in 2.3.2? I don't think

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533205#comment-16533205 ] Saisai Shao commented on SPARK-24723: - Hi [~mengxr], I would like to know the goal o

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534529#comment-16534529 ] Saisai Shao commented on SPARK-24723: - After discussed with Xiangrui offline, resour

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534565#comment-16534565 ] Saisai Shao commented on SPARK-24723: - [~mengxr] [~jiangxb1987] There's one solutio

[jira] [Comment Edited] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534565#comment-16534565 ] Saisai Shao edited comment on SPARK-24723 at 7/6/18 8:25 AM: -

[jira] [Resolved] (SPARK-24646) Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes

[ https://issues.apache.org/jira/browse/SPARK-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24646. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21633 [https://gith

[jira] [Assigned] (SPARK-24646) Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes

[ https://issues.apache.org/jira/browse/SPARK-24646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24646: --- Assignee: Saisai Shao > Support wildcard '*' for to spark.yarn.dist.forceDownloadSchemes >

[jira] [Resolved] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24678. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21658 [https://gith

[jira] [Assigned] (SPARK-24678) We should use 'PROCESS_LOCAL' first for Spark-Streaming

[ https://issues.apache.org/jira/browse/SPARK-24678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24678: --- Assignee: sharkd tu > We should use 'PROCESS_LOCAL' first for Spark-Streaming > ---

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

[ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539630#comment-16539630 ] Saisai Shao commented on SPARK-24781: - Thanks Felix. Does this have to be in 2.3.2?

[jira] [Commented] (SPARK-24781) Using a reference from Dataset in Filter/Sort might not work.

[ https://issues.apache.org/jira/browse/SPARK-24781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16539635#comment-16539635 ] Saisai Shao commented on SPARK-24781: - I see. I will wait for this before cutting a

[jira] [Updated] (SPARK-24804) There are duplicate words in the title in the DatasetSuite

[ https://issues.apache.org/jira/browse/SPARK-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24804: Fix Version/s: (was: 2.3.2) > There are duplicate words in the title in the DatasetSuite > ---

[jira] [Updated] (SPARK-24804) There are duplicate words in the title in the DatasetSuite

[ https://issues.apache.org/jira/browse/SPARK-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24804: Target Version/s: (was: 2.3.2) > There are duplicate words in the title in the DatasetSuite > --

[jira] [Updated] (SPARK-24804) There are duplicate words in the title in the DatasetSuite

[ https://issues.apache.org/jira/browse/SPARK-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24804: Priority: Trivial (was: Minor) > There are duplicate words in the title in the DatasetSuite > ---

[jira] [Commented] (SPARK-24804) There are duplicate words in the title in the DatasetSuite

[ https://issues.apache.org/jira/browse/SPARK-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544386#comment-16544386 ] Saisai Shao commented on SPARK-24804: - Please don't set target version or fix versio

[jira] [Comment Edited] (SPARK-24804) There are duplicate words in the title in the DatasetSuite

[ https://issues.apache.org/jira/browse/SPARK-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544386#comment-16544386 ] Saisai Shao edited comment on SPARK-24804 at 7/15/18 1:29 AM:

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545962#comment-16545962 ] Saisai Shao commented on SPARK-24615: - Sorry [~tgraves] for the late response. Yes, 

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546590#comment-16546590 ] Saisai Shao commented on SPARK-24615: - Yes, currently the user is responsible for as

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547263#comment-16547263 ] Saisai Shao commented on SPARK-24615: - Sure, I will also add it as Xiangrui also sug

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548662#comment-16548662 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves] I'm rewriting the design doc ba

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549276#comment-16549276 ] Saisai Shao commented on SPARK-24615: - Thanks [~tgraves] for the suggestion.  {quote

[jira] [Updated] (SPARK-24037) stateful operators

[ https://issues.apache.org/jira/browse/SPARK-24037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24037: Fix Version/s: 2.4.0 > stateful operators > -- > > Key: SPARK-2403

[jira] [Updated] (SPARK-24037) stateful operators

[ https://issues.apache.org/jira/browse/SPARK-24037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24037: Fix Version/s: (was: 2.4.0) > stateful operators > -- > > Key:

[jira] [Updated] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24307: Fix Version/s: 2.4.0 > Support sending messages over 2GB from memory > ---

[jira] [Resolved] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24307. - Resolution: Fixed Assignee: Imran Rashid > Support sending messages over 2GB from memory >

[jira] [Commented] (SPARK-24307) Support sending messages over 2GB from memory

[ https://issues.apache.org/jira/browse/SPARK-24307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550167#comment-16550167 ] Saisai Shao commented on SPARK-24307: - Issue resolved by pull request 21440 [https:/

[jira] [Resolved] (SPARK-24195) sc.addFile for local:/ path is broken

[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24195. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21533 [https://gith

[jira] [Assigned] (SPARK-24195) sc.addFile for local:/ path is broken

[ https://issues.apache.org/jira/browse/SPARK-24195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24195: --- Assignee: Li Yuanjian > sc.addFile for local:/ path is broken > ---

[jira] [Commented] (SPARK-24723) Discuss necessary info and access in barrier mode + YARN

[ https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550182#comment-16550182 ] Saisai Shao commented on SPARK-24723: - Hi [~mengxr], I don't think YARN has such fea

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550266#comment-16550266 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves] I'm still not sure how to handl

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552233#comment-16552233 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves] what you mentioned above is als

[jira] [Assigned] (SPARK-24594) Introduce metrics for YARN executor allocation problems

[ https://issues.apache.org/jira/browse/SPARK-24594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24594: --- Assignee: Attila Zsolt Piros > Introduce metrics for YARN executor allocation problems > -

[jira] [Resolved] (SPARK-24594) Introduce metrics for YARN executor allocation problems

[ https://issues.apache.org/jira/browse/SPARK-24594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24594. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21635 [https://gith

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553679#comment-16553679 ] Saisai Shao commented on SPARK-24615: - [~Tagar] yes! > Accelerator-aware task sched

[jira] [Resolved] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

[ https://issues.apache.org/jira/browse/SPARK-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24297. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21474 [https://gith

[jira] [Assigned] (SPARK-24297) Change default value for spark.maxRemoteBlockSizeFetchToMem to be < 2GB

[ https://issues.apache.org/jira/browse/SPARK-24297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-24297: --- Assignee: Imran Rashid > Change default value for spark.maxRemoteBlockSizeFetchToMem to be

[jira] [Commented] (SPARK-24867) Add AnalysisBarrier to DataFrameWriter

[ https://issues.apache.org/jira/browse/SPARK-24867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554986#comment-16554986 ] Saisai Shao commented on SPARK-24867: - [~smilegator] what's the ETA of this issue?

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16555080#comment-16555080 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves] [~irashid] thanks a lot for you

[jira] [Commented] (SPARK-24867) Add AnalysisBarrier to DataFrameWriter

[ https://issues.apache.org/jira/browse/SPARK-24867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556760#comment-16556760 ] Saisai Shao commented on SPARK-24867: - I see, thanks! Please let me know when the JI

[jira] [Updated] (SPARK-24964) Please add OWASP Dependency Check to all comonent builds(pom.xml)

[ https://issues.apache.org/jira/browse/SPARK-24964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24964: Target Version/s: (was: 2.3.2, 2.4.0, 3.0.0, 2.3.3) > Please add OWASP Dependency Check to all

[jira] [Commented] (SPARK-24964) Please add OWASP Dependency Check to all comonent builds(pom.xml)

[ https://issues.apache.org/jira/browse/SPARK-24964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561327#comment-16561327 ] Saisai Shao commented on SPARK-24964: - I'm going to remove the set target version, u

[jira] [Commented] (SPARK-24932) Allow update mode for streaming queries with join

[ https://issues.apache.org/jira/browse/SPARK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561331#comment-16561331 ] Saisai Shao commented on SPARK-24932: - I'm going to remove the target version for th

[jira] [Updated] (SPARK-24932) Allow update mode for streaming queries with join

[ https://issues.apache.org/jira/browse/SPARK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24932: Target Version/s: (was: 2.3.2) > Allow update mode for streaming queries with join > ---

[jira] [Commented] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561334#comment-16561334 ] Saisai Shao commented on SPARK-24957: - Not necessary to mark as blocker, we still ha

[jira] [Updated] (SPARK-24957) Decimal arithmetic can lead to wrong values using codegen

[ https://issues.apache.org/jira/browse/SPARK-24957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-24957: Target Version/s: 2.4.0 > Decimal arithmetic can lead to wrong values using codegen >

[jira] [Resolved] (SPARK-24975) Spark history server REST API /api/v1/version returns error 404

[ https://issues.apache.org/jira/browse/SPARK-24975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-24975. - Resolution: Duplicate > Spark history server REST API /api/v1/version returns error 404 > --

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563730#comment-16563730 ] Saisai Shao commented on SPARK-24615: - Hi [~tgraves], I think eval() might unnecessa

[jira] [Comment Edited] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563730#comment-16563730 ] Saisai Shao edited comment on SPARK-24615 at 7/31/18 2:16 PM:

<    1   2   3   4   5   6   7   8   9   10   >