[jira] [Updated] (SPARK-21607) Can dropTempView function add a param like dropTempView(viewName: String, dropSelfOnly: Boolean)

2017-08-01 Thread ant_nebula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ant_nebula updated SPARK-21607: --- Attachment: spark stream scene.png > Can dropTempView function add a param like dropTempView(viewName

[jira] [Created] (SPARK-21607) Can dropTempView function add a param like dropTempView(viewName: String, dropSelfOnly: Boolean)

2017-08-01 Thread ant_nebula (JIRA)
ant_nebula created SPARK-21607: -- Summary: Can dropTempView function add a param like dropTempView(viewName: String, dropSelfOnly: Boolean) Key: SPARK-21607 URL: https://issues.apache.org/jira/browse/SPARK-21607

[jira] [Updated] (SPARK-21606) HiveServer2/HiveThriftServer2 catches OOMs on request threads

2017-08-01 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-21606: Description: Refer to Hive, ThriftCLIService methods such as ExecuteStatement are apparently capab

[jira] [Created] (SPARK-21606) HiveServer2/HiveThriftServer2 catches OOMs on request threads

2017-08-01 Thread zuotingbing (JIRA)
zuotingbing created SPARK-21606: --- Summary: HiveServer2/HiveThriftServer2 catches OOMs on request threads Key: SPARK-21606 URL: https://issues.apache.org/jira/browse/SPARK-21606 Project: Spark

[jira] [Updated] (SPARK-21604) if the object extends Logging, i suggest to remove the var LOG which is useless.

2017-08-01 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-21604: Description: if the object extends Logging, i suggest to remove the var LOG which is useless. (wa

[jira] [Updated] (SPARK-21604) if the object extends Logging, i suggest to remove the var LOG which is useless.

2017-08-01 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-21604: Summary: if the object extends Logging, i suggest to remove the var LOG which is useless. (was: E

[jira] [Updated] (SPARK-21605) Let IntelliJ IDEA correctly detect Language level and Target byte code version

2017-08-01 Thread Chang chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang chen updated SPARK-21605: --- Labels: IDE maven (was: ) > Let IntelliJ IDEA correctly detect Language level and Target byte code v

[jira] [Created] (SPARK-21605) Let IntelliJ IDEA correctly detect Language level and Target byte code version

2017-08-01 Thread Chang chen (JIRA)
Chang chen created SPARK-21605: -- Summary: Let IntelliJ IDEA correctly detect Language level and Target byte code version Key: SPARK-21605 URL: https://issues.apache.org/jira/browse/SPARK-21605 Project: S

[jira] [Resolved] (SPARK-21578) Add JavaSparkContextSuite

2017-08-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21578. - Resolution: Fixed > Add JavaSparkContextSuite > - > > Key: SPARK-

[jira] [Created] (SPARK-21604) Error class name for log, and if the object extends Logging, i suggest to remove the var LOG which is useless.

2017-08-01 Thread zuotingbing (JIRA)
zuotingbing created SPARK-21604: --- Summary: Error class name for log, and if the object extends Logging, i suggest to remove the var LOG which is useless. Key: SPARK-21604 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (SPARK-21603) The wholestage codegen will be much slower then wholestage codegen is closed when the function is too long

2017-08-01 Thread eaton (JIRA)
eaton created SPARK-21603: - Summary: The wholestage codegen will be much slower then wholestage codegen is closed when the function is too long Key: SPARK-21603 URL: https://issues.apache.org/jira/browse/SPARK-21603

[jira] [Created] (SPARK-21602) Add map_keys and map_values functions to R

2017-08-01 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21602: Summary: Add map_keys and map_values functions to R Key: SPARK-21602 URL: https://issues.apache.org/jira/browse/SPARK-21602 Project: Spark Issue Type: Improv

[jira] [Commented] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110250#comment-16110250 ] jifei_yang commented on SPARK-21601: Just learned that the spark team is now using ma

[jira] [Closed] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jifei_yang closed SPARK-21601. -- Resolution: Won't Do > Modify the JDK version of the Maven compilation > --

[jira] [Commented] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110234#comment-16110234 ] jifei_yang commented on SPARK-21601: [https://github.com/apache/spark/pull/18807] >

[jira] [Commented] (SPARK-21599) Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

2017-08-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110231#comment-16110231 ] Xiao Li commented on SPARK-21599: - https://github.com/apache/spark/pull/18804 > Collecti

[jira] [Assigned] (SPARK-21599) Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

2017-08-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21599: --- Assignee: Dilip Biswal > Collecting column statistics for datasource tables may fail with > java.ut

[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110226#comment-16110226 ] Liang-Chi Hsieh commented on SPARK-21591: - IIUC, basically the aggregation in Spa

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-01 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110209#comment-16110209 ] Kevin Zhang commented on SPARK-21590: - I think the only problem is the non-negative c

[jira] [Commented] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110204#comment-16110204 ] jifei_yang commented on SPARK-21601: As follows, 1.8 1.8 1.8 > Modify th

[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110196#comment-16110196 ] Yanbo Liang commented on SPARK-21591: - [~viirya] Yep, this is the way we are using, b

[jira] [Updated] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jifei_yang updated SPARK-21601: --- Description: When using maven to compile spark, I want to add a modified jdk property. This is user-f

[jira] [Created] (SPARK-21601) Modify the JDK version of the Maven compilation

2017-08-01 Thread jifei_yang (JIRA)
jifei_yang created SPARK-21601: -- Summary: Modify the JDK version of the Maven compilation Key: SPARK-21601 URL: https://issues.apache.org/jira/browse/SPARK-21601 Project: Spark Issue Type: Impro

[jira] [Comment Edited] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110169#comment-16110169 ] Liang-Chi Hsieh edited comment on SPARK-21591 at 8/2/17 2:41 AM: --

[jira] [Comment Edited] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110169#comment-16110169 ] Liang-Chi Hsieh edited comment on SPARK-21591 at 8/2/17 2:41 AM: --

[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110169#comment-16110169 ] Liang-Chi Hsieh commented on SPARK-21591: - The most straightforward way is simila

[jira] [Updated] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-01 Thread Kevin Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Zhang updated SPARK-21590: Description: I want to calculate (unique) daily access count using structured streaming (2.2.0).

[jira] [Commented] (SPARK-21600) The description of "this requires spark.shuffle.service.enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear

2017-08-01 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110095#comment-16110095 ] guoxiaolongzte commented on SPARK-21600: I will do it. > The description of "thi

[jira] [Created] (SPARK-21600) The description of "this requires spark.shuffle.service.enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear

2017-08-01 Thread guoxiaolongzte (JIRA)
guoxiaolongzte created SPARK-21600: -- Summary: The description of "this requires spark.shuffle.service.enabled to be set" for the spark.dynamicAllocation.enabled configuration item is not clear Key: SPARK-21600 U

[jira] [Commented] (SPARK-21595) introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow

2017-08-01 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110053#comment-16110053 ] Tejas Patil commented on SPARK-21595: - This config was introduced by me in SPARK-1345

[jira] [Commented] (SPARK-21598) Collect usability/events information from Spark History Server

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110028#comment-16110028 ] Marcelo Vanzin commented on SPARK-21598: [~steve_l] has been working for some tim

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110014#comment-16110014 ] Hyukjin Kwon commented on SPARK-12717: -- Would you mind if I ask open a backport? Jus

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110007#comment-16110007 ] Bryan Cutler commented on SPARK-12717: -- Thanks [~hyukjin.kwon]! What are your thoug

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-08-01 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110006#comment-16110006 ] Michael Schmeißer commented on SPARK-650: - Please see my comment from 05/Dec/16 12:

[jira] [Updated] (SPARK-21565) aggregate query fails with watermark on eventTime but works with watermark on timestamp column generated by current_timestamp

2017-08-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21565: - Description: *Short Description: * Aggregation query fails with eventTime as watermark column wh

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1610#comment-1610 ] Shixiong Zhu commented on SPARK-21590: -- Yeah, this is a bug. A timestamp can be nega

[jira] [Created] (SPARK-21599) Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

2017-08-01 Thread Dilip Biswal (JIRA)
Dilip Biswal created SPARK-21599: Summary: Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException Key: SPARK-21599 URL: https://issues.apache.org/jira/browse/SPARK-21599

[jira] [Commented] (SPARK-18580) Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream

2017-08-01 Thread Oleg Muravskiy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109946#comment-16109946 ] Oleg Muravskiy commented on SPARK-18580: Thanks, [~ozzieba]! >From my point of vi

[jira] [Resolved] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12717. -- Resolution: Fixed Fix Version/s: 2.3.0 > pyspark broadcast fails when using multiple thr

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109923#comment-16109923 ] Hyukjin Kwon commented on SPARK-12717: -- I see. Thank you. Yes, I am seeing now. > p

[jira] [Resolved] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21573. --- Resolution: Fixed Assignee: shane knapp Fix Version/s: 2.3.0 Looks good, in that the

[jira] [Assigned] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-12717: - Assignee: Bryan Cutler [~hyukjin.kwon] I added you to the Committers group in JIRA, maybe that l

[jira] [Commented] (SPARK-12717) pyspark broadcast fails when using multiple threads

2017-08-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109911#comment-16109911 ] Hyukjin Kwon commented on SPARK-12717: -- Issue resolved by pull request 18695 https:/

[jira] [Created] (SPARK-21598) Collect usability/events information from Spark History Server

2017-08-01 Thread Eric Vandenberg (JIRA)
Eric Vandenberg created SPARK-21598: --- Summary: Collect usability/events information from Spark History Server Key: SPARK-21598 URL: https://issues.apache.org/jira/browse/SPARK-21598 Project: Spark

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-08-01 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109881#comment-16109881 ] shane knapp commented on SPARK-21573: - ok, we pushed that PR. it should fix things.

[jira] [Created] (SPARK-21597) Avg event time calculated in progress may be wrong

2017-08-01 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-21597: Summary: Avg event time calculated in progress may be wrong Key: SPARK-21597 URL: https://issues.apache.org/jira/browse/SPARK-21597 Project: Spark Issue Type

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-08-01 Thread Sital Kedia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109784#comment-16109784 ] Sital Kedia commented on SPARK-19112: - [~sowen], [~tgraves] - Using zstd compression

[jira] [Commented] (SPARK-10878) Race condition when resolving Maven coordinates via Ivy

2017-08-01 Thread Min Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109778#comment-16109778 ] Min Shen commented on SPARK-10878: -- We hit the same issue in our infrastructure where co

[jira] [Commented] (SPARK-21110) Structs should be usable in inequality filters

2017-08-01 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109734#comment-16109734 ] Andrew Ray commented on SPARK-21110: I'm working on this > Structs should be usable

[jira] [Resolved] (SPARK-21339) spark-shell --packages option does not add jars to classpath on windows

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21339. Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.3.0

[jira] [Updated] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

2017-08-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-13669: - Component/s: Scheduler > Job will always fail in the external shuffle service unavailable situati

[jira] [Commented] (SPARK-21330) Bad partitioning does not allow to read a JDBC table with extreme values on the partition column

2017-08-01 Thread Andrew Ray (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109568#comment-16109568 ] Andrew Ray commented on SPARK-21330: https://github.com/apache/spark/pull/18800 > Ba

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-08-01 Thread shane knapp (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109465#comment-16109465 ] shane knapp commented on SPARK-21573: - i'm chatting w/@joshrosen @ 130pm to discuss h

[jira] [Created] (SPARK-21596) Audit the places calling HDFSMetadataLog.get

2017-08-01 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-21596: Summary: Audit the places calling HDFSMetadataLog.get Key: SPARK-21596 URL: https://issues.apache.org/jira/browse/SPARK-21596 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21573) Tests failing with run-tests.py SyntaxError occasionally in Jenkins

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109440#comment-16109440 ] Sean Owen commented on SPARK-21573: --- FWIW I noticed it fails on 4 and 7 but not on 6. G

[jira] [Resolved] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21593. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull request

[jira] [Assigned] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21593: - Assignee: Sean Owen > Fix broken configuration page > - > >

[jira] [Assigned] (SPARK-21592) Skip maven-compiler-plugin main and test compilations in Maven build

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21592: - Assignee: Grzegorz Slowikowski > Skip maven-compiler-plugin main and test compilations in Maven

[jira] [Resolved] (SPARK-21592) Skip maven-compiler-plugin main and test compilations in Maven build

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21592. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18750 [https://github.co

[jira] [Resolved] (SPARK-20079) Re registration of AM hangs spark cluster in yarn-client mode

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-20079. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0 > Re regist

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109311#comment-16109311 ] Sean Owen commented on SPARK-650: - I still don't see an argument against my primary suggest

[jira] [Resolved] (SPARK-21522) Flaky test: LauncherServerSuite.testStreamFiltering

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21522. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-21522) Flaky test: LauncherServerSuite.testStreamFiltering

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109298#comment-16109298 ] Marcelo Vanzin commented on SPARK-21522: Since the bot didn't add the link: https

[jira] [Commented] (SPARK-21571) Spark history server leaves incomplete or unreadable history files around forever.

2017-08-01 Thread Eric Vandenberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109282#comment-16109282 ] Eric Vandenberg commented on SPARK-21571: - Link to pull request https://github.co

[jira] [Commented] (SPARK-18580) Use spark.streaming.backpressure.initialRate in DirectKafkaInputDStream

2017-08-01 Thread Oz Ben-Ami (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109265#comment-16109265 ] Oz Ben-Ami commented on SPARK-18580: +1 We're using our own dynamic allocation to sca

[jira] [Updated] (SPARK-20657) Speed up Stage page

2017-08-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-20657: --- Description: The Stage page in the UI is very slow when a large number of tasks exist (tens

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-08-01 Thread Louis Bergelson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109179#comment-16109179 ] Louis Bergelson commented on SPARK-650: --- I can't understand how people are dismissing

[jira] [Commented] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109101#comment-16109101 ] Artur Sukhenko commented on SPARK-21593: [~srowen] Yes, {code} spark.storage.repl

[jira] [Updated] (SPARK-21595) introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow

2017-08-01 Thread Stephan Reiling (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Reiling updated SPARK-21595: Description: My pyspark code has the following statement: {code:java} # assign row key fo

[jira] [Created] (SPARK-21595) introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow

2017-08-01 Thread Stephan Reiling (JIRA)
Stephan Reiling created SPARK-21595: --- Summary: introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow Key: SPARK-21595 URL: https://issues.apache.org/jira/browse/SPARK

[jira] [Updated] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-08-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-21563: - Component/s: Scheduler > Race condition when serializing TaskDescriptions and adding jars > -

[jira] [Commented] (SPARK-21563) Race condition when serializing TaskDescriptions and adding jars

2017-08-01 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109060#comment-16109060 ] Imran Rashid commented on SPARK-21563: -- Thanks for the detailed report [~aash], make

[jira] [Updated] (SPARK-21594) Missing probability output from MutilayerPerceptronClassifier

2017-08-01 Thread Joseph Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph Wang updated SPARK-21594: Summary: Missing probability output from MutilayerPerceptronClassifier (was: Probability output fr

[jira] [Commented] (SPARK-21594) Probability output from MutilayerPerceptronClassifier

2017-08-01 Thread Joseph Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109017#comment-16109017 ] Joseph Wang commented on SPARK-21594: - Hi, I have tested with the Spark2.2.0 prebuilt

[jira] [Resolved] (SPARK-21594) Probability output from MutilayerPerceptronClassifier

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21594. --- Resolution: Invalid This should go to u...@spark.apache.org > Probability output from MutilayerPerce

[jira] [Created] (SPARK-21594) Probability output from MutilayerPerceptronClassifier

2017-08-01 Thread Joseph Wang (JIRA)
Joseph Wang created SPARK-21594: --- Summary: Probability output from MutilayerPerceptronClassifier Key: SPARK-21594 URL: https://issues.apache.org/jira/browse/SPARK-21594 Project: Spark Issue Typ

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Description: Latest configuration page for Spark 2.2.0 has broken menu list and named anchor

[jira] [Commented] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108966#comment-16108966 ] Artur Sukhenko commented on SPARK-21593: Yes, as well as having {code}### Dynamic

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Description: Latest configuration page for Spark 2.2.0 has broken menu list and named anchor

[jira] [Commented] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108960#comment-16108960 ] Sean Owen commented on SPARK-21593: --- I couldn't make out what you were reporting from t

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Description: Latest configuration page for Spark 2.2.0 has broken menu list and named anchor

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Attachment: (was: doc_211.png) > Fix broken configuration page >

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Attachment: doc_latest.jpg doc_211.jpg dyn_211.jpg

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Attachment: (was: doc_latest.png) > Fix broken configuration page > -

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Description: Latest configuration page for Spark 2.2.0 has broken menu list and named anchor

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Description: Latest configuration page for Spark 2.2.0 has broken menu list and named anchor

[jira] [Updated] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Sukhenko updated SPARK-21593: --- Attachment: doc_latest.png doc_211.png Latest documentation page with broken

[jira] [Created] (SPARK-21593) Fix broken configuration page

2017-08-01 Thread Artur Sukhenko (JIRA)
Artur Sukhenko created SPARK-21593: -- Summary: Fix broken configuration page Key: SPARK-21593 URL: https://issues.apache.org/jira/browse/SPARK-21593 Project: Spark Issue Type: Bug C

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

2017-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108913#comment-16108913 ] Thomas Graves commented on SPARK-21585: --- https://github.com/apache/spark/pull/18788

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

2017-08-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108911#comment-16108911 ] Sean Owen commented on SPARK-21585: --- Yeah I've noticed -- not sure why or how to fix it

[jira] [Resolved] (SPARK-21585) Application Master marking application status as Failed for Client Mode

2017-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21585. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Application M

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

2017-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108901#comment-16108901 ] Thomas Graves commented on SPARK-21585: --- [~srowen]do you know if the github pull re

[jira] [Resolved] (SPARK-21388) GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold

2017-08-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-21388. - Resolution: Fixed Assignee: zhengruifeng Fix Version/s: 2.3.0 > GBT inherit from

[jira] [Commented] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

2017-08-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108897#comment-16108897 ] Steve Loughran commented on SPARK-21374: I understand...the patch shows the issue

[jira] [Commented] (SPARK-21514) Hive has updated with new support for S3 and InsertIntoHiveTable.scala should update also

2017-08-01 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108882#comment-16108882 ] Steve Loughran commented on SPARK-21514: Can you link this JIRA to the specific H

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

2017-08-01 Thread Vinod KC (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108880#comment-16108880 ] Vinod KC commented on SPARK-21390: -- [~kiszk]case class is compiled differently in the sp

[jira] [Updated] (SPARK-14516) Clustering evaluator

2017-08-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-14516: Priority: Major (was: Minor) > Clustering evaluator > > > Key

[jira] [Updated] (SPARK-14516) Clustering evaluator

2017-08-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-14516: Issue Type: New Feature (was: Brainstorming) > Clustering evaluator > > >

[jira] [Issue Comment Deleted] (SPARK-18535) Redact sensitive information from Spark logs and UI

2017-08-01 Thread Diogo Munaro Vieira (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Diogo Munaro Vieira updated SPARK-18535: Comment: was deleted (was: I did a merge request for version 2.1.2: https://github

[jira] [Updated] (SPARK-21591) Implement treeAggregate on Dataset API

2017-08-01 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-21591: Description: The Tungsten execution engine substantially improved the efficiency of memory and CPU

  1   2   >