[jira] [Commented] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943398#comment-15943398 ] Apache Spark commented on SPARK-20107: -- User 'wangyum' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20088) Do not create new SparkContext in SparkR createSparkContext

2017-03-27 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-20088. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17423

[jira] [Resolved] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20104. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17438

[jira] [Assigned] (SPARK-20104) Don't estimate IsNull or IsNotNull predicates for non-leaf node

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20104: --- Assignee: Zhenhua Wang > Don't estimate IsNull or IsNotNull predicates for non-leaf node >

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Summary: Speed up HadoopMapReduceCommitProtocol#commitJob for many output files (was: Speed up

[jira] [Assigned] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20107: Assignee: Apache Spark > Speed up HadoopMapReduceCommitProtocol#commitJob for many output

[jira] [Assigned] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20107: Assignee: (was: Apache Spark) > Speed up HadoopMapReduceCommitProtocol#commitJob for

[jira] [Assigned] (SPARK-20088) Do not create new SparkContext in SparkR createSparkContext

2017-03-27 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-20088: - Assignee: Hossein Falaki > Do not create new SparkContext in SparkR createSparkContext

[jira] [Updated] (SPARK-20037) impossible to set kafka offsets using kafka 0.10 and spark 2.0.0

2017-03-27 Thread Daniel Nuriyev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Nuriyev updated SPARK-20037: --- Attachment: Main.java > impossible to set kafka offsets using kafka 0.10 and spark 2.0.0 >

[jira] [Resolved] (SPARK-20105) Add tests for checkType and type string in structField in R

2017-03-27 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung resolved SPARK-20105. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-20037) impossible to set kafka offsets using kafka 0.10 and spark 2.0.0

2017-03-27 Thread Daniel Nuriyev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943642#comment-15943642 ] Daniel Nuriyev commented on SPARK-20037: My system is absolutely simple: a topic whose offset

[jira] [Commented] (SPARK-20037) impossible to set kafka offsets using kafka 0.10 and spark 2.0.0

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943652#comment-15943652 ] Sean Owen commented on SPARK-20037: --- I cannot reproduce this in my application. There is more to your

[jira] [Commented] (SPARK-20037) impossible to set kafka offsets using kafka 0.10 and spark 2.0.0

2017-03-27 Thread Daniel Nuriyev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943654#comment-15943654 ] Daniel Nuriyev commented on SPARK-20037: Thank you for your feedback, This problem started when I

[jira] [Commented] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943755#comment-15943755 ] Apache Spark commented on SPARK-20087: -- User 'noodle-fb' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20087: Assignee: (was: Apache Spark) > Include accumulators / taskMetrics when sending

[jira] [Assigned] (SPARK-20087) Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd listeners

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20087: Assignee: Apache Spark > Include accumulators / taskMetrics when sending TaskKilled to

[jira] [Resolved] (SPARK-20102) Fix two minor build script issues blocking 2.1.1 RC + master snapshot builds

2017-03-27 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-20102. Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 Fixed for 2.1.1 and master.

[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943799#comment-15943799 ] Thomas Graves commented on SPARK-19904: --- Is this done or what is this waiting on? > SPIP Add

[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-27 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943827#comment-15943827 ] Cody Koeninger commented on SPARK-19904: It has been added to apache/spark-website git repo

[jira] [Commented] (SPARK-20083) Change matrix toArray to not create a new array when matrix is already column major

2017-03-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943857#comment-15943857 ] yuhao yang commented on SPARK-20083: So the result array will allow users to manipulate the matrix

[jira] [Commented] (SPARK-20083) Change matrix toArray to not create a new array when matrix is already column major

2017-03-27 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943873#comment-15943873 ] Seth Hendrickson commented on SPARK-20083: -- Yes, that would be the intention. We have to take

[jira] [Created] (SPARK-20110) Windowed aggregation do not work when the timestamp is a nested field

2017-03-27 Thread Alexis Seigneurin (JIRA)
Alexis Seigneurin created SPARK-20110: - Summary: Windowed aggregation do not work when the timestamp is a nested field Key: SPARK-20110 URL: https://issues.apache.org/jira/browse/SPARK-20110

[jira] [Commented] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Block

2017-03-27 Thread John Compitello (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943966#comment-15943966 ] John Compitello commented on SPARK-20109: - I have a PR for this issue in the works, I'd like to

[jira] [Created] (SPARK-20109) Need a way to convert from IndexedRowMatrix to Block

2017-03-27 Thread John Compitello (JIRA)
John Compitello created SPARK-20109: --- Summary: Need a way to convert from IndexedRowMatrix to Block Key: SPARK-20109 URL: https://issues.apache.org/jira/browse/SPARK-20109 Project: Spark

[jira] [Updated] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20059: Description: {{HBaseCredentialProvider}} uses system classloader instead of child classloader,

[jira] [Created] (SPARK-20106) Nonlazy caching of DataFrame after orderBy/sortBy

2017-03-27 Thread Richard Liebscher (JIRA)
Richard Liebscher created SPARK-20106: - Summary: Nonlazy caching of DataFrame after orderBy/sortBy Key: SPARK-20106 URL: https://issues.apache.org/jira/browse/SPARK-20106 Project: Spark

[jira] [Commented] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942976#comment-15942976 ] Saisai Shao commented on SPARK-20059: - [~sowen], it probably is not the same issue. In SPARK-20019,

[jira] [Resolved] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved SPARK-20061. Resolution: Duplicate > Reading a file with colon (:) from S3 fails with

[jira] [Commented] (SPARK-20103) Spark structured steaming from kafka - last message processed again after resume from checkpoint

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942948#comment-15942948 ] Sean Owen commented on SPARK-20103: --- Looks like a duplicate of things like SPARK-20050 > Spark

[jira] [Comment Edited] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942976#comment-15942976 ] Saisai Shao edited comment on SPARK-20059 at 3/27/17 10:02 AM: --- [~sowen],

[jira] [Updated] (SPARK-20106) Nonlazy caching of DataFrame after orderBy/sort

2017-03-27 Thread Richard Liebscher (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Liebscher updated SPARK-20106: -- Summary: Nonlazy caching of DataFrame after orderBy/sort (was: Nonlazy caching of

[jira] [Commented] (SPARK-19999) Test failures in Spark Core due to java.nio.Bits.unaligned()

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942942#comment-15942942 ] Sean Owen commented on SPARK-1: --- [~Sonia] please see http://spark.apache.org/contributing.html --

[jira] [Commented] (SPARK-20059) HbaseCredentialProvider uses wrong classloader

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942960#comment-15942960 ] Sean Owen commented on SPARK-20059: --- Is this actually the same issue as SPARK-20019 or SPARK-11421? >

[jira] [Commented] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943008#comment-15943008 ] Steve Loughran commented on SPARK-20061: ":" is one of those "implicitly forbidden characters in

[jira] [Reopened] (SPARK-19999) Test failures in Spark Core due to java.nio.Bits.unaligned()

2017-03-27 Thread Sonia Garudi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sonia Garudi reopened SPARK-1: -- I have reopened this issue and attached a patch. The patch checks for the system architecture and

[jira] [Updated] (SPARK-19999) Test failures in Spark Core due to java.nio.Bits.unaligned()

2017-03-27 Thread Sonia Garudi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sonia Garudi updated SPARK-1: - Attachment: Core.patch > Test failures in Spark Core due to java.nio.Bits.unaligned() >

[jira] [Commented] (SPARK-19999) Test failures in Spark Core due to java.nio.Bits.unaligned()

2017-03-27 Thread Sonia Garudi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942743#comment-15942743 ] Sonia Garudi commented on SPARK-1: -- Can somebody review the patch please ? > Test failures in

[jira] [Commented] (SPARK-20106) Nonlazy caching of DataFrame after orderBy/sort

2017-03-27 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943133#comment-15943133 ] Herman van Hovell commented on SPARK-20106: --- Caching requires use the backing RDD. That

[jira] [Comment Edited] (SPARK-20106) Nonlazy caching of DataFrame after orderBy/sort

2017-03-27 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943133#comment-15943133 ] Herman van Hovell edited comment on SPARK-20106 at 3/27/17 12:03 PM: -

[jira] [Commented] (SPARK-15473) CSV fails to write and read back empty dataframe

2017-03-27 Thread Ryan Magnusson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943080#comment-15943080 ] Ryan Magnusson commented on SPARK-15473: [~hyukjin.kwon] are you working on this, or can I take

[jira] [Closed] (SPARK-20106) Nonlazy caching of DataFrame after orderBy/sort

2017-03-27 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell closed SPARK-20106. - Resolution: Not A Problem > Nonlazy caching of DataFrame after orderBy/sort >

[jira] [Created] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-20111: - Summary: codegen bug surfaced by GraphFrames issue 165 Key: SPARK-20111 URL: https://issues.apache.org/jira/browse/SPARK-20111 Project: Spark

[jira] [Commented] (SPARK-19634) Feature parity for descriptive statistics in MLlib

2017-03-27 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944030#comment-15944030 ] Seth Hendrickson commented on SPARK-19634: -- I'm coming to this a bit late, but I'm finding

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: codegen_sorter_crash.log > SIGSEGV in GeneratedIterator.sort_addToSorter >

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: (was: codegen_sorter_crash) > SIGSEGV in GeneratedIterator.sort_addToSorter >

[jira] [Created] (SPARK-20113) overwrite mode appends data on MySQL table that does not have a primary key

2017-03-27 Thread Bhanu Akaveeti (JIRA)
Bhanu Akaveeti created SPARK-20113: -- Summary: overwrite mode appends data on MySQL table that does not have a primary key Key: SPARK-20113 URL: https://issues.apache.org/jira/browse/SPARK-20113

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Description: I'm seeing a very weird crash in {{GeneratedIterator.sort_addToSorter}}. The hs_err_pid and

[jira] [Commented] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944089#comment-15944089 ] Herman van Hovell commented on SPARK-20111: --- [~josephkb] this might be already fixed in the

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: (was: codegen_sorter_crash.log) > SIGSEGV in GeneratedIterator.sort_addToSorter >

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944100#comment-15944100 ] Mitesh commented on SPARK-20112: This kind of looks like

[jira] [Commented] (SPARK-19634) Feature parity for descriptive statistics in MLlib

2017-03-27 Thread Timothy Hunter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944019#comment-15944019 ] Timothy Hunter commented on SPARK-19634: [~dongjin] [~wm624] sorry it looks like I missed your

[jira] [Updated] (SPARK-18127) Add hooks and extension points to Spark

2017-03-27 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-18127: Component/s: (was: Spark Core) SQL > Add hooks and extension points to Spark >

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Description: I'm seeing a very weird crash in {{GeneratedIterator.sort_addToSorter}}. The hs_err_pid and

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

2017-03-27 Thread Darshan Mehta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944077#comment-15944077 ] Darshan Mehta commented on SPARK-14560: --- [~lovemylover] [~mhornbech] Were you able to figure

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

2017-03-27 Thread Morten Hornbech (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944080#comment-15944080 ] Morten Hornbech commented on SPARK-14560: - No, not really. We were able to work around it by

[jira] [Commented] (SPARK-20113) overwrite mode appends data on MySQL table that does not have a primary key

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944079#comment-15944079 ] Sean Owen commented on SPARK-20113: --- If there is no primary key, how would anything know that the data

[jira] [Commented] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Timothy Hunter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944004#comment-15944004 ] Timothy Hunter commented on SPARK-20111: As Spark SQL is making more and more forays into code

[jira] [Commented] (SPARK-19876) Add OneTime trigger executor

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944073#comment-15944073 ] Apache Spark commented on SPARK-19876: -- User 'tdas' has created a pull request for this issue:

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: hs_err_pid19271.log codegen_sorter_crash > SIGSEGV in

[jira] [Created] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
Mitesh created SPARK-20112: -- Summary: SIGSEGV in GeneratedIterator.sort_addToSorter Key: SPARK-20112 URL: https://issues.apache.org/jira/browse/SPARK-20112 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

2017-03-27 Thread Morten Hornbech (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944094#comment-15944094 ] Morten Hornbech commented on SPARK-14560: - No. We worried it might just trigger some other bad

[jira] [Updated] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

2017-03-27 Thread Mitesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mitesh updated SPARK-20112: --- Attachment: codegen_sorter_crash.log > SIGSEGV in GeneratedIterator.sort_addToSorter >

[jira] [Updated] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-20111: -- Description: In GraphFrames, test {{test("named edges")}} in {{PatternMatchSuite}}

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944046#comment-15944046 ] Thomas Graves commented on SPARK-19143: --- Yes I can be Shephard. > API in Spark for distributing

[jira] [Resolved] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19904. --- Resolution: Fixed Fix Version/s: 2.2.0 There's not a bright line between what goes in the

[jira] [Commented] (SPARK-20113) overwrite mode appends data on MySQL table that does not have a primary key

2017-03-27 Thread Bhanu Akaveeti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944088#comment-15944088 ] Bhanu Akaveeti commented on SPARK-20113: I was expecting entire row comparison, but is there a

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

2017-03-27 Thread Darshan Mehta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944090#comment-15944090 ] Darshan Mehta commented on SPARK-14560: --- [~mhornbech] thanks for the prompt response. Before

[jira] [Comment Edited] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944113#comment-15944113 ] Joseph K. Bradley edited comment on SPARK-20111 at 3/27/17 9:59 PM:

[jira] [Resolved] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-20111. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > codegen bug

[jira] [Commented] (SPARK-20111) codegen bug surfaced by GraphFrames issue 165

2017-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944113#comment-15944113 ] Joseph K. Bradley commented on SPARK-20111: --- Yep, you're right. Is the fix worth backporting,

[jira] [Created] (SPARK-20108) Spark query is getting failed with exception

2017-03-27 Thread ZS EDGE (JIRA)
ZS EDGE created SPARK-20108: --- Summary: Spark query is getting failed with exception Key: SPARK-20108 URL: https://issues.apache.org/jira/browse/SPARK-20108 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16938) Cannot resolve column name after a join

2017-03-27 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943382#comment-15943382 ] Michel Lemay commented on SPARK-16938: -- I just stumbled upon a similar issue with 'union' on two

[jira] [Updated] (SPARK-20107) Speed up FileOutputCommitter#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Description: Set {{mapreduce.fileoutputcommitter.algorithm.version=2}} to speed up

[jira] [Commented] (SPARK-19803) Flaky BlockManagerProactiveReplicationSuite tests

2017-03-27 Thread Shubham Chopra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943460#comment-15943460 ] Shubham Chopra commented on SPARK-19803: The PR enforces a refresh of the peer list cached at the

[jira] [Resolved] (SPARK-19803) Flaky BlockManagerProactiveReplicationSuite tests

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19803. - Resolution: Fixed Issue resolved by pull request 17325

[jira] [Created] (SPARK-20116) Remove task-level functionality from the DAGScheduler

2017-03-27 Thread Kay Ousterhout (JIRA)
Kay Ousterhout created SPARK-20116: -- Summary: Remove task-level functionality from the DAGScheduler Key: SPARK-20116 URL: https://issues.apache.org/jira/browse/SPARK-20116 Project: Spark

[jira] [Resolved] (SPARK-20100) Consolidate SessionState construction

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20100. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17433

[jira] [Assigned] (SPARK-19088) Optimize sequence type deserialization codegen

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-19088: --- Assignee: Michal Šenkýř > Optimize sequence type deserialization codegen >

[jira] [Resolved] (SPARK-19088) Optimize sequence type deserialization codegen

2017-03-27 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19088. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16541

[jira] [Created] (SPARK-20117) TaskSetManager checkSpeculatableTasks variables immutability and Use string interpolation

2017-03-27 Thread jianran.tfh (JIRA)
jianran.tfh created SPARK-20117: --- Summary: TaskSetManager checkSpeculatableTasks variables immutability and Use string interpolation Key: SPARK-20117 URL: https://issues.apache.org/jira/browse/SPARK-20117

[jira] [Assigned] (SPARK-20117) TaskSetManager checkSpeculatableTasks variables immutability and Use string interpolation

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20117: Assignee: Apache Spark > TaskSetManager checkSpeculatableTasks variables immutability and

[jira] [Commented] (SPARK-20117) TaskSetManager checkSpeculatableTasks variables immutability and Use string interpolation

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944410#comment-15944410 ] Apache Spark commented on SPARK-20117: -- User 'jianran' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20117) TaskSetManager checkSpeculatableTasks variables immutability and Use string interpolation

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20117: Assignee: (was: Apache Spark) > TaskSetManager checkSpeculatableTasks variables

[jira] [Commented] (SPARK-20070) Redact datasource explain output

2017-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944417#comment-15944417 ] Apache Spark commented on SPARK-20070: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Updated] (SPARK-19757) Executor with task scheduled could be killed due to idleness

2017-03-27 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19757: - Component/s: Scheduler > Executor with task scheduled could be killed due to idleness >

[jira] [Commented] (SPARK-19634) Feature parity for descriptive statistics in MLlib

2017-03-27 Thread Timothy Hunter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944215#comment-15944215 ] Timothy Hunter commented on SPARK-19634: [~sethah], yes, thanks for bringing up these concerns.

[jira] [Updated] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan

2017-03-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang updated SPARK-20114: --- Description: Creating this jira to track the feature parity for PrefixSpan and sequential pattern

[jira] [Commented] (SPARK-19476) Running threads in Spark DataFrame foreachPartition() causes NullPointerException

2017-03-27 Thread Lucy Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944163#comment-15944163 ] Lucy Yu commented on SPARK-19476: - bq. I don't think in general you're expected to be able to do this

[jira] [Updated] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan

2017-03-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang updated SPARK-20114: --- Description: Creating this jira to track the feature parity for PrefixSpan and sequential pattern

[jira] [Created] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan

2017-03-27 Thread yuhao yang (JIRA)
yuhao yang created SPARK-20114: -- Summary: spark.ml parity for sequential pattern mining - PrefixSpan Key: SPARK-20114 URL: https://issues.apache.org/jira/browse/SPARK-20114 Project: Spark Issue

[jira] [Created] (SPARK-20115) Fix DAGScheduler to recompute all the lost shuffle blocks when external shuffle service is unavailable

2017-03-27 Thread Udit Mehrotra (JIRA)
Udit Mehrotra created SPARK-20115: - Summary: Fix DAGScheduler to recompute all the lost shuffle blocks when external shuffle service is unavailable Key: SPARK-20115 URL:

[jira] [Comment Edited] (SPARK-19634) Feature parity for descriptive statistics in MLlib

2017-03-27 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944030#comment-15944030 ] Seth Hendrickson edited comment on SPARK-19634 at 3/27/17 10:23 PM:

[jira] [Commented] (SPARK-19612) Tests failing with timeout

2017-03-27 Thread Kay Ousterhout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944181#comment-15944181 ] Kay Ousterhout commented on SPARK-19612: And another:

[jira] [Commented] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan

2017-03-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944239#comment-15944239 ] yuhao yang commented on SPARK-20114: Currently I prefer to implement the dummy PrefixSpanModel as the

[jira] [Comment Edited] (SPARK-20114) spark.ml parity for sequential pattern mining - PrefixSpan

2017-03-27 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944239#comment-15944239 ] yuhao yang edited comment on SPARK-20114 at 3/27/17 11:42 PM: -- Currently I

[jira] [Commented] (SPARK-19551) Theme for PySpark documenation could do with improving

2017-03-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943211#comment-15943211 ] Hyukjin Kwon commented on SPARK-19551: -- [~arthur-tacca], I am just curious about this. Do you have

[jira] [Commented] (SPARK-20107) Speed up FileOutputCommitter#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943220#comment-15943220 ] Yuming Wang commented on SPARK-20107: - I will create a PR later > Speed up

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943243#comment-15943243 ] Saisai Shao commented on SPARK-19143: - Attach a WIP branch

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943284#comment-15943284 ] Thomas Graves commented on SPARK-19143: --- I assume this needs to go through the new spip process:

[jira] [Commented] (SPARK-15473) CSV fails to write and read back empty dataframe

2017-03-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943183#comment-15943183 ] Hyukjin Kwon commented on SPARK-15473: -- Yup, I am not working on this. Please take over this. > CSV

  1   2   >