[jira] [Created] (SPARK-23421) Document the behavior change in SPARK-22356

2018-02-14 Thread Xiao Li (JIRA)
Xiao Li created SPARK-23421: --- Summary: Document the behavior change in SPARK-22356 Key: SPARK-23421 URL: https://issues.apache.org/jira/browse/SPARK-23421 Project: Spark Issue Type: Bug C

[jira] [Commented] (SPARK-23421) Document the behavior change in SPARK-22356

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363631#comment-16363631 ] Apache Spark commented on SPARK-23421: -- User 'gatorsmile' has created a pull request

[jira] [Commented] (SPARK-22356) data source table should support overlapped columns between data and partition schema

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363632#comment-16363632 ] Apache Spark commented on SPARK-22356: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-23421) Document the behavior change in SPARK-22356

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23421: Assignee: Apache Spark (was: Xiao Li) > Document the behavior change in SPARK-22356 > ---

[jira] [Assigned] (SPARK-23421) Document the behavior change in SPARK-22356

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23421: Assignee: Xiao Li (was: Apache Spark) > Document the behavior change in SPARK-22356 > ---

[jira] [Updated] (SPARK-23340) Upgrade Apache ORC to 1.4.3

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23340: Priority: Blocker (was: Critical) > Upgrade Apache ORC to 1.4.3 > --- > >

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363634#comment-16363634 ] Marco Gaido commented on SPARK-23420: - I don't remember the ticket number but this ma

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363657#comment-16363657 ] Marco Gaido commented on SPARK-23402: - I tried with Postgres 10, driver 42.2.1 and I

[jira] [Updated] (SPARK-23409) RandomForest/DecisionTree (syntactic) pruning of redundant subtrees

2018-02-14 Thread Alessandro Solimando (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando updated SPARK-23409: - Description: Improvement: redundancy elimination from decision trees where all th

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363726#comment-16363726 ] Pallapothu Jyothi Swaroop commented on SPARK-23402: --- [~mgaido] please c

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363741#comment-16363741 ] Marco Gaido commented on SPARK-23402: - Yes the table existed. please try with the cur

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363749#comment-16363749 ] Pallapothu Jyothi Swaroop commented on SPARK-23402: --- I am using 2.2.1 v

[jira] [Comment Edited] (SPARK-23308) ignoreCorruptFiles should not ignore retryable IOException

2018-02-14 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360809#comment-16360809 ] Steve Loughran edited comment on SPARK-23308 at 2/14/18 11:27 AM: -

[jira] [Created] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Gabor Somogyi (JIRA)
Gabor Somogyi created SPARK-23422: - Summary: YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1 Key: SPARK-23422 URL: https://issues.apache.org/jira/browse/SPARK-23422 Project: Sp

[jira] [Commented] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363880#comment-16363880 ] Gabor Somogyi commented on SPARK-23422: --- I'm working on the solution. > YarnShuffl

[jira] [Updated] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-23422: -- Description: YarnShuffleIntegrationSuite fails when SPARK_PREPEND_CLASSES set to 1. Normally m

[jira] [Assigned] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23422: Assignee: Apache Spark > YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES se

[jira] [Assigned] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23422: Assignee: (was: Apache Spark) > YarnShuffleIntegrationSuite failure when SPARK_PREPEND

[jira] [Commented] (SPARK-23422) YarnShuffleIntegrationSuite failure when SPARK_PREPEND_CLASSES set to 1

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363946#comment-16363946 ] Apache Spark commented on SPARK-23422: -- User 'gaborgsomogyi' has created a pull requ

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364053#comment-16364053 ] Sean Owen commented on SPARK-23402: --- [~swaroopp] you would need to try with master. It'

[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database

2018-02-14 Thread Pallapothu Jyothi Swaroop (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364088#comment-16364088 ] Pallapothu Jyothi Swaroop commented on SPARK-23402: --- Ok I will try with

[jira] [Resolved] (SPARK-23394) Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does)

2018-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23394. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by pu

[jira] [Assigned] (SPARK-23394) Storage info's Cached Partitions doesn't consider the replications (but sc.getRDDStorageInfo does)

2018-02-14 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23394: -- Assignee: Attila Zsolt Piros > Storage info's Cached Partitions doesn't consider the r

[jira] [Commented] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364175#comment-16364175 ] Apache Spark commented on SPARK-18844: -- User 'sandecho' has created a pull request f

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-14 Thread Mitchell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364190#comment-16364190 ] Mitchell commented on SPARK-23420: -- Greetings Marco, thanks for the response. Without pu

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364251#comment-16364251 ] Sean Owen commented on SPARK-23420: --- I think that logic is mostly correct, because thos

[jira] [Commented] (SPARK-21302) history server WebUI show HTTP ERROR 500

2018-02-14 Thread bharath kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364257#comment-16364257 ] bharath kumar commented on SPARK-21302: --- I saw this issue as well with RM Webui, wh

[jira] [Updated] (SPARK-21302) history server WebUI show HTTP ERROR 500

2018-02-14 Thread bharath kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] bharath kumar updated SPARK-21302: -- Attachment: nullpointer.PNG > history server WebUI show HTTP ERROR 500 > --

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-14 Thread Mitchell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364284#comment-16364284 ] Mitchell commented on SPARK-23420: -- Sean, I'm a little confused by your response. From w

[jira] [Created] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Igor Berman (JIRA)
Igor Berman created SPARK-23423: --- Summary: Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors Key: SPARK-23423 URL: https://issues.apache.org/jira/browse/SPARK-234

[jira] [Commented] (SPARK-23420) Datasource loading not handling paths with regex chars.

2018-02-14 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364324#comment-16364324 ] Sean Owen commented on SPARK-23420: --- Hm. That's a different problem then. Those are URI

[jira] [Updated] (SPARK-23380) Make toPandas fallback to non-Arrow optimization if possible

2018-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23380: - Summary: Make toPandas fallback to non-Arrow optimization if possible (was: Make toPandas fall b

[jira] [Updated] (SPARK-23401) Improve test cases for all supported types and unsupported types

2018-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-23401: - Issue Type: Sub-task (was: Test) Parent: SPARK-22216 > Improve test cases for all suppor

[jira] [Updated] (SPARK-23377) Bucketizer with multiple columns persistence bug

2018-02-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-23377: -- Priority: Blocker (was: Critical) > Bucketizer with multiple columns persistence bug >

[jira] [Commented] (SPARK-23234) ML python test failure due to default outputCol

2018-02-14 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364510#comment-16364510 ] Joseph K. Bradley commented on SPARK-23234: --- Is this still a blocker now that [

[jira] [Created] (SPARK-23424) Add codegenStageId in comment

2018-02-14 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-23424: Summary: Add codegenStageId in comment Key: SPARK-23424 URL: https://issues.apache.org/jira/browse/SPARK-23424 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Sujith (JIRA)
Sujith created SPARK-23425: -- Summary: load data for hdfs file path with wild card usage is not working properly Key: SPARK-23425 URL: https://issues.apache.org/jira/browse/SPARK-23425 Project: Spark

[jira] [Updated] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith updated SPARK-23425: --- Attachment: wildcard_issue.PNG > load data for hdfs file path with wild card usage is not working properly >

[jira] [Commented] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364538#comment-16364538 ] Sujith commented on SPARK-23425: I am working towards resolving this bug, please let me k

[jira] [Assigned] (SPARK-23426) Use `hive` ORC implementation for Spark 2.3.0

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23426: Assignee: (was: Apache Spark) > Use `hive` ORC implementation for Spark 2.3.0 > --

[jira] [Commented] (SPARK-23426) Use `hive` ORC implementation for Spark 2.3.0

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364563#comment-16364563 ] Apache Spark commented on SPARK-23426: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Created] (SPARK-23426) Use `hive` ORC implementation for Spark 2.3.0

2018-02-14 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-23426: - Summary: Use `hive` ORC implementation for Spark 2.3.0 Key: SPARK-23426 URL: https://issues.apache.org/jira/browse/SPARK-23426 Project: Spark Issue Type: B

[jira] [Assigned] (SPARK-23426) Use `hive` ORC implementation for Spark 2.3.0

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23426: Assignee: Apache Spark > Use `hive` ORC implementation for Spark 2.3.0 > -

[jira] [Updated] (SPARK-23426) Use `hive` ORC impl and disable PPD for Spark 2.3.0

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23426: -- Summary: Use `hive` ORC impl and disable PPD for Spark 2.3.0 (was: Use `hive` ORC implementati

[jira] [Updated] (SPARK-23426) Use `hive` ORC implementation for Spark 2.3.0

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23426: -- Issue Type: Task (was: Bug) > Use `hive` ORC implementation for Spark 2.3.0 >

[jira] [Assigned] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23425: Assignee: (was: Apache Spark) > load data for hdfs file path with wild card usage is n

[jira] [Assigned] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23425: Assignee: Apache Spark > load data for hdfs file path with wild card usage is not working

[jira] [Commented] (SPARK-23425) load data for hdfs file path with wild card usage is not working properly

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364605#comment-16364605 ] Apache Spark commented on SPARK-23425: -- User 'sujith71955' has created a pull reques

[jira] [Assigned] (SPARK-23424) Add codegenStageId in comment

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23424: Assignee: Apache Spark > Add codegenStageId in comment > - > >

[jira] [Assigned] (SPARK-23424) Add codegenStageId in comment

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23424: Assignee: (was: Apache Spark) > Add codegenStageId in comment > --

[jira] [Commented] (SPARK-23424) Add codegenStageId in comment

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364653#comment-16364653 ] Apache Spark commented on SPARK-23424: -- User 'kiszk' has created a pull request for

[jira] [Commented] (SPARK-23234) ML python test failure due to default outputCol

2018-02-14 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364680#comment-16364680 ] Marco Gaido commented on SPARK-23234: - [~josephkb] maybe it is not a blocker, but sin

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364682#comment-16364682 ] Kazuaki Ishizaki commented on SPARK-23415: -- I am working for this. > BufferHold

[jira] [Commented] (SPARK-23415) BufferHolderSparkSubmitSuite is flaky

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364686#comment-16364686 ] Dongjoon Hyun commented on SPARK-23415: --- Thank you so much, [~kiszk]! > BufferHold

[jira] [Created] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM in the driver

2018-02-14 Thread Dhiraj (JIRA)
Dhiraj created SPARK-23427: -- Summary: spark.sql.autoBroadcastJoinThreshold causing OOM in the driver Key: SPARK-23427 URL: https://issues.apache.org/jira/browse/SPARK-23427 Project: Spark Issue T

[jira] [Updated] (SPARK-23427) spark.sql.autoBroadcastJoinThreshold causing OOM in the driver

2018-02-14 Thread Dhiraj (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dhiraj updated SPARK-23427: --- Description: We are facing issue around value of spark.sql.autoBroadcastJoinThreshold. With spark.sql.autoBr

[jira] [Reopened] (SPARK-21783) Turn on ORC filter push-down by default

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-21783: --- > Turn on ORC filter push-down by default > --- > >

[jira] [Commented] (SPARK-21783) Turn on ORC filter push-down by default

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364723#comment-16364723 ] Dongjoon Hyun commented on SPARK-21783: --- Unfortunately, this is reopened by SPARK-2

[jira] [Updated] (SPARK-21783) Turn on ORC filter push-down by default

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21783: -- Fix Version/s: (was: 2.3.0) > Turn on ORC filter push-down by default > ---

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364788#comment-16364788 ] Bruce Robbins commented on SPARK-23410: --- I am probably misunderstanding the issue,

[jira] [Created] (SPARK-23428) Revert

2018-02-14 Thread Xiao Li (JIRA)
Xiao Li created SPARK-23428: --- Summary: Revert Key: SPARK-23428 URL: https://issues.apache.org/jira/browse/SPARK-23428 Project: Spark Issue Type: Bug Components: SQL Affects Versions:

[jira] [Updated] (SPARK-23428) Revert

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23428: Description: {noformat} test("invalid json with leading nulls - from dataset") { import testImplicits

[jira] [Updated] (SPARK-23428) Revert [SPARK-23094] Fix invalid character handling in JsonDataSource

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23428: Summary: Revert [SPARK-23094] Fix invalid character handling in JsonDataSource (was: Revert ) > Revert [

[jira] [Updated] (SPARK-20901) Feature parity for ORC with Parquet

2018-02-14 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-20901: -- Affects Version/s: 2.3.0 > Feature parity for ORC with Parquet > --

[jira] [Updated] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Igor Berman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Berman updated SPARK-23423: Description: Hi I've noticed rather strange behavior of MesosCoarseGrainedSchedulerBackend when r

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364849#comment-16364849 ] Maxim Gekk commented on SPARK-23410: [~bersprockets] does your json contain BOM in th

[jira] [Commented] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos commented on SPARK-23423: - Hi [~igor.berman]. Looking at

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/14/18 10:17 PM:

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/14/18 10:17 PM:

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364866#comment-16364866 ] Bruce Robbins commented on SPARK-23410: --- [~maxgekk] My simple test input of [{"fi

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364849#comment-16364849 ] Maxim Gekk edited comment on SPARK-23410 at 2/14/18 10:20 PM: -

[jira] [Assigned] (SPARK-23368) Avoid unnecessary Exchange or Sort after projection

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23368: Assignee: Apache Spark > Avoid unnecessary Exchange or Sort after projection > ---

[jira] [Commented] (SPARK-23368) Avoid unnecessary Exchange or Sort after projection

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364867#comment-16364867 ] Apache Spark commented on SPARK-23368: -- User 'maryannxue' has created a pull request

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364866#comment-16364866 ] Bruce Robbins edited comment on SPARK-23410 at 2/14/18 10:21 PM: --

[jira] [Assigned] (SPARK-23368) Avoid unnecessary Exchange or Sort after projection

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23368: Assignee: (was: Apache Spark) > Avoid unnecessary Exchange or Sort after projection >

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/14/18 10:23 PM:

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/14/18 10:26 PM:

[jira] [Comment Edited] (SPARK-23423) Application declines any offers when killed+active executors rich spark.dynamicAllocation.maxExecutors

2018-02-14 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364861#comment-16364861 ] Stavros Kontopoulos edited comment on SPARK-23423 at 2/14/18 10:27 PM:

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-23410: --- Attachment: utf16WithBOM.json > Unable to read jsons in charset different from UTF-8 > --

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364875#comment-16364875 ] Maxim Gekk commented on SPARK-23410: I attached the file on which I tested on 2.2.1:

[jira] [Resolved] (SPARK-23406) Stream-stream self joins does not work

2018-02-14 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-23406. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20598 [https://g

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Maxim Gekk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364889#comment-16364889 ] Maxim Gekk commented on SPARK-23410: I am working on a fix, just in case > Unable to

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364916#comment-16364916 ] Bruce Robbins commented on SPARK-23410: --- On Spark 2.2.1, I got the same result as y

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23410: Component/s: (was: Input/Output) SQL > Unable to read jsons in charset different from

[jira] [Resolved] (SPARK-23428) Revert [SPARK-23094] Fix invalid character handling in JsonDataSource

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23428. - Resolution: Invalid > Revert [SPARK-23094] Fix invalid character handling in JsonDataSource > --

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364929#comment-16364929 ] Bruce Robbins commented on SPARK-23410: --- bq. I am working on a fix, just in case O

[jira] [Comment Edited] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364929#comment-16364929 ] Bruce Robbins edited comment on SPARK-23410 at 2/14/18 11:17 PM: --

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23410: Target Version/s: 2.3.0 > Unable to read jsons in charset different from UTF-8 > --

[jira] [Updated] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23410: Priority: Blocker (was: Major) > Unable to read jsons in charset different from UTF-8 > --

[jira] [Commented] (SPARK-23410) Unable to read jsons in charset different from UTF-8

2018-02-14 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364954#comment-16364954 ] Xiao Li commented on SPARK-23410: - This is a regression we need to resolve in Spark 2.3.

[jira] [Commented] (SPARK-23094) Json Readers choose wrong encoding when bad records are present and fail

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364956#comment-16364956 ] Apache Spark commented on SPARK-23094: -- User 'gatorsmile' has created a pull request

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-02-14 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Issue Type: Umbrella (was: Improvement) > Additional Memory Tuning Metrics > -

[jira] [Created] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2018-02-14 Thread Edwina Lu (JIRA)
Edwina Lu created SPARK-23429: - Summary: Add executor memory metrics to heartbeat and expose in executors REST API Key: SPARK-23429 URL: https://issues.apache.org/jira/browse/SPARK-23429 Project: Spark

[jira] [Commented] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2018-02-14 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364967#comment-16364967 ] Edwina Lu commented on SPARK-23429: --- Subtask of SPARK-23206 > Add executor memory metr

[jira] [Created] (SPARK-23430) Cannot sort "Executor ID" or "Host" columns in the task table

2018-02-14 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-23430: Summary: Cannot sort "Executor ID" or "Host" columns in the task table Key: SPARK-23430 URL: https://issues.apache.org/jira/browse/SPARK-23430 Project: Spark

[jira] [Updated] (SPARK-23429) Add executor memory metrics to heartbeat and expose in executors REST API

2018-02-14 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23429: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-23206 > Add executor memory metrics to h

[jira] [Created] (SPARK-23431) Expose the new executor memory metrics at the stage level

2018-02-14 Thread Edwina Lu (JIRA)
Edwina Lu created SPARK-23431: - Summary: Expose the new executor memory metrics at the stage level Key: SPARK-23431 URL: https://issues.apache.org/jira/browse/SPARK-23431 Project: Spark Issue Typ

[jira] [Created] (SPARK-23432) Expose executor memory metrics in the web UI for executors and stages

2018-02-14 Thread Edwina Lu (JIRA)
Edwina Lu created SPARK-23432: - Summary: Expose executor memory metrics in the web UI for executors and stages Key: SPARK-23432 URL: https://issues.apache.org/jira/browse/SPARK-23432 Project: Spark

[jira] [Assigned] (SPARK-23430) Cannot sort "Executor ID" or "Host" columns in the task table

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23430: Assignee: Apache Spark (was: Shixiong Zhu) > Cannot sort "Executor ID" or "Host" columns

[jira] [Assigned] (SPARK-23430) Cannot sort "Executor ID" or "Host" columns in the task table

2018-02-14 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23430: Assignee: Shixiong Zhu (was: Apache Spark) > Cannot sort "Executor ID" or "Host" columns

  1   2   >