[jira] [Commented] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133947#comment-16133947 ] Yan Facai (颜发才) commented on SPARK-21748: - There seems to be something wrong with CI. The

[jira] [Comment Edited] (SPARK-21748) Migrate the implementation of HashingTF from MLlib to ML

[ https://issues.apache.org/jira/browse/SPARK-21748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133947#comment-16133947 ] Yan Facai (颜发才) edited comment on SPARK-21748 at 8/19/17 4:43 AM: -- There

[jira] [Commented] (SPARK-21720) Filter predicate with many conditions throw stackoverflow error

[ https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133878#comment-16133878 ] poplav commented on SPARK-21720: [~kiszk], Thank you for the update. I am following your PR for this at

[jira] [Resolved] (SPARK-21566) Python method for summary

[ https://issues.apache.org/jira/browse/SPARK-21566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-21566. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18762

[jira] [Assigned] (SPARK-21566) Python method for summary

[ https://issues.apache.org/jira/browse/SPARK-21566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-21566: --- Assignee: Andrew Ray > Python method for summary > - > >

[jira] [Created] (SPARK-21788) Handle more exceptions when stopping a streaming query

Shixiong Zhu created SPARK-21788: Summary: Handle more exceptions when stopping a streaming query Key: SPARK-21788 URL: https://issues.apache.org/jira/browse/SPARK-21788 Project: Spark Issue

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:29 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:19 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:13 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:12 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:12 PM:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos edited comment on SPARK-21752 at 8/18/17 8:11 PM:

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133565#comment-16133565 ] Stavros Kontopoulos commented on SPARK-21752: - [~jsnowacki] What I am doing is not manual its

[jira] [Commented] (SPARK-21787) Support for pushing down filters for date types in ORC

[ https://issues.apache.org/jira/browse/SPARK-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133522#comment-16133522 ] Dongjoon Hyun commented on SPARK-21787: --- I'll make a PR for this. > Support for pushing down

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133493#comment-16133493 ] Sean Owen commented on SPARK-21770: --- Yeah good point, the better question is perhaps what model

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133483#comment-16133483 ] Joseph K. Bradley commented on SPARK-21770: --- I vaguely recall discussing this before but forget

[jira] [Commented] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

[ https://issues.apache.org/jira/browse/SPARK-21784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133381#comment-16133381 ] Xiao Li commented on SPARK-21784: - https://github.com/apache/spark/pull/18994 > Add ALTER TABLE ADD

[jira] [Commented] (SPARK-21787) Support for pushing down filters for date types in ORC

[ https://issues.apache.org/jira/browse/SPARK-21787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133378#comment-16133378 ] Dongjoon Hyun commented on SPARK-21787: --- Thank you for reporting! I linked this to SPARK-20901,

[jira] [Commented] (SPARK-21772) HiveException unable to move results from srcf to destf in InsertIntoHiveTable

[ https://issues.apache.org/jira/browse/SPARK-21772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133271#comment-16133271 ] Xiao Li commented on SPARK-21772: - https://github.com/apache/spark/pull/18985 > HiveException unable to

[jira] [Resolved] (SPARK-21213) Support collecting partition-level statistics: rowCount and sizeInBytes

[ https://issues.apache.org/jira/browse/SPARK-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21213. - Resolution: Fixed Fix Version/s: 2.3.0 > Support collecting partition-level statistics: rowCount

[jira] [Assigned] (SPARK-21213) Support collecting partition-level statistics: rowCount and sizeInBytes

[ https://issues.apache.org/jira/browse/SPARK-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21213: --- Assignee: Maria > Support collecting partition-level statistics: rowCount and sizeInBytes >

[jira] [Commented] (SPARK-17423) Support IGNORE NULLS option in Window functions

[ https://issues.apache.org/jira/browse/SPARK-17423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133095#comment-16133095 ] Gaurav Shah commented on SPARK-17423: - is there a way to do second non-null value ? changing the

[jira] [Commented] (SPARK-21677) json_tuple throws NullPointException when column is null as string type.

[ https://issues.apache.org/jira/browse/SPARK-21677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133091#comment-16133091 ] Hyukjin Kwon commented on SPARK-21677: -- Issue resolved by pull request 18930

[jira] [Resolved] (SPARK-21778) Simpler Dataset.sample API in Scala / Java

[ https://issues.apache.org/jira/browse/SPARK-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21778. -- Resolution: Resolved Assignee: Reynold Xin Fix Version/s: 2.3.0 Issue resolved

[jira] [Created] (SPARK-21787) Support for pushing down filters for date types in ORC

Stefan de Koning created SPARK-21787: Summary: Support for pushing down filters for date types in ORC Key: SPARK-21787 URL: https://issues.apache.org/jira/browse/SPARK-21787 Project: Spark

[jira] [Resolved] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

[ https://issues.apache.org/jira/browse/SPARK-21768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21768. --- Resolution: Duplicate > spark.csv.read Empty String Parsed as NULL when nullValue is Set >

[jira] [Commented] (SPARK-21768) spark.csv.read Empty String Parsed as NULL when nullValue is Set

[ https://issues.apache.org/jira/browse/SPARK-21768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132191#comment-16132191 ] Marco Gaido commented on SPARK-21768: - This is a duplicate of SPARK-17916. > spark.csv.read Empty

[jira] [Resolved] (SPARK-4981) Add a streaming singular value decomposition

[ https://issues.apache.org/jira/browse/SPARK-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4981. -- Resolution: Won't Fix > Add a streaming singular value decomposition >

[jira] [Commented] (SPARK-4981) Add a streaming singular value decomposition

[ https://issues.apache.org/jira/browse/SPARK-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16132118#comment-16132118 ] Nick Pentreath commented on SPARK-4981: --- Hey folks, as interesting as this would be, I think it's

[jira] [Created] (SPARK-21786) The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)

Jinhua Fu created SPARK-21786: - Summary: The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s) Key: SPARK-21786 URL:

[jira] [Comment Edited] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131945#comment-16131945 ] Jakub Nowacki edited comment on SPARK-21752 at 8/18/17 9:11 AM: [~skonto]

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131945#comment-16131945 ] Jakub Nowacki commented on SPARK-21752: --- [~skonto] What you are doing is in fact starting manually

[jira] [Commented] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131921#comment-16131921 ] StanZhai commented on SPARK-21774: -- I've opened a PR (https://github.com/apache/spark/pull/18986) for

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131917#comment-16131917 ] Sean Owen commented on SPARK-21770: --- All 0 outputs mean the model says all classes are impossible. As

[jira] [Updated] (SPARK-21785) Support create table from a file schema

[ https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-21785: - Issue Type: Wish (was: New Feature) > Support create table from a file schema >

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:41 AM: --- This is introduced

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:41 AM: --- This is introduced

[jira] [Comment Edited] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131908#comment-16131908 ] Feng Zhu edited comment on SPARK-21774 at 8/18/17 8:39 AM: --- This is introduced

[jira] [Created] (SPARK-21785) Support create table from a file schema

liupengcheng created SPARK-21785: Summary: Support create table from a file schema Key: SPARK-21785 URL: https://issues.apache.org/jira/browse/SPARK-21785 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-21774) The rule PromoteStrings cast string to a wrong data type

[ https://issues.apache.org/jira/browse/SPARK-21774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131908#comment-16131908 ] Feng Zhu commented on SPARK-21774: -- This is introduced by the PR-15880

[jira] [Commented] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

[ https://issues.apache.org/jira/browse/SPARK-21784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131900#comment-16131900 ] Suresh Thalamati commented on SPARK-21784: -- I am working on implementation of this task. > Add

[jira] [Created] (SPARK-21784) Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys

Suresh Thalamati created SPARK-21784: Summary: Add ALTER TABLE ADD CONSTRANT DDL to support defining primary key and foreign keys Key: SPARK-21784 URL: https://issues.apache.org/jira/browse/SPARK-21784

[jira] [Commented] (SPARK-21763) InferSchema option does not infer the correct schema (timestamp) from xlsx file.

[ https://issues.apache.org/jira/browse/SPARK-21763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131890#comment-16131890 ] Liang-Chi Hsieh commented on SPARK-21763: - I'm afraid that the inferSchema is done by the library

[jira] [Commented] (SPARK-19109) ORC metadata section can sometimes exceed protobuf message size limit

[ https://issues.apache.org/jira/browse/SPARK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131877#comment-16131877 ] Dongjoon Hyun commented on SPARK-19109: --- Hi, [~nseggert] and [~wangchao2017]. Could you give us a

[jira] [Comment Edited] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131868#comment-16131868 ] Siddharth Murching edited comment on SPARK-21770 at 8/18/17 7:48 AM: -

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131868#comment-16131868 ] Siddharth Murching commented on SPARK-21770: Good question: * Predictions on all-zero input

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131865#comment-16131865 ] Sergey Serebryakov commented on SPARK-21782: I played with Scala's `hashing.byteswap32()` and

[jira] [Updated] (SPARK-21783) Turn on ORC filter push-down by default

[ https://issues.apache.org/jira/browse/SPARK-21783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21783: -- Description: Like Parquet (SPARK-9207), it would be great to turn on ORC option, too. This

[jira] [Created] (SPARK-21783) Turn on ORC filter push-down by default

Dongjoon Hyun created SPARK-21783: - Summary: Turn on ORC filter push-down by default Key: SPARK-21783 URL: https://issues.apache.org/jira/browse/SPARK-21783 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20984) Reading back from ORC format gives error on big endian systems.

[ https://issues.apache.org/jira/browse/SPARK-20984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131837#comment-16131837 ] Dongjoon Hyun commented on SPARK-20984: --- Hi, All. I added this as a blocker of SPARK-20984. >

[jira] [Commented] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131828#comment-16131828 ] Kent Yao commented on SPARK-21771: -- https://github.com/apache/spark/pull/18983 git webhook doesn't work,

[jira] [Comment Edited] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131828#comment-16131828 ] Kent Yao edited comment on SPARK-21771 at 8/18/17 7:05 AM: ---

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131818#comment-16131818 ] Sean Owen commented on SPARK-21782: --- I think scrambling the seed a bit is the smallest change that

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131796#comment-16131796 ] cen yuhai commented on SPARK-16188: --- [~xianlongZhang] yes, you are right, I has implemented this

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131794#comment-16131794 ] Sergey Serebryakov commented on SPARK-21782: Your understanding is correct. Either reusing

[jira] [Updated] (SPARK-21771) SparkSQLEnv creates a useless meta hive client

[ https://issues.apache.org/jira/browse/SPARK-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21771: -- Issue Type: Improvement (was: Bug) > SparkSQLEnv creates a useless meta hive client >

[jira] [Commented] (SPARK-21782) Repartition creates skews when numPartitions is a power of 2

[ https://issues.apache.org/jira/browse/SPARK-21782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131784#comment-16131784 ] Sean Owen commented on SPARK-21782: --- Is the problem summary just: with a power of 2 bound, similar

[jira] [Commented] (SPARK-21770) ProbabilisticClassificationModel: Improve normalization of all-zero raw predictions

[ https://issues.apache.org/jira/browse/SPARK-21770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131775#comment-16131775 ] Sean Owen commented on SPARK-21770: --- What's the current behavior for the prediction? I feel like we

[jira] [Commented] (SPARK-16188) Spark sql create a lot of small files

[ https://issues.apache.org/jira/browse/SPARK-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131772#comment-16131772 ] xianlongZhang commented on SPARK-16188: --- cen yuhai,thanks for your advice, but my company's data

[jira] [Resolved] (SPARK-21776) How to use the memory-mapped file on Spark??

[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21776. --- Resolution: Invalid > How to use the memory-mapped file on Spark?? >

[jira] [Updated] (SPARK-21776) How to use the memory-mapped file on Spark??

[ https://issues.apache.org/jira/browse/SPARK-21776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21776: -- Priority: Trivial (was: Blocker) > How to use the memory-mapped file on Spark?? >