[jira] [Commented] (SPARK-21043) Add unionByName API to Dataset

2017-06-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045773#comment-16045773 ] Takeshi Yamamuro commented on SPARK-21043: -- Thank you for ping me! Yea, I'll try > Add

[jira] [Commented] (SPARK-17237) DataFrame fill after pivot causing org.apache.spark.sql.AnalysisException

2017-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051288#comment-16051288 ] Takeshi Yamamuro commented on SPARK-17237: -- I checked the other behaviours and I probably think

[jira] [Updated] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21021: - Issue Type: Improvement (was: Bug) > Reading partitioned parquet does not respect

[jira] [Commented] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048871#comment-16048871 ] Takeshi Yamamuro commented on SPARK-21021: -- This is an expected behaviour, so it is not a bug. I

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-05-07 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000148#comment-16000148 ] Takeshi Yamamuro commented on SPARK-19112: -- I also put the result here: {code} scaleFactor: 4

[jira] [Comment Edited] (SPARK-19112) add codec for ZStandard

2017-05-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000778#comment-16000778 ] Takeshi Yamamuro edited comment on SPARK-19112 at 5/8/17 3:24 PM: -- I

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-05-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000778#comment-16000778 ] Takeshi Yamamuro commented on SPARK-19112: -- I just run spark-sql-perf (scalefacotr=4) on

[jira] [Commented] (SPARK-20998) BroadcastHashJoin producing wrong results

2017-06-06 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039499#comment-16039499 ] Takeshi Yamamuro commented on SPARK-20998: -- How about v2.1? The version has the same issue? >

[jira] [Commented] (SPARK-22033) BufferHolder size checks should account for the specific VM array size limitations

2017-09-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170109#comment-16170109 ] Takeshi Yamamuro commented on SPARK-22033: -- just a head-up: this is probably related to the 2G

[jira] [Comment Edited] (SPARK-22033) BufferHolder size checks should account for the specific VM array size limitations

2017-09-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170109#comment-16170109 ] Takeshi Yamamuro edited comment on SPARK-22033 at 9/18/17 2:57 PM: ---

[jira] [Commented] (SPARK-21998) SortMergeJoinExec should calculate its outputOrdering independent of its children's outputOrdering

2017-09-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165403#comment-16165403 ] Takeshi Yamamuro commented on SPARK-21998: -- I think the orders depend on their children, e.g.

[jira] [Commented] (SPARK-22000) org.codehaus.commons.compiler.CompileException: toString method is not declared

2017-09-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16165667#comment-16165667 ] Takeshi Yamamuro commented on SPARK-22000: -- What's the query? >

[jira] [Commented] (SPARK-12823) Cannot create UDF with StructType input

2017-09-19 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171870#comment-16171870 ] Takeshi Yamamuro commented on SPARK-12823: -- what do u think when is the good timing to take on

[jira] [Commented] (SPARK-12823) Cannot create UDF with StructType input

2017-09-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171019#comment-16171019 ] Takeshi Yamamuro commented on SPARK-12823: -- As wenchen suggested, why don't you use `Row` as an

[jira] [Commented] (SPARK-21785) Support create table from a file schema

2017-10-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192433#comment-16192433 ] Takeshi Yamamuro commented on SPARK-21785: -- {code} scala> sql("""CREATE TABLE customer USING

[jira] [Commented] (SPARK-21785) Support create table from a file schema

2017-10-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192362#comment-16192362 ] Takeshi Yamamuro commented on SPARK-21785: -- `CREATE TABLE xxx USING parquet OPTIONS (path

[jira] [Commented] (SPARK-22204) Explain output for SQL with commands shows no optimization

2017-10-04 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192454#comment-16192454 ] Takeshi Yamamuro commented on SPARK-22204: -- good catch. The optimization is internally applied

[jira] [Commented] (SPARK-22176) Dataset.show(Int.MaxValue) hits integer overflows

2017-10-02 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189272#comment-16189272 ] Takeshi Yamamuro commented on SPARK-22176: -- [~smilegator] close this? thanks. >

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-10-05 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194170#comment-16194170 ] Takeshi Yamamuro commented on SPARK-22211: -- Probably, the suggested solution does not work when

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203095#comment-16203095 ] Takeshi Yamamuro commented on SPARK-22271: -- More->Attach Files? btw, text file (csv or

[jira] [Commented] (SPARK-22271) Describe results in "null" for the value of "mean" of a numeric variable

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203080#comment-16203080 ] Takeshi Yamamuro commented on SPARK-22271: -- You need to give us the data and the schema, too? >

[jira] [Commented] (SPARK-22270) Renaming DF column breaks sparkPlan.outputOrdering

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16203081#comment-16203081 ] Takeshi Yamamuro commented on SPARK-22270: -- Probably, this is duplicate to

[jira] [Commented] (SPARK-22211) LimitPushDown optimization for FullOuterJoin generates wrong results

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202863#comment-16202863 ] Takeshi Yamamuro commented on SPARK-22211: -- Aha, I misunderstood. But, I think the case 3 is not

[jira] [Commented] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202871#comment-16202871 ] Takeshi Yamamuro commented on SPARK-22266: -- This is not a bug, so I changed to improvement. >

[jira] [Updated] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-22266: - Issue Type: Improvement (was: Bug) > The same aggregate function was evaluated multiple

[jira] [Comment Edited] (SPARK-22266) The same aggregate function was evaluated multiple times

2017-10-12 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202871#comment-16202871 ] Takeshi Yamamuro edited comment on SPARK-22266 at 10/13/17 12:39 AM: -

[jira] [Commented] (SPARK-21931) add LNNVL function

2017-09-06 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154898#comment-16154898 ] Takeshi Yamamuro commented on SPARK-21931: -- Why we need to support oracle-specific functions as

[jira] [Updated] (SPARK-21931) add LNNVL function

2017-09-06 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21931: - Affects Version/s: (was: 2.3.0) > add LNNVL function > -- > >

[jira] [Created] (SPARK-21973) Add a new option to filter queries to run in TPCDSQueryBenchmark

2017-09-11 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-21973: Summary: Add a new option to filter queries to run in TPCDSQueryBenchmark Key: SPARK-21973 URL: https://issues.apache.org/jira/browse/SPARK-21973 Project:

[jira] [Commented] (SPARK-18591) Replace hash-based aggregates with sort-based ones if inputs already sorted

2017-09-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160388#comment-16160388 ] Takeshi Yamamuro commented on SPARK-18591: -- I just kindly give a head-up for the discussion on

[jira] [Created] (SPARK-22122) Respect WITH clauses to count input rows in TPCDSQueryBenchmark

2017-09-25 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-22122: Summary: Respect WITH clauses to count input rows in TPCDSQueryBenchmark Key: SPARK-22122 URL: https://issues.apache.org/jira/browse/SPARK-22122 Project:

[jira] [Closed] (SPARK-21971) Too many open files in Spark due to concurrent files being opened

2017-09-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-21971. Resolution: Not A Problem > Too many open files in Spark due to concurrent files being

[jira] [Created] (SPARK-22176) Dataset.show(Int.MaxValue) hits integer overflows

2017-09-30 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-22176: Summary: Dataset.show(Int.MaxValue) hits integer overflows Key: SPARK-22176 URL: https://issues.apache.org/jira/browse/SPARK-22176 Project: Spark

[jira] [Updated] (SPARK-22152) Add Dataset flatten function

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-22152: - Issue Type: New Feature (was: Wish) > Add Dataset flatten function >

[jira] [Updated] (SPARK-22152) Add Dataset flatten function

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-22152: - Component/s: (was: Spark Core) SQL > Add Dataset flatten function >

[jira] [Commented] (SPARK-22152) Add Dataset flatten function

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183570#comment-16183570 ] Takeshi Yamamuro commented on SPARK-22152: -- `Dataset[Option[T]]` is used in a natural usecase?

[jira] [Comment Edited] (SPARK-22152) Add Dataset flatten function

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183570#comment-16183570 ] Takeshi Yamamuro edited comment on SPARK-22152 at 9/28/17 2:14 AM: ---

[jira] [Closed] (SPARK-22149) spark.shuffle.memoryFraction (deprecated) in spark 2

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-22149. Resolution: Invalid > spark.shuffle.memoryFraction (deprecated) in spark 2 >

[jira] [Commented] (SPARK-22149) spark.shuffle.memoryFraction (deprecated) in spark 2

2017-09-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183575#comment-16183575 ] Takeshi Yamamuro commented on SPARK-22149: -- I think you should first ask in the spark mailing

[jira] [Commented] (SPARK-22152) Add Dataset flatten function

2017-09-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183828#comment-16183828 ] Takeshi Yamamuro commented on SPARK-22152: -- cc: [~smilegator][~cloud_fan] > Add Dataset flatten

[jira] [Comment Edited] (SPARK-22152) Add Dataset flatten function

2017-09-28 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183828#comment-16183828 ] Takeshi Yamamuro edited comment on SPARK-22152 at 9/28/17 8:04 AM: --- cc:

[jira] [Commented] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

2017-08-24 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141234#comment-16141234 ] Takeshi Yamamuro commented on SPARK-21828: -- You can't set target version or something and these

[jira] [Updated] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

2017-08-24 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21828: - Component/s: (was: ML) >

[jira] [Updated] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

2017-08-24 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21828: - Flags: (was: Important) >

[jira] [Updated] (SPARK-21828) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB...again

2017-08-24 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21828: - Target Version/s: (was: 2.1.0, 2.2.0) >

[jira] [Closed] (SPARK-17414) Set type is not supported for creating data frames

2017-08-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-17414. Resolution: Fixed Fix Version/s: 2.3.0 > Set type is not supported for creating

[jira] [Created] (SPARK-21870) Split codegen'd aggregation code into small functions for the HotSpot

2017-08-29 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-21870: Summary: Split codegen'd aggregation code into small functions for the HotSpot Key: SPARK-21870 URL: https://issues.apache.org/jira/browse/SPARK-21870

[jira] [Updated] (SPARK-21870) Split codegen'd aggregation code into small functions for the HotSpot

2017-08-29 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21870: - Description: In SPARK-21603, we got performance regression if the HotSpot didn't compile

[jira] [Updated] (SPARK-21870) Split codegen'd aggregation code into small functions for the HotSpot

2017-08-29 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-21870: - Description: In SPARK-21603, we got performance regression if the HotSpot didn't compile

[jira] [Created] (SPARK-21871) Check actual bytecode size when compiling generated code

2017-08-29 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-21871: Summary: Check actual bytecode size when compiling generated code Key: SPARK-21871 URL: https://issues.apache.org/jira/browse/SPARK-21871 Project: Spark

[jira] [Closed] (SPARK-19426) Add support for custom coalescers on Data

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro closed SPARK-19426. Resolution: Later > Add support for custom coalescers on Data >

[jira] [Commented] (SPARK-19426) Add support for custom coalescers on Data

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190578#comment-16190578 ] Takeshi Yamamuro commented on SPARK-19426: -- I'll close for now cuz the priority is not much

[jira] [Commented] (SPARK-22193) SortMergeJoinExec: typo correction

2017-10-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190580#comment-16190580 ] Takeshi Yamamuro commented on SPARK-22193: -- You probably don't file a jira for trivial fixes. >

[jira] [Commented] (SPARK-22248) spark marks all columns as null when its unable to parse single column

2017-10-11 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201309#comment-16201309 ] Takeshi Yamamuro commented on SPARK-22248: -- Once failed, I feel it is difficult to recover the

[jira] [Commented] (SPARK-22248) spark marks all columns as null when its unable to parse single column

2017-10-11 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201438#comment-16201438 ] Takeshi Yamamuro commented on SPARK-22248: -- yea, ok. Probably, you need to support both modes:

[jira] [Commented] (SPARK-22223) ObjectHashAggregate introduces unnecessary shuffle

2017-10-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199597#comment-16199597 ] Takeshi Yamamuro commented on SPARK-3: -- The hash-based aggregate implementation requires the

[jira] [Commented] (SPARK-22223) ObjectHashAggregate introduces unnecessary shuffle

2017-10-11 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199837#comment-16199837 ] Takeshi Yamamuro commented on SPARK-3: -- Probably, this ticket is related to

[jira] [Commented] (SPARK-22825) Incorrect results of Casting Array to String

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296304#comment-16296304 ] Takeshi Yamamuro commented on SPARK-22825: -- [~smilegator] you're taking on this now? >

[jira] [Commented] (SPARK-22825) Incorrect results of Casting Array to String

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296308#comment-16296308 ] Takeshi Yamamuro commented on SPARK-22825: -- ok > Incorrect results of Casting Array to String >

[jira] [Created] (SPARK-22800) Add a SSB query suite

2017-12-15 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-22800: Summary: Add a SSB query suite Key: SPARK-22800 URL: https://issues.apache.org/jira/browse/SPARK-22800 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-22771) SQL concat for binary

2017-12-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290174#comment-16290174 ] Takeshi Yamamuro commented on SPARK-22771: -- ok, I'll take > SQL concat for binary >

[jira] [Commented] (SPARK-22771) SQL concat for binary

2017-12-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290101#comment-16290101 ] Takeshi Yamamuro commented on SPARK-22771: -- [~smilegator] It's worth fixing this? > SQL concat

[jira] [Commented] (SPARK-22771) SQL concat for binary

2017-12-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289539#comment-16289539 ] Takeshi Yamamuro commented on SPARK-22771: -- fyi: postgresql has the behaiouvr as you said;

[jira] [Issue Comment Deleted] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-17647: - Comment: was deleted (was: It seems the master still handle ''%\\%' as `(?s).*\Q%\E`

[jira] [Commented] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295992#comment-16295992 ] Takeshi Yamamuro commented on SPARK-17647: -- It seems the master still handle ''%\\%' as

[jira] [Commented] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295976#comment-16295976 ] Takeshi Yamamuro commented on SPARK-17647: -- I'm looking into the code and I'll make a follow-up

[jira] [Comment Edited] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296234#comment-16296234 ] Takeshi Yamamuro edited comment on SPARK-17647 at 12/19/17 5:29 AM:

[jira] [Commented] (SPARK-17647) SQL LIKE does not handle backslashes correctly

2017-12-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296234#comment-16296234 ] Takeshi Yamamuro commented on SPARK-17647: -- Probably, is it okay to set

[jira] [Created] (SPARK-22553) Drop FROM in nonReserved

2017-11-18 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-22553: Summary: Drop FROM in nonReserved Key: SPARK-22553 URL: https://issues.apache.org/jira/browse/SPARK-22553 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22553) Drop FROM in nonReserved

2017-11-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258283#comment-16258283 ] Takeshi Yamamuro commented on SPARK-22553: -- cc: [~hvanhovell] [~smilegator] > Drop FROM in

[jira] [Commented] (SPARK-23771) Uneven Rowgroup size after repartition

2018-05-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474061#comment-16474061 ] Takeshi Yamamuro commented on SPARK-23771: -- I think this has been already fixed in newer Sparks,

[jira] [Created] (SPARK-24204) Verify a write schema in OrcFileFormat

2018-05-07 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24204: Summary: Verify a write schema in OrcFileFormat Key: SPARK-24204 URL: https://issues.apache.org/jira/browse/SPARK-24204 Project: Spark Issue Type:

[jira] [Commented] (SPARK-24204) Verify a write schema in OrcFileFormat

2018-05-07 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466742#comment-16466742 ] Takeshi Yamamuro commented on SPARK-24204: -- This fix is like:

[jira] [Created] (SPARK-24206) Improve DataSource benchmark code for read and pushdown

2018-05-07 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24206: Summary: Improve DataSource benchmark code for read and pushdown Key: SPARK-24206 URL: https://issues.apache.org/jira/browse/SPARK-24206 Project: Spark

[jira] [Commented] (SPARK-19512) codegen for compare structs fails

2018-05-09 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468417#comment-16468417 ] Takeshi Yamamuro commented on SPARK-19512: -- can you put a simple query to reproduce here? >

[jira] [Commented] (SPARK-24201) IllegalArgumentException originating from ClosureCleaner in Java 9+

2018-05-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467023#comment-16467023 ] Takeshi Yamamuro commented on SPARK-24201: -- IIUC spark doesn't support java9+ (the doc should be

[jira] [Commented] (SPARK-19512) codegen for compare structs fails

2018-05-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467043#comment-16467043 ] Takeshi Yamamuro commented on SPARK-19512: -- I checked in the released v2.3.0 and the master

[jira] [Commented] (SPARK-21274) Implement EXCEPT ALL and INTERSECT ALL

2018-05-08 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467056#comment-16467056 ] Takeshi Yamamuro commented on SPARK-21274: -- yea, I'm interested in the performance differences

[jira] [Created] (SPARK-24111) Add TPCDS v2.7 (latest) queries in TPCDSQueryBenchmark

2018-04-27 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24111: Summary: Add TPCDS v2.7 (latest) queries in TPCDSQueryBenchmark Key: SPARK-24111 URL: https://issues.apache.org/jira/browse/SPARK-24111 Project: Spark

[jira] [Commented] (SPARK-24109) Remove class SnappyOutputStreamWrapper

2018-04-27 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456000#comment-16456000 ] Takeshi Yamamuro commented on SPARK-24109: -- IMO it'd be better to keep this ticket open because

[jira] [Commented] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2018-05-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469984#comment-16469984 ] Takeshi Yamamuro commented on SPARK-23519: -- I think typical databases can't use duplicate column

[jira] [Commented] (SPARK-24233) union operation on read of dataframe does nor produce correct result

2018-05-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469975#comment-16469975 ] Takeshi Yamamuro commented on SPARK-24233: -- Can you simplify your query? Also, can you put the

[jira] [Commented] (SPARK-24204) Verify a write schema in Json/Orc/ParquetFileFormat

2018-05-10 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471283#comment-16471283 ] Takeshi Yamamuro commented on SPARK-24204: -- ok, I'll do it later. Thanks for the description

[jira] [Updated] (SPARK-24260) Support for multi-statement SQL in SparkSession.sql API

2018-05-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-24260: - Component/s: (was: Spark Core) SQL > Support for multi-statement

[jira] [Commented] (SPARK-24262) Fix typo in UDF error message

2018-05-13 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473687#comment-16473687 ] Takeshi Yamamuro commented on SPARK-24262: -- cc: [~holdenk] Probably, you forgot to close this?

[jira] [Commented] (SPARK-24369) A bug when having multiple distinct aggregations

2018-05-23 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16488411#comment-16488411 ] Takeshi Yamamuro commented on SPARK-24369: -- ok, I will look into this. Thanks! > A bug when

[jira] [Created] (SPARK-24327) Add an option to quote a partition column name in JDBCRelation

2018-05-20 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24327: Summary: Add an option to quote a partition column name in JDBCRelation Key: SPARK-24327 URL: https://issues.apache.org/jira/browse/SPARK-24327 Project:

[jira] [Commented] (SPARK-24328) Fix scala.MatchError in literals.sql.out

2018-05-20 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482171#comment-16482171 ] Takeshi Yamamuro commented on SPARK-24328: -- I fixed in 

[jira] [Created] (SPARK-24328) Fix scala.MatchError in literals.sql.out

2018-05-20 Thread Takeshi Yamamuro (JIRA)
Takeshi Yamamuro created SPARK-24328: Summary: Fix scala.MatchError in literals.sql.out Key: SPARK-24328 URL: https://issues.apache.org/jira/browse/SPARK-24328 Project: Spark Issue

[jira] [Comment Edited] (SPARK-24540) Support for multiple delimiter in Spark CSV read

2018-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro edited comment on SPARK-24540 at 6/16/18 5:28 AM: ---

[jira] [Comment Edited] (SPARK-24540) Support for multiple delimiter in Spark CSV read

2018-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro edited comment on SPARK-24540 at 6/16/18 5:28 AM: ---

[jira] [Commented] (SPARK-24540) Support for multiple delimiter in Spark CSV read

2018-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514684#comment-16514684 ] Takeshi Yamamuro commented on SPARK-24540: -- Probably, this is a restriction of univocity

[jira] [Commented] (SPARK-24498) Add JDK compiler for runtime codegen

2018-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514696#comment-16514696 ] Takeshi Yamamuro commented on SPARK-24498: -- I'm also interested in this, so I'll look into

[jira] [Commented] (SPARK-24463) Add catalyst rule to reorder TypedFilters separated by Filters to reduce serde operations

2018-06-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514733#comment-16514733 ] Takeshi Yamamuro commented on SPARK-24463: -- Can you describe more in the description? > Add

[jira] [Commented] (SPARK-24570) SparkSQL - show schemas/tables in dropdowns of SQL client tools (ie Squirrel SQL, DBVisualizer.etc)

2018-06-15 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514629#comment-16514629 ] Takeshi Yamamuro commented on SPARK-24570: -- Is this a Spark itself issue? > SparkSQL - show

[jira] [Commented] (SPARK-24423) Add a new option `query` for JDBC sources

2018-06-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514735#comment-16514735 ] Takeshi Yamamuro commented on SPARK-24423: -- Are'u still working on this? > Add a new option

[jira] [Resolved] (SPARK-24399) Reused Exchange is used where it should not be

2018-06-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24399. -- Resolution: Fixed Fix Version/s: 2.3.1 > Reused Exchange is used where it

[jira] [Resolved] (SPARK-24407) Spark + Parquet + Snappy: Overall compression ratio loses after spark shuffles data

2018-06-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-24407. -- Resolution: Invalid > Spark + Parquet + Snappy: Overall compression ratio loses after

[jira] [Commented] (SPARK-24407) Spark + Parquet + Snappy: Overall compression ratio loses after spark shuffles data

2018-06-16 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514736#comment-16514736 ] Takeshi Yamamuro commented on SPARK-24407: -- You should ask in the spark-user mailing list. >

[jira] [Updated] (SPARK-24327) Verify and normalize a partition column name based on the JDBC resolved schema

2018-06-14 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-24327: - Description: We need to modify JDBC datasource code to verify and normalize a partition

<    2   3   4   5   6   7   8   9   10   11   >