[jira] [Updated] (SPARK-38666) Missing aggregate filter checks

2022-03-26 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38666: -- Description: h3. Window function in filter {noformat} select sum(a) filter (where

[jira] [Updated] (SPARK-38308) Select of a stream of window expressions fails

2022-03-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38308: -- Affects Version/s: 3.4.0 > Select of a stream of window expressions fails >

[jira] [Commented] (SPARK-38528) NullPointerException when selecting a generator in a Stream of aggregate expressions

2022-03-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17505149#comment-17505149 ] Bruce Robbins commented on SPARK-38528: --- This is a bug in {{ExtractGenerator}} in which an array

[jira] [Created] (SPARK-38528) NullPointerException when selecting a generator in a Stream of aggregate expressions

2022-03-11 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38528: - Summary: NullPointerException when selecting a generator in a Stream of aggregate expressions Key: SPARK-38528 URL: https://issues.apache.org/jira/browse/SPARK-38528

[jira] [Comment Edited] (SPARK-38285) ClassCastException: GenericArrayData cannot be cast to InternalRow

2022-02-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497662#comment-17497662 ] Bruce Robbins edited comment on SPARK-38285 at 2/24/22, 7:19 PM: - I see

[jira] [Commented] (SPARK-38285) ClassCastException: GenericArrayData cannot be cast to InternalRow

2022-02-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497662#comment-17497662 ] Bruce Robbins commented on SPARK-38285: --- I see your point. It appears to be caused by [this

[jira] [Comment Edited] (SPARK-38285) ClassCastException: GenericArrayData cannot be cast to InternalRow

2022-02-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497111#comment-17497111 ] Bruce Robbins edited comment on SPARK-38285 at 2/24/22, 2:01 AM: - Since

[jira] [Commented] (SPARK-38285) ClassCastException: GenericArrayData cannot be cast to InternalRow

2022-02-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497111#comment-17497111 ] Bruce Robbins commented on SPARK-38285: --- Since {{eo.b}} is an array of sttructs, don't you need to

[jira] [Commented] (SPARK-38308) Select of a stream of window expressions fails

2022-02-23 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497035#comment-17497035 ] Bruce Robbins commented on SPARK-38308: --- The cause is similar issue to that of SPARK-38221. The

[jira] [Created] (SPARK-38308) Select of a stream of window expressions fails

2022-02-23 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38308: - Summary: Select of a stream of window expressions fails Key: SPARK-38308 URL: https://issues.apache.org/jira/browse/SPARK-38308 Project: Spark Issue Type:

[jira] [Commented] (SPARK-38221) Group by a stream of complex expressions fails

2022-02-15 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17492869#comment-17492869 ] Bruce Robbins commented on SPARK-38221: --- I think I have an idea what's going on. I will submit a

[jira] [Created] (SPARK-38221) Group by a stream of complex expressions fails

2022-02-15 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38221: - Summary: Group by a stream of complex expressions fails Key: SPARK-38221 URL: https://issues.apache.org/jira/browse/SPARK-38221 Project: Spark Issue Type:

[jira] [Updated] (SPARK-38146) UDAF fails to aggregate TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails to aggregate TIMESTAMP_NTZ column (was: UDAF fails with unsafe row

[jira] [Updated] (SPARK-38146) UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column

2022-02-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38146: -- Summary: UDAF fails with unsafe row buffer containing a TIMESTAMP_NTZ column (was: UDAF

[jira] [Comment Edited] (SPARK-38146) UDAF fails with unsafe rows containing a TIMESTAMP_NTZ column

2022-02-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489207#comment-17489207 ] Bruce Robbins edited comment on SPARK-38146 at 2/9/22, 2:23 AM: This

[jira] [Commented] (SPARK-38146) UDAF fails with unsafe rows containing a TIMESTAMP_NTZ column

2022-02-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489207#comment-17489207 ] Bruce Robbins commented on SPARK-38146: --- This affects master only and has a simple fix:

[jira] [Created] (SPARK-38146) UDAF fails with unsafe rows containing a TIMESTAMP_NTZ column

2022-02-08 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38146: - Summary: UDAF fails with unsafe rows containing a TIMESTAMP_NTZ column Key: SPARK-38146 URL: https://issues.apache.org/jira/browse/SPARK-38146 Project: Spark

[jira] [Commented] (SPARK-38133) Grouping by timestamp_ntz will sometimes corrupt the results

2022-02-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17488514#comment-17488514 ] Bruce Robbins commented on SPARK-38133: --- [~dongjoon]  >Does this happen on master branch only As

[jira] [Commented] (SPARK-38133) Grouping by timestamp_ntz will sometimes corrupt the results

2022-02-07 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17488441#comment-17488441 ] Bruce Robbins commented on SPARK-38133: --- I think I have a handle on what is causing this, and will

[jira] [Created] (SPARK-38133) Grouping by timestamp_ntz will sometimes corrupt the results

2022-02-07 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38133: - Summary: Grouping by timestamp_ntz will sometimes corrupt the results Key: SPARK-38133 URL: https://issues.apache.org/jira/browse/SPARK-38133 Project: Spark

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38075: -- Affects Version/s: 3.2.1 > Hive script transform with order by and limit will return fake

[jira] [Commented] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17484468#comment-17484468 ] Bruce Robbins commented on SPARK-38075: --- It's a small iterator issue. I will make a PR shortly. >

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38075: -- Affects Version/s: 3.1.2 > Hive script transform with order by and limit will return fake

[jira] [Updated] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-30 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-38075: -- Affects Version/s: 3.2.0 > Hive script transform with order by and limit will return fake

[jira] [Created] (SPARK-38075) Hive script transform with order by and limit will return fake rows

2022-01-30 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-38075: - Summary: Hive script transform with order by and limit will return fake rows Key: SPARK-38075 URL: https://issues.apache.org/jira/browse/SPARK-38075 Project: Spark

[jira] [Commented] (SPARK-38000) Sort node incorrectly removed from the optimized logical plan

2022-01-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-38000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17481392#comment-17481392 ] Bruce Robbins commented on SPARK-38000: --- I can reproduce on 3.2.0, but it seems to be fixed on

[jira] [Commented] (SPARK-37947) Cannot use _outer generators in a lateral view

2022-01-17 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17477412#comment-17477412 ] Bruce Robbins commented on SPARK-37947: --- While a minor issue (you could always use "{{outer }}"

[jira] [Created] (SPARK-37947) Cannot use _outer generators in a lateral view

2022-01-17 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-37947: - Summary: Cannot use _outer generators in a lateral view Key: SPARK-37947 URL: https://issues.apache.org/jira/browse/SPARK-37947 Project: Spark Issue Type:

[jira] [Commented] (SPARK-37832) Orc struct serializer should look up field converters in an array rather than a linked list

2022-01-06 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470235#comment-17470235 ] Bruce Robbins commented on SPARK-37832: --- I will post of PR shortly. > Orc struct serializer

[jira] [Created] (SPARK-37832) Orc struct serializer should look up field converters in an array rather than a linked list

2022-01-06 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-37832: - Summary: Orc struct serializer should look up field converters in an array rather than a linked list Key: SPARK-37832 URL: https://issues.apache.org/jira/browse/SPARK-37832

[jira] [Updated] (SPARK-37803) When deserializing an Orc struct, reuse the result row when possible

2022-01-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37803: -- Description: Create new benchmarks for struct deserializer improvement. (was: For each Orc

[jira] [Updated] (SPARK-37803) Create new benchmarks for struct deserializer improvement.

2022-01-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37803: -- Description: Create new benchmarks for struct deserializer improvement (SPARK-37812) (was:

[jira] [Updated] (SPARK-37803) Create new benchmarks for struct deserializer improvement.

2022-01-04 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37803: -- Summary: Create new benchmarks for struct deserializer improvement. (was: When deserializing

[jira] [Created] (SPARK-37812) When deserializing an Orc struct, reuse the result row when possible

2022-01-04 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-37812: - Summary: When deserializing an Orc struct, reuse the result row when possible Key: SPARK-37812 URL: https://issues.apache.org/jira/browse/SPARK-37812 Project:

[jira] [Updated] (SPARK-37803) When deserializing an Orc struct, reuse the result row when possible

2022-01-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37803: -- Attachment: pr_results.txt > When deserializing an Orc struct, reuse the result row when

[jira] [Updated] (SPARK-37803) When deserializing an Orc struct, reuse the result row when possible

2022-01-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37803: -- Attachment: master_results.txt > When deserializing an Orc struct, reuse the result row when

[jira] [Commented] (SPARK-37803) When deserializing an Orc struct, reuse the result row when possible

2022-01-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468283#comment-17468283 ] Bruce Robbins commented on SPARK-37803: --- I will attempt a PR in the next few hours (waiting for

[jira] [Created] (SPARK-37803) When deserializing an Orc struct, reuse the result row when possible

2022-01-03 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-37803: - Summary: When deserializing an Orc struct, reuse the result row when possible Key: SPARK-37803 URL: https://issues.apache.org/jira/browse/SPARK-37803 Project:

[jira] [Commented] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-12 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442848#comment-17442848 ] Bruce Robbins commented on SPARK-37270: --- [~yumwang] Seems related to SPARK-33848. > Incorect

[jira] [Commented] (SPARK-37270) Incorect result of filter using isNull condition

2021-11-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442433#comment-17442433 ] Bruce Robbins commented on SPARK-37270: --- I can reproduce locally. In 3.1, the above snippet

[jira] [Updated] (SPARK-37175) Performance improvement to hash joins with many duplicate keys

2021-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37175: -- Description: I noticed that HashedRelations with many duplicate keys perform significantly

[jira] [Updated] (SPARK-37175) Performance improvement to hash joins with many duplicate keys

2021-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37175: -- Description: I noticed that HashedRelations with many duplicate keys perform significantly

[jira] [Updated] (SPARK-37175) Performance improvement to hash joins with many duplicate keys

2021-10-31 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-37175: -- Attachment: hash_rel_examples.txt > Performance improvement to hash joins with many duplicate

[jira] [Created] (SPARK-37175) Performance improvement to hash joins with many duplicate keys

2021-10-31 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-37175: - Summary: Performance improvement to hash joins with many duplicate keys Key: SPARK-37175 URL: https://issues.apache.org/jira/browse/SPARK-37175 Project: Spark

[jira] [Created] (SPARK-36568) Missed broadcast join in V2 plan

2021-08-23 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-36568: - Summary: Missed broadcast join in V2 plan Key: SPARK-36568 URL: https://issues.apache.org/jira/browse/SPARK-36568 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

2021-06-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365764#comment-17365764 ] Bruce Robbins commented on SPARK-35817: --- [~xkrogen] Thanks! {quote}I guess we should create a map

[jira] [Commented] (SPARK-35817) Queries against wide Avro tables can be slow

2021-06-18 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17365697#comment-17365697 ] Bruce Robbins commented on SPARK-35817: --- The referenced line of code is meant to respect case

[jira] [Created] (SPARK-35817) Queries against wide Avro tables can be slow

2021-06-18 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-35817: - Summary: Queries against wide Avro tables can be slow Key: SPARK-35817 URL: https://issues.apache.org/jira/browse/SPARK-35817 Project: Spark Issue Type:

[jira] [Commented] (SPARK-34731) ConcurrentModificationException in EventLoggingListener when redacting properties

2021-06-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17355289#comment-17355289 ] Bruce Robbins commented on SPARK-34731: --- I am working from memory, but I remember that you lose

[jira] [Commented] (SPARK-35178) maven autodownload failing

2021-04-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326910#comment-17326910 ] Bruce Robbins commented on SPARK-35178: --- In INFRA-21767, Daniel Gruno responded: {quote} Please

[jira] [Commented] (SPARK-35178) maven autodownload failing

2021-04-21 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326874#comment-17326874 ] Bruce Robbins commented on SPARK-35178: --- I also posted

[jira] [Created] (SPARK-35178) maven autodownload failing

2021-04-21 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-35178: - Summary: maven autodownload failing Key: SPARK-35178 URL: https://issues.apache.org/jira/browse/SPARK-35178 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-34731) ConcurrentModificationException in EventLoggingListener when redacting properties

2021-03-15 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-34731: -- Affects Version/s: 3.1.1 > ConcurrentModificationException in EventLoggingListener when

[jira] [Commented] (SPARK-24758) Create table wants to use /user/hive/warehouse in clean clone

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301039#comment-17301039 ] Bruce Robbins commented on SPARK-24758: --- Issue no longer present on master. > Create table wants

[jira] [Resolved] (SPARK-24758) Create table wants to use /user/hive/warehouse in clean clone

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-24758. --- Resolution: Fixed > Create table wants to use /user/hive/warehouse in clean clone >

[jira] [Commented] (SPARK-24814) Relationship between catalog and datasources

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301038#comment-17301038 ] Bruce Robbins commented on SPARK-24814: --- Very old Jira, now obsolete. > Relationship between

[jira] [Resolved] (SPARK-24814) Relationship between catalog and datasources

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-24814. --- Resolution: Resolved > Relationship between catalog and datasources >

[jira] [Resolved] (SPARK-33098) Explicit or implicit casts involving partition columns can sometimes result in a MetaException.

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-33098. --- Resolution: Duplicate > Explicit or implicit casts involving partition columns can

[jira] [Commented] (SPARK-33098) Explicit or implicit casts involving partition columns can sometimes result in a MetaException.

2021-03-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301037#comment-17301037 ] Bruce Robbins commented on SPARK-33098: --- Fixed by PR for SPARK-27421. > Explicit or implicit

[jira] [Created] (SPARK-34731) ConcurrentModificationException in EventLoggingListener when redacting properties

2021-03-12 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-34731: - Summary: ConcurrentModificationException in EventLoggingListener when redacting properties Key: SPARK-34731 URL: https://issues.apache.org/jira/browse/SPARK-34731

[jira] [Updated] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2020-11-19 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33482: -- Description: Sample query: {noformat}

[jira] [Created] (SPARK-33482) V2 Datasources that extend FileScan preclude exchange reuse

2020-11-18 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-33482: - Summary: V2 Datasources that extend FileScan preclude exchange reuse Key: SPARK-33482 URL: https://issues.apache.org/jira/browse/SPARK-33482 Project: Spark

[jira] [Updated] (SPARK-33314) Avro reader drops rows

2020-11-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33314: -- Labels: correctness (was: ) > Avro reader drops rows > -- > >

[jira] [Created] (SPARK-33314) Avro reader drops rows

2020-11-01 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-33314: - Summary: Avro reader drops rows Key: SPARK-33314 URL: https://issues.apache.org/jira/browse/SPARK-33314 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-33098) Explicit or implicit casts involving partition columns can sometimes result in a MetaException.

2020-10-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: The following cases throw {{MetaException(message:Filtering is supported only on

[jira] [Updated] (SPARK-33098) Explicit or implicit casts involving partition columns can sometimes result in a MetaException.

2020-10-29 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Summary: Explicit or implicit casts involving partition columns can sometimes result in a

[jira] [Resolved] (SPARK-31342) Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582

2020-10-25 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-31342. --- Resolution: Duplicate > Fail by default if Parquet DATE or TIMESTAMP data is before October

[jira] [Commented] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211453#comment-17211453 ] Bruce Robbins commented on SPARK-33098: --- I  left out one case, which I added to the bottom to the 

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-09 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Updated] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-08 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-33098: -- Description: Comparing a partition column against a literal with the wrong type works if you

[jira] [Created] (SPARK-33098) Exception when using 'in' to compare a partition column to a literal with the wrong type

2020-10-08 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-33098: - Summary: Exception when using 'in' to compare a partition column to a literal with the wrong type Key: SPARK-33098 URL: https://issues.apache.org/jira/browse/SPARK-33098

[jira] [Updated] (SPARK-32779) Spark/Hive3 interaction potentially causes deadlock

2020-09-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-32779: -- Summary: Spark/Hive3 interaction potentially causes deadlock (was: Spark/Hive3 HMS

[jira] [Updated] (SPARK-32779) Spark/Hive3 HMS interaction potentially causes deadlock

2020-09-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-32779: -- Summary: Spark/Hive3 HMS interaction potentially causes deadlock (was: Spark/Hive3

[jira] [Created] (SPARK-32779) Spark/Hive3 interaction potentially causes deadlock

2020-09-02 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-32779: - Summary: Spark/Hive3 interaction potentially causes deadlock Key: SPARK-32779 URL: https://issues.apache.org/jira/browse/SPARK-32779 Project: Spark Issue

[jira] [Updated] (SPARK-32281) Spark wipes out SORTED spec in metastore when DESC is used

2020-07-11 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-32281: -- Summary: Spark wipes out SORTED spec in metastore when DESC is used (was: Spark wipes out

[jira] [Created] (SPARK-32281) Spark wipes out SORTED spec in metastore when when DESC is used

2020-07-11 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-32281: - Summary: Spark wipes out SORTED spec in metastore when when DESC is used Key: SPARK-32281 URL: https://issues.apache.org/jira/browse/SPARK-32281 Project: Spark

[jira] [Resolved] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2020-05-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-27497. --- Fix Version/s: 3.1.0 2.4.6 Resolution: Fixed > Spark wipes out

[jira] [Commented] (SPARK-27497) Spark wipes out bucket spec in metastore when updating table stats

2020-05-10 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103860#comment-17103860 ] Bruce Robbins commented on SPARK-27497: --- This is fixed in recent versions of 2.4.x and 3.x.

[jira] [Resolved] (SPARK-31598) LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps

2020-05-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins resolved SPARK-31598. --- Fix Version/s: 3.1.0 3.0.0 Resolution: Fixed >

[jira] [Commented] (SPARK-31598) LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps

2020-05-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097470#comment-17097470 ] Bruce Robbins commented on SPARK-31598: --- Thanks. SPARK-31557 was used for fixing both DATEs and

[jira] [Created] (SPARK-31598) LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps

2020-04-28 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31598: - Summary: LegacySimpleTimestampFormatter incorrectly interprets pre-Gregorian timestamps Key: SPARK-31598 URL: https://issues.apache.org/jira/browse/SPARK-31598

[jira] [Commented] (SPARK-31557) Legacy parser incorrectly interprets pre-Gregorian dates

2020-04-25 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092318#comment-17092318 ] Bruce Robbins commented on SPARK-31557: --- In case Jiras are no longer getting updated, here's a PR: 

[jira] [Created] (SPARK-31557) Legacy parser incorrectly interprets pre-Gregorian dates

2020-04-24 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31557: - Summary: Legacy parser incorrectly interprets pre-Gregorian dates Key: SPARK-31557 URL: https://issues.apache.org/jira/browse/SPARK-31557 Project: Spark

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-15 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084288#comment-17084288 ] Bruce Robbins commented on SPARK-31423: --- Thanks. It seems we can either - close this as "not a

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-15 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084218#comment-17084218 ] Bruce Robbins commented on SPARK-31423: --- OK, so this is a case of a limitation of the ORC library,

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-14 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083611#comment-17083611 ] Bruce Robbins commented on SPARK-31423: --- {quote}It is questionable how to handle the date in such

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-13 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082498#comment-17082498 ] Bruce Robbins commented on SPARK-31423: --- [~cloud_fan]  {quote}FYI this is the behavior of Spark

[jira] [Created] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

2020-04-11 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31423: - Summary: DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC Key: SPARK-31423 URL: https://issues.apache.org/jira/browse/SPARK-31423

[jira] [Created] (SPARK-31342) Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582

2020-04-03 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31342: - Summary: Fail by default if Parquet DATE or TIMESTAMP data is before October 15, 1582 Key: SPARK-31342 URL: https://issues.apache.org/jira/browse/SPARK-31342

[jira] [Commented] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-04-03 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074745#comment-17074745 ] Bruce Robbins commented on SPARK-30951: --- {quote} we can fail by default when reading datetime

[jira] [Commented] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-04-01 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073026#comment-17073026 ] Bruce Robbins commented on SPARK-30951: --- Since there is some inherit danger in a migrating user

[jira] [Commented] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-03-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066170#comment-17066170 ] Bruce Robbins commented on SPARK-30951: --- I added a subtask for ORC. The issue is only for date

[jira] [Created] (SPARK-31238) Incompatible ORC dates with Spark 2.4

2020-03-24 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-31238: - Summary: Incompatible ORC dates with Spark 2.4 Key: SPARK-31238 URL: https://issues.apache.org/jira/browse/SPARK-31238 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-03-24 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066153#comment-17066153 ] Bruce Robbins commented on SPARK-30951: --- [~cloud_fan] {quote} For ORC, it follows the Java

[jira] [Updated] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-02-27 Thread Bruce Robbins (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-30951: -- Description: tl;dr: We recently discovered some Spark 2.x sites that have lots of data

[jira] [Created] (SPARK-30951) Potential data loss for legacy applications after switch to proleptic Gregorian calendar

2020-02-25 Thread Bruce Robbins (Jira)
Bruce Robbins created SPARK-30951: - Summary: Potential data loss for legacy applications after switch to proleptic Gregorian calendar Key: SPARK-30951 URL: https://issues.apache.org/jira/browse/SPARK-30951

[jira] [Commented] (SPARK-27466) LEAD function with 'ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING' causes exception in Spark

2019-06-27 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874292#comment-16874292 ] Bruce Robbins commented on SPARK-27466: --- Hi [~hvanhovell] and/or [~yhuai], any comment on my

<    1   2   3   4   5   >