[jira] [Commented] (SPARK-33300) Rule SimplifyCasts will not work for nested columns

2020-11-03 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225625#comment-17225625 ] Aoyuan Liao commented on SPARK-33300: - [~tobe] Actually, it is not a bug. It did simplify nested

[jira] [Commented] (SPARK-33300) Rule SimplifyCasts will not work for nested columns

2020-11-02 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224937#comment-17224937 ] Aoyuan Liao commented on SPARK-33300: - I would like to work on this. > Rule SimplifyCasts will not

[jira] [Commented] (SPARK-33150) Groupby key may not be unique when using window

2020-10-30 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223971#comment-17223971 ] Aoyuan Liao commented on SPARK-33150: - [~DieterDP] The time difference is passed via fold attribute.

[jira] [Commented] (SPARK-32789) Wildcards not working in get_json_object

2020-10-26 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221044#comment-17221044 ] Aoyuan Liao commented on SPARK-32789: - [~tuhren] Not sure if HIve supports wildcard for dictionary.

[jira] [Resolved] (SPARK-32789) Wildcards not working in get_json_object

2020-10-26 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao resolved SPARK-32789. - Resolution: Not A Problem > Wildcards not working in get_json_object >

[jira] [Commented] (SPARK-33150) Groupby key may not be unique when using window

2020-10-26 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220984#comment-17220984 ] Aoyuan Liao commented on SPARK-33150: - [~DieterDP] After I looked deeper into code, the issue is not

[jira] [Comment Edited] (SPARK-33221) UDF in when operation applied in all the rows regardless the condition

2020-10-22 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219390#comment-17219390 ] Aoyuan Liao edited comment on SPARK-33221 at 10/23/20, 12:28 AM: -

[jira] [Resolved] (SPARK-33221) UDF in when operation applied in all the rows regardless the condition

2020-10-22 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao resolved SPARK-33221. - Resolution: Not A Problem > UDF in when operation applied in all the rows regardless the

[jira] [Commented] (SPARK-33221) UDF in when operation applied in all the rows regardless the condition

2020-10-22 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219390#comment-17219390 ] Aoyuan Liao commented on SPARK-33221: - [~marcodena] It is not a bug but intended. In 

[jira] [Resolved] (SPARK-19297) Add ability for --packages tag to pull latest version

2020-10-21 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao resolved SPARK-19297. - Target Version/s: 3.0.1 Resolution: Fixed > Add ability for --packages tag to pull

[jira] [Commented] (SPARK-19297) Add ability for --packages tag to pull latest version

2020-10-21 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218660#comment-17218660 ] Aoyuan Liao commented on SPARK-19297: - latest.release can be used. {code:java} // bin/spark-submit

[jira] [Commented] (SPARK-21529) Improve the error message for unsupported Uniontype

2020-10-21 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218653#comment-17218653 ] Aoyuan Liao commented on SPARK-21529: - [~teabot] I think catalyst still doesn't support uniontype.

[jira] [Commented] (SPARK-25985) Verify the SPARK-24613 Cache with UDF could not be matched with subsequent dependent caches

2020-10-20 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218019#comment-17218019 ] Aoyuan Liao commented on SPARK-25985: - [~smilegator] I think recacheByCondition doesn't keep the

[jira] [Commented] (SPARK-31760) Simplification Based on Containment

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217180#comment-17217180 ] Aoyuan Liao commented on SPARK-31760: - [~yumwang] To implement the containment rule:  If A contains

[jira] [Updated] (SPARK-31760) Simplification Based on Containment

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao updated SPARK-31760: Attachment: image-2020-10-19-17-18-31-849.png > Simplification Based on Containment >

[jira] [Commented] (SPARK-33181) SQL Reference: Run SQL on files directly

2020-10-19 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217023#comment-17217023 ] Aoyuan Liao commented on SPARK-33181: - [~smilegator] If this one hasn't been done, can I work on it?

[jira] [Commented] (SPARK-33071) Join with ambiguous column succeeding but giving wrong output

2020-10-13 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213312#comment-17213312 ] Aoyuan Liao commented on SPARK-33071: - The master branch has the same bug. > Join with ambiguous

[jira] [Commented] (SPARK-32069) Improve error message on reading unexpected directory which is not a table partition

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211312#comment-17211312 ] Aoyuan Liao commented on SPARK-32069: - [~Gengliang.Wang] If no one is working on it, can I take this

[jira] [Commented] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211305#comment-17211305 ] Aoyuan Liao commented on SPARK-9686: [~srowen] I think SPARK-28426 fixed this. > Spark Thrift server

[jira] [Commented] (SPARK-33039) Misleading watermark calculation in structure streaming

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211288#comment-17211288 ] Aoyuan Liao commented on SPARK-33039: - [~srowen] This is actually not a bug. The user didn't fully

[jira] [Commented] (SPARK-16859) History Server storage information is missing

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211284#comment-17211284 ] Aoyuan Liao commented on SPARK-16859: - [~srowen] After configuring 

[jira] [Commented] (SPARK-31430) Bug in the approximate quantile computation.

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211282#comment-17211282 ] Aoyuan Liao commented on SPARK-31430: - [~srowen] This is already fixed. > Bug in the approximate

[jira] [Commented] (SPARK-9686) Spark Thrift server doesn't return correct JDBC metadata

2020-10-09 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-9686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211280#comment-17211280 ] Aoyuan Liao commented on SPARK-9686: [~srowen] This is resolved on 3.0.1. Should we mark it as fixed?

[jira] [Updated] (SPARK-32978) Incorrect number of dynamic part metric

2020-10-07 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aoyuan Liao updated SPARK-32978: Description: How to reproduce this issue: {code:sql} create table dynamic_partition(i bigint,

[jira] [Commented] (SPARK-16859) History Server storage information is missing

2020-10-07 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209932#comment-17209932 ] Aoyuan Liao commented on SPARK-16859: - "spark.eventLog.logBlockUpdates.enabled=true" works for me on

[jira] [Commented] (SPARK-33071) Join with ambiguous column succeeding but giving wrong output

2020-10-06 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209064#comment-17209064 ] Aoyuan Liao commented on SPARK-33071: - Yeah, I agree with you. IMHO, would prefer a error message

[jira] [Commented] (SPARK-33039) Misleading watermark calculation in structure streaming

2020-10-06 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209030#comment-17209030 ] Aoyuan Liao commented on SPARK-33039: - Since the window: [2020-08-29 10:15:00, 2020-08-29 10:15:10]

[jira] [Commented] (SPARK-33071) Join with ambiguous column succeeding but giving wrong output

2020-10-06 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209026#comment-17209026 ] Aoyuan Liao commented on SPARK-33071: - Need to talk with experienced contributors to see if it is a

[jira] [Commented] (SPARK-33039) Misleading watermark calculation in structure streaming

2020-10-05 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208409#comment-17208409 ] Aoyuan Liao commented on SPARK-33039: - It is not a bug. It is documented that " It is important to

[jira] [Comment Edited] (SPARK-32007) Spark Driver Supervise does not work reliably

2020-10-01 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205918#comment-17205918 ] Aoyuan Liao edited comment on SPARK-32007 at 10/2/20, 1:40 AM: ---

[jira] [Commented] (SPARK-32007) Spark Driver Supervise does not work reliably

2020-10-01 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205918#comment-17205918 ] Aoyuan Liao commented on SPARK-32007: - [~surajs21] Can you please post master's log for more

[jira] [Commented] (SPARK-28001) Dataframe throws 'socket.timeout: timed out' exception

2020-09-10 Thread Aoyuan Liao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193992#comment-17193992 ] Aoyuan Liao commented on SPARK-28001: - [~StanislavKo] If possible, can you post the dataset you