[GitHub] [spark] cloud-fan commented on a change in pull request #28926: [SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28926: URL: https://github.com/apache/spark/pull/28926#discussion_r447466465 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala ## @@ -1854,4 +1854,16 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #28935: [SPARK-20680][SQL] Adding HiveVoidType in Spark to be compatible with Hive

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28935: URL: https://github.com/apache/spark/pull/28935#discussion_r447454481 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -2184,7 +2184,7 @@ class AstBuilder(conf:

[GitHub] [spark] viirya commented on pull request #28952: [SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled

2020-06-30 Thread GitBox
viirya commented on pull request #28952: URL: https://github.com/apache/spark/pull/28952#issuecomment-651588754 Does jenkins not work? This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] cloud-fan commented on a change in pull request #28935: [SPARK-20680][SQL] Adding HiveVoidType in Spark to be compatible with Hive

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28935: URL: https://github.com/apache/spark/pull/28935#discussion_r447453969 ## File path: sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala ## @@ -131,8 +135,9 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #28926: [SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28926: URL: https://github.com/apache/spark/pull/28926#discussion_r447465413 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ## @@ -2612,6 +2612,9 @@ object

[GitHub] [spark] cloud-fan commented on a change in pull request #28926: [SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28926: URL: https://github.com/apache/spark/pull/28926#discussion_r447464759 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala ## @@ -2679,6 +2682,11 @@ object

[GitHub] [spark] liancheng commented on pull request #28948: [SPARK-31935][SQL][FOLLOWUP] Hadoop file system config should be effective in data source options

2020-06-30 Thread GitBox
liancheng commented on pull request #28948: URL: https://github.com/apache/spark/pull/28948#issuecomment-651569654 @cloud-fan, thanks for fixing this! We should probably backport this one to 2.4 as well. This is an

[GitHub] [spark] beliefer commented on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer commented on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-651601992 @hvanhovell I have checked PostgreSQL, Vertica, Oracle, Redshift, Presto, Teradata, NTH_VALUE is always used as a window function, not as an aggregate function.

[GitHub] [spark] dilipbiswal commented on a change in pull request #28953: [SPARK-32013][SQL] Support query execution before/after reading/writing DataFrame over JDBC

2020-06-30 Thread GitBox
dilipbiswal commented on a change in pull request #28953: URL: https://github.com/apache/spark/pull/28953#discussion_r447433000 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala ## @@ -122,6 +122,19 @@ object JdbcUtils

[GitHub] [spark] AngersZhuuuu commented on pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
AngersZh commented on pull request #28805: URL: https://github.com/apache/spark/pull/28805#issuecomment-651565621 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] GuoPhilipse commented on a change in pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on a change in pull request #28951: URL: https://github.com/apache/spark/pull/28951#discussion_r447448239 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ## @@ -831,4 +831,77 @@ class AnalysisSuite

[GitHub] [spark] cloud-fan commented on pull request #28948: [SPARK-31935][SQL][FOLLOWUP] Hadoop file system config should be effective in data source options

2020-06-30 Thread GitBox
cloud-fan commented on pull request #28948: URL: https://github.com/apache/spark/pull/28948#issuecomment-651582057 Yea, I'll open a new PR for 2.4 after this one gets merged. This is an automated message from the Apache Git

[GitHub] [spark] cloud-fan commented on pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
cloud-fan commented on pull request #28805: URL: https://github.com/apache/spark/pull/28805#issuecomment-651597035 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] beliefer commented on a change in pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer commented on a change in pull request #28685: URL: https://github.com/apache/spark/pull/28685#discussion_r447462884 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala ## @@ -363,6 +363,11 @@ abstract class

[GitHub] [spark] dilipbiswal commented on pull request #28647: [SPARK-31828][SQL] Retain table properties at CreateTableLikeCommand

2020-06-30 Thread GitBox
dilipbiswal commented on pull request #28647: URL: https://github.com/apache/spark/pull/28647#issuecomment-651566036 Changes look good to me. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] beliefer edited a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer edited a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-651601992 @hvanhovell Thanks. 1.Yes, this PR follows up the implement of `LEAD/LEG`, so `NTH_VALUE` is the same as `LEAD/LEG`, only works for an unbounded frame (`UNBOUNDED

[GitHub] [spark] maropu commented on a change in pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

2020-06-30 Thread GitBox
maropu commented on a change in pull request #28852: URL: https://github.com/apache/spark/pull/28852#discussion_r447497252 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala ## @@ -226,4 +227,16 @@ object StaticSQLConf {

[GitHub] [spark] maropu commented on pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

2020-06-30 Thread GitBox
maropu commented on pull request #28852: URL: https://github.com/apache/spark/pull/28852#issuecomment-651635382 Looks okay. cc: @cloud-fan @dongjoon-hyun @HyukjinKwon This is an automated message from the Apache Git

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28761: [SPARK-25557][SQL] Nested column predicate pushdown for ORC

2020-06-30 Thread GitBox
HyukjinKwon commented on a change in pull request #28761: URL: https://github.com/apache/spark/pull/28761#discussion_r447507397 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcTest.scala ## @@ -78,12 +78,16 @@ abstract class OrcTest

[GitHub] [spark] manuzhang commented on pull request #28916: [SPARK-32083][SQL] Coalesce to one partition when all partitions are empty in AQE

2020-06-30 Thread GitBox
manuzhang commented on pull request #28916: URL: https://github.com/apache/spark/pull/28916#issuecomment-651643006 Thanks for pointing that out. Let me try with a new PR. This is an automated message from the Apache Git

[GitHub] [spark] Ngone51 commented on a change in pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28911: URL: https://github.com/apache/spark/pull/28911#discussion_r447553884 ## File path: core/src/test/scala/org/apache/spark/shuffle/HostLocalShuffleFetchSuite.scala ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] Fokko commented on a change in pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

2020-06-30 Thread GitBox
Fokko commented on a change in pull request #28821: URL: https://github.com/apache/spark/pull/28821#discussion_r447563125 ## File path: sql/core/src/test/resources/sql-tests/results/udf/udf-window.sql.out ## @@ -154,17 +154,17 @@ SELECT val_timestamp, udf(cate),

[GitHub] [spark] GuoPhilipse commented on pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on pull request #28951: URL: https://github.com/apache/spark/pull/28951#issuecomment-651692515 retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dbaliafroozeh commented on a change in pull request #28885: [SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse

2020-06-30 Thread GitBox
dbaliafroozeh commented on a change in pull request #28885: URL: https://github.com/apache/spark/pull/28885#discussion_r447624697 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/reuse/Reuse.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] manuzhang opened a new pull request #28954: [SPARK-32083][SQL] Apply CoalesceShufflePartitions when input RDD has 0 partitions with AQE

2020-06-30 Thread GitBox
manuzhang opened a new pull request #28954: URL: https://github.com/apache/spark/pull/28954 ### What changes were proposed in this pull request? As suggested by @cloud-fan in https://github.com/apache/spark/pull/28916#issuecomment-651527077, apply `CoalesceShufflePartitions` with

[GitHub] [spark] gaborgsomogyi edited a comment on pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector

2020-06-30 Thread GitBox
gaborgsomogyi edited a comment on pull request #28863: URL: https://github.com/apache/spark/pull/28863#issuecomment-651617129 @dongjoon-hyun Yes, I've spent almost gross 1 month to make it work w/ `MiniKDC` but no success. Please see my comment

[GitHub] [spark] maropu commented on a change in pull request #28898: [SPARK-32059][SQL] Allow nested schema pruning thru window/sort/filter plans

2020-06-30 Thread GitBox
maropu commented on a change in pull request #28898: URL: https://github.com/apache/spark/pull/28898#discussion_r447488633 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala ## @@ -153,6 +169,7 @@ object

[GitHub] [spark] beliefer edited a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer edited a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-651601992 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] LantaoJin commented on a change in pull request #28935: [SPARK-20680][SQL] Adding HiveVoidType in Spark to be compatible with Hive

2020-06-30 Thread GitBox
LantaoJin commented on a change in pull request #28935: URL: https://github.com/apache/spark/pull/28935#discussion_r447502360 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -2184,7 +2184,7 @@ class AstBuilder(conf:

[GitHub] [spark] maropu commented on pull request #28953: [SPARK-32013][SQL] Support query execution before/after reading/writing DataFrame over JDBC

2020-06-30 Thread GitBox
maropu commented on pull request #28953: URL: https://github.com/apache/spark/pull/28953#issuecomment-651616703 > For ETL workload, there is a common requirement to perform SQL statement before/after reading/writing over JDBC. Here's examples; Create a view with specific conditions

[GitHub] [spark] xuanyuanking commented on pull request #28936: [SPARK-32126][SS] Scope Session.active in IncrementalExecution

2020-06-30 Thread GitBox
xuanyuanking commented on pull request #28936: URL: https://github.com/apache/spark/pull/28936#issuecomment-651623749 Sorry for the late. Agree we need a new Jira for this. I should create it by myself, thanks for your help! @dongjoon-hyun

[GitHub] [spark] HeartSaVioR commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory

2020-06-30 Thread GitBox
HeartSaVioR commented on pull request #28930: URL: https://github.com/apache/spark/pull/28930#issuecomment-651633651 Thanks all for reviewing and merging! This is an automated message from the Apache Git Service. To respond

[GitHub] [spark] GuoPhilipse commented on a change in pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on a change in pull request #28951: URL: https://github.com/apache/spark/pull/28951#discussion_r447512274 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ## @@ -831,4 +831,57 @@ class AnalysisSuite

[GitHub] [spark] GuoPhilipse commented on a change in pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on a change in pull request #28951: URL: https://github.com/apache/spark/pull/28951#discussion_r447512748 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ## @@ -831,4 +831,77 @@ class AnalysisSuite

[GitHub] [spark] maropu commented on a change in pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation

2020-06-30 Thread GitBox
maropu commented on a change in pull request #28804: URL: https://github.com/apache/spark/pull/28804#discussion_r447512443 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ## @@ -2196,6 +2196,13 @@ object SQLConf { .checkValue(bit

[GitHub] [spark] peter-toth commented on a change in pull request #28885: [SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse

2020-06-30 Thread GitBox
peter-toth commented on a change in pull request #28885: URL: https://github.com/apache/spark/pull/28885#discussion_r447567127 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/reuse/Reuse.scala ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] Ngone51 commented on a change in pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28924: URL: https://github.com/apache/spark/pull/28924#discussion_r447580528 ## File path: core/src/main/scala/org/apache/spark/util/RpcUtils.scala ## @@ -54,6 +56,12 @@ private[spark] object RpcUtils { RpcTimeout(conf,

[GitHub] [spark] Ngone51 commented on pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on pull request #28924: URL: https://github.com/apache/spark/pull/28924#issuecomment-651710437 ping @tgravescs @jiangxb1987 Could you also take a look? thanks! This is an automated message from the Apache

[GitHub] [spark] Ngone51 commented on pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on pull request #28924: URL: https://github.com/apache/spark/pull/28924#issuecomment-651710234 @holdenk Thanks for review. I've also updated the PR description for the user-facing part. This is an

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
AngersZh commented on a change in pull request #28805: URL: https://github.com/apache/spark/pull/28805#discussion_r447610292 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala ## @@ -108,4 +108,10 @@ class

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
AngersZh commented on a change in pull request #28805: URL: https://github.com/apache/spark/pull/28805#discussion_r447610783 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala ## @@ -108,4 +108,10 @@ class

[GitHub] [spark] attilapiros commented on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros commented on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651747960 What about the following? As `createNormalizedInternedPathname` is only used here in production:

[GitHub] [spark] attilapiros edited a comment on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros edited a comment on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651747960 What about the following? As `createNormalizedInternedPathname` is only used here in production:

[GitHub] [spark] xuanyuanking commented on a change in pull request #28941: [SPARK-32124][CORE][SHS] Fix taskEndReasonFromJson to handle event logs from old Spark versions

2020-06-30 Thread GitBox
xuanyuanking commented on a change in pull request #28941: URL: https://github.com/apache/spark/pull/28941#discussion_r447481083 ## File path: core/src/main/scala/org/apache/spark/util/JsonProtocol.scala ## @@ -1078,7 +1078,10 @@ private[spark] object JsonProtocol {

[GitHub] [spark] xuanyuanking commented on a change in pull request #28937: [SPARK-32115][SQL] Fix SUBSTRING to handle integer overflows

2020-06-30 Thread GitBox
xuanyuanking commented on a change in pull request #28937: URL: https://github.com/apache/spark/pull/28937#discussion_r447484628 ## File path: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java ## @@ -341,8 +341,17 @@ public UTF8String substringSQL(int

[GitHub] [spark] beliefer edited a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer edited a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-651601992 @hvanhovell Thanks. 1.Yes, this PR follows up the implement of `LEAD/LEG`, so `NTH_VALUE` is the same as `LEAD/LEG`, only works for an unbounded frame (`UNBOUNDED

[GitHub] [spark] cloud-fan commented on pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory

2020-06-30 Thread GitBox
cloud-fan commented on pull request #28930: URL: https://github.com/apache/spark/pull/28930#issuecomment-651628933 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] cloud-fan closed pull request #28930: [SPARK-29999][SS][FOLLOWUP] Fix test to check the actual metadata log directory

2020-06-30 Thread GitBox
cloud-fan closed pull request #28930: URL: https://github.com/apache/spark/pull/28930 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] dilipbiswal commented on pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
dilipbiswal commented on pull request #28951: URL: https://github.com/apache/spark/pull/28951#issuecomment-651648804 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] rajatahujaatinmobi commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

2020-06-30 Thread GitBox
rajatahujaatinmobi commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-651695582 > > rekicked the checks, once they pass I can commit > > continuous-integration/appveyor/pr is failing. Not sure why is that so? > rekicked the

[GitHub] [spark] HyukjinKwon commented on pull request #28955: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][FOLLOW-UP] Avoids changing test utils and minimizes the diff

2020-06-30 Thread GitBox
HyukjinKwon commented on pull request #28955: URL: https://github.com/apache/spark/pull/28955#issuecomment-651747473 cc @viirya, @dbtsai, @MaxGekk, @cloud-fan This is an automated message from the Apache Git Service. To

[GitHub] [spark] HyukjinKwon opened a new pull request #28955: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][FOLLOW-UP] Avoids changing test utils and minimizes the diff

2020-06-30 Thread GitBox
HyukjinKwon opened a new pull request #28955: URL: https://github.com/apache/spark/pull/28955 ### What changes were proposed in this pull request? This PR proposes to minimize the diff at https://github.com/apache/spark/pull/27728. Basically it addresses the comments

[GitHub] [spark] maropu commented on a change in pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
maropu commented on a change in pull request #28951: URL: https://github.com/apache/spark/pull/28951#discussion_r447475628 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala ## @@ -337,7 +337,8 @@ trait CheckAnalysis extends

[GitHub] [spark] maropu commented on a change in pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
maropu commented on a change in pull request #28951: URL: https://github.com/apache/spark/pull/28951#discussion_r447476067 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala ## @@ -831,4 +831,57 @@ class AnalysisSuite extends

[GitHub] [spark] maropu commented on pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
maropu commented on pull request #28951: URL: https://github.com/apache/spark/pull/28951#issuecomment-651610683 nit: `test2` not used in the PR description? This is an automated message from the Apache Git Service. To

[GitHub] [spark] peter-toth commented on pull request #28885: [SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse

2020-06-30 Thread GitBox
peter-toth commented on pull request #28885: URL: https://github.com/apache/spark/pull/28885#issuecomment-651701192 > Changes look good to me, added 3 minor comments. Thanks @dbaliafroozeh. @cloud-fan, @maryannxue could you review this PR?

[GitHub] [spark] manuzhang commented on pull request #28916: [SPARK-32083][SQL] Coalesce to one partition when all partitions are empty in AQE

2020-06-30 Thread GitBox
manuzhang commented on pull request #28916: URL: https://github.com/apache/spark/pull/28916#issuecomment-651746359 @cloud-fan @viirya Please help review the new PR https://github.com/apache/spark/pull/28954. This is an

[GitHub] [spark] manuzhang closed pull request #28916: [SPARK-32083][SQL] Coalesce to one partition when all partitions are empty in AQE

2020-06-30 Thread GitBox
manuzhang closed pull request #28916: URL: https://github.com/apache/spark/pull/28916 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [spark] wankunde commented on pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped

2020-06-30 Thread GitBox
wankunde commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-651607143 @srowen @holdenk Yes, a test is a use case. Another use case is to loop a user application containing sparkcontext.

[GitHub] [spark] xuanyuanking commented on pull request #28937: [SPARK-32115][SQL] Fix SUBSTRING to handle integer overflows

2020-06-30 Thread GitBox
xuanyuanking commented on pull request #28937: URL: https://github.com/apache/spark/pull/28937#issuecomment-651619834 Thank you for reviewing! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] GuoPhilipse commented on pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on pull request #28951: URL: https://github.com/apache/spark/pull/28951#issuecomment-651632797 > nit: `test2` not used in the PR description? have updated in the PR, it is a normal case ,it is used for comparing between the normal and abnormal cases.

[GitHub] [spark] Ngone51 commented on a change in pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28924: URL: https://github.com/apache/spark/pull/28924#discussion_r447579643 ## File path: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala ## @@ -93,6 +94,7 @@ class BlockManagerSuite extends SparkFunSuite

[GitHub] [spark] Fokko commented on a change in pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

2020-06-30 Thread GitBox
Fokko commented on a change in pull request #28821: URL: https://github.com/apache/spark/pull/28821#discussion_r447634917 ## File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestData.scala ## @@ -73,6 +74,17 @@ private[sql] trait SQLTestData { self => df

[GitHub] [spark] cloud-fan commented on a change in pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28805: URL: https://github.com/apache/spark/pull/28805#discussion_r447479242 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala ## @@ -108,4 +108,10 @@ class

[GitHub] [spark] cloud-fan commented on a change in pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28805: URL: https://github.com/apache/spark/pull/28805#discussion_r447480353 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PruneFileSourcePartitionsSuite.scala ## @@ -108,4 +108,10 @@ class

[GitHub] [spark] TJX2014 commented on a change in pull request #28926: [SPARK-32133][SQL] Forbid time field steps for date start/end in Sequence

2020-06-30 Thread GitBox
TJX2014 commented on a change in pull request #28926: URL: https://github.com/apache/spark/pull/28926#discussion_r447536856 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala ## @@ -1854,4 +1854,18 @@ class

[GitHub] [spark] LantaoJin commented on a change in pull request #28935: [SPARK-20680][SQL] Adding HiveVoidType in Spark to be compatible with Hive

2020-06-30 Thread GitBox
LantaoJin commented on a change in pull request #28935: URL: https://github.com/apache/spark/pull/28935#discussion_r447548764 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -2184,7 +2184,7 @@ class AstBuilder(conf:

[GitHub] [spark] Ngone51 commented on a change in pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28924: URL: https://github.com/apache/spark/pull/28924#discussion_r447558818 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -95,6 +97,13 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] attilapiros commented on a change in pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros commented on a change in pull request #28940: URL: https://github.com/apache/spark/pull/28940#discussion_r447551067 ## File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExecutorDiskUtils.java ## @@ -50,14 +58,18 @@ public static File

[GitHub] [spark] Fokko commented on a change in pull request #28754: [SPARK-10520][SQL] Allow average out of DateType

2020-06-30 Thread GitBox
Fokko commented on a change in pull request #28754: URL: https://github.com/apache/spark/pull/28754#discussion_r447634293 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/Average.scala ## @@ -40,10 +40,17 @@ case class

[GitHub] [spark] maropu commented on pull request #28952: [SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled

2020-06-30 Thread GitBox
maropu commented on pull request #28952: URL: https://github.com/apache/spark/pull/28952#issuecomment-651607497 Yea, it seems jenkins got sick last night... This is an automated message from the Apache Git Service. To

[GitHub] [spark] cloud-fan commented on a change in pull request #28952: [SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28952: URL: https://github.com/apache/spark/pull/28952#discussion_r447482241 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala ## @@ -130,6 +130,17 @@ class

[GitHub] [spark] gaborgsomogyi commented on pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector

2020-06-30 Thread GitBox
gaborgsomogyi commented on pull request #28863: URL: https://github.com/apache/spark/pull/28863#issuecomment-651617129 @dongjoon-hyun I've spent almost gross 1 month to make it work w/ `MiniKDC` but no success. Please see my comment

[GitHub] [spark] beliefer edited a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

2020-06-30 Thread GitBox
beliefer edited a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-651601992 @hvanhovell Thanks. 1.Yes, this PR follows up the implement of `LEAD/LEG`, so `NTH_VALUE` is the same as `LEAD/LEG`, only works for an unbounded frame (`UNBOUNDED

[GitHub] [spark] Ngone51 commented on a change in pull request #28911: [SPARK-32077][CORE] Support host-local shuffle data reading when external shuffle service is disabled

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28911: URL: https://github.com/apache/spark/pull/28911#discussion_r447554793 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -1391,10 +1391,11 @@ package object config { private[spark]

[GitHub] [spark] Ngone51 commented on a change in pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
Ngone51 commented on a change in pull request #28924: URL: https://github.com/apache/spark/pull/28924#discussion_r447555905 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala ## @@ -168,6 +177,37 @@ class BlockManagerMasterEndpoint(

[GitHub] [spark] HyukjinKwon commented on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
HyukjinKwon commented on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651759667 I was assuming we can't reuse `io.File` per: > * the internal code in java.io.File would normalize it later, creating a new "foo/bar" > * String copy.

[GitHub] [spark] cloud-fan opened a new pull request #28956: [SPARK-31797][SQL][FOLLOWUP] TIMESTAMP_SECONDS supports fractional input

2020-06-30 Thread GitBox
cloud-fan opened a new pull request #28956: URL: https://github.com/apache/spark/pull/28956 ### What changes were proposed in this pull request? This is a followup of https://github.com/apache/spark/pull/28534 , to make `TIMESTAMP_SECONDS` function support fractional input

[GitHub] [spark] tgravescs commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

2020-06-30 Thread GitBox
tgravescs commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-651810749 test this please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] pan3793 commented on a change in pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
pan3793 commented on a change in pull request #28940: URL: https://github.com/apache/spark/pull/28940#discussion_r447728376 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java ## @@ -146,12 +147,23 @@

[GitHub] [spark] cloud-fan commented on a change in pull request #28882: [SPARK-31751][SQL]Serde property `path` overwrites hive table property location

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28882: URL: https://github.com/apache/spark/pull/28882#discussion_r447758438 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ## @@ -545,7 +545,14 @@ private[spark] class

[GitHub] [spark] attilapiros commented on a change in pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros commented on a change in pull request #28940: URL: https://github.com/apache/spark/pull/28940#discussion_r447671636 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java ## @@ -147,22 +147,20 @@

[GitHub] [spark] attilapiros commented on a change in pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros commented on a change in pull request #28940: URL: https://github.com/apache/spark/pull/28940#discussion_r447682473 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java ## @@ -147,22 +147,20 @@

[GitHub] [spark] srowen commented on pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped

2020-06-30 Thread GitBox
srowen commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-651821297 OK, it's just quite a hacky approach here. Can we not clear these in a shutdown / close method? Or worst case try soft references?

[GitHub] [spark] HyukjinKwon commented on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
HyukjinKwon commented on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651844863 Build started: [CORE] `org.apache.spark.network.shuffle.ExternalShuffleBlockResolverSuite`

[GitHub] [spark] cloud-fan commented on a change in pull request #28935: [SPARK-20680][SQL] Adding HiveVoidType in Spark to be compatible with Hive

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28935: URL: https://github.com/apache/spark/pull/28935#discussion_r447763215 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -2184,7 +2184,7 @@ class AstBuilder(conf:

[GitHub] [spark] tgravescs commented on pull request #28918: [SPARK-32068][WEBUI] Correct task lauchtime show issue due to timezone in stage tab

2020-06-30 Thread GitBox
tgravescs commented on pull request #28918: URL: https://github.com/apache/spark/pull/28918#issuecomment-651808855 thanks @TJX2014 merged to master and branch-3.0 This is an automated message from the Apache Git Service. To

[GitHub] [spark] gaborgsomogyi commented on pull request #28635: [SPARK-31337][SQL]Support MS SQL Kerberos login in JDBC connector

2020-06-30 Thread GitBox
gaborgsomogyi commented on pull request #28635: URL: https://github.com/apache/spark/pull/28635#issuecomment-651814299 @thereverand just seen your comment. It has never stated that username/password is working. Propagating u/p through the whole cluster will enlarge the attack surface.

[GitHub] [spark] cloud-fan commented on a change in pull request #28805: [SPARK-28169][SQL] Convert scan predicate condition to CNF

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28805: URL: https://github.com/apache/spark/pull/28805#discussion_r447761441 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/PrunePartitionSuiteBase.scala ## @@ -0,0 +1,76 @@ +/* + * Licensed to the

[GitHub] [spark] GuoPhilipse commented on pull request #28951: [SPARK-32131][SQL] union and set operations have wrong exception infomation

2020-06-30 Thread GitBox
GuoPhilipse commented on pull request #28951: URL: https://github.com/apache/spark/pull/28951#issuecomment-651759412 It seesm i cannot trigger the test build, @dilipbiswal @maropu @HyukjinKwon @holdenk ,do you have any ideas?

[GitHub] [spark] attilapiros edited a comment on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros edited a comment on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651772632 We have to measure it for sure. Regarding simplicity of the solution I described it is basically sth like:

[GitHub] [spark] attilapiros commented on pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
attilapiros commented on pull request #28940: URL: https://github.com/apache/spark/pull/28940#issuecomment-651772632 We have to measure it for sure. Regarding simplicity of the solution I described it is basically sth like:

[GitHub] [spark] tgravescs commented on a change in pull request #28924: [SPARK-32091][CORE] Ignore timeout error when remove blocks on the lost executor

2020-06-30 Thread GitBox
tgravescs commented on a change in pull request #28924: URL: https://github.com/apache/spark/pull/28924#discussion_r447684033 ## File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala ## @@ -132,4 +132,6 @@ private[spark] object

[GitHub] [spark] cloud-fan commented on a change in pull request #28956: [SPARK-31797][SQL][FOLLOWUP] TIMESTAMP_SECONDS supports fractional input

2020-06-30 Thread GitBox
cloud-fan commented on a change in pull request #28956: URL: https://github.com/apache/spark/pull/28956#discussion_r447701004 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala ## @@ -1142,28 +1142,6 @@ class

[GitHub] [spark] cloud-fan commented on pull request #28956: [SPARK-31797][SQL][FOLLOWUP] TIMESTAMP_SECONDS supports fractional input

2020-06-30 Thread GitBox
cloud-fan commented on pull request #28956: URL: https://github.com/apache/spark/pull/28956#issuecomment-651807202 cc @TJX2014 @bart-samwel @HyukjinKwon @MaxGekk This is an automated message from the Apache Git Service.

[GitHub] [spark] asfgit closed pull request #28918: [SPARK-32068][WEBUI] Correct task lauchtime show issue due to timezone in stage tab

2020-06-30 Thread GitBox
asfgit closed pull request #28918: URL: https://github.com/apache/spark/pull/28918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] pan3793 commented on a change in pull request #28940: [SPARK-32121][SHUFFLE][TEST] Fix ExternalShuffleBlockResolverSuite failed on Windows

2020-06-30 Thread GitBox
pan3793 commented on a change in pull request #28940: URL: https://github.com/apache/spark/pull/28940#discussion_r447727579 ## File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java ## @@ -147,22 +147,20 @@

[GitHub] [spark] tgravescs commented on a change in pull request #28412: [SPARK-31608][CORE][WEBUI] Add a new type of KVStore to make loading UI faster

2020-06-30 Thread GitBox
tgravescs commented on a change in pull request #28412: URL: https://github.com/apache/spark/pull/28412#discussion_r447727051 ## File path: core/src/main/scala/org/apache/spark/deploy/history/HistoryServerMemoryManager.scala ## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28955: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][FOLLOW-UP] Avoids changing test utils and minimizes the diff

2020-06-30 Thread GitBox
HyukjinKwon commented on a change in pull request #28955: URL: https://github.com/apache/spark/pull/28955#discussion_r447743220 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala ## @@ -546,64 +552,67 @@ abstract

[GitHub] [spark] HyukjinKwon commented on a change in pull request #28955: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][FOLLOW-UP] Avoids changing test utils and minimizes the diff

2020-06-30 Thread GitBox
HyukjinKwon commented on a change in pull request #28955: URL: https://github.com/apache/spark/pull/28955#discussion_r447742259 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala ## @@ -501,38 +508,37 @@ abstract

  1   2   3   4   5   6   7   8   >