[GitHub] [spark] AmplabJenkins commented on issue #26089: [SPARK-29423][SS] lazily initialize StreamingQueryManager in SessionState

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26089: [SPARK-29423][SS] lazily initialize StreamingQueryManager in SessionState URL: https://github.com/apache/spark/pull/26089#issuecomment-541276468 Merged build finished. Test PASSed.

[GitHub] [spark] SparkQA commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql

2019-10-11 Thread GitBox
SparkQA commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql URL: https://github.com/apache/spark/pull/26055#issuecomment-541276352 **[Test build #111942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111942/testReport)** for PR 26055 at

[GitHub] [spark] AmplabJenkins commented on issue #26089: [SPARK-29423][SS] lazily initialize StreamingQueryManager in SessionState

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26089: [SPARK-29423][SS] lazily initialize StreamingQueryManager in SessionState URL: https://github.com/apache/spark/pull/26089#issuecomment-541276470 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql URL: https://github.com/apache/spark/pull/26055#issuecomment-541276467 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26055: [SPARK-29368][SQL][TEST] Port interval.sql URL: https://github.com/apache/spark/pull/26055#issuecomment-541276465 Merged build finished. Test PASSed. This is an automated message from

[GitHub] [spark] AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541236872 Merged build finished. Test PASSed.

[GitHub] [spark] zhengruifeng commented on issue #25929: [SPARK-29116][PYTHON][ML] Refactor py classes related to DecisionTree

2019-10-11 Thread GitBox
zhengruifeng commented on issue #25929: [SPARK-29116][PYTHON][ML] Refactor py classes related to DecisionTree URL: https://github.com/apache/spark/pull/25929#issuecomment-541277278 retest this please This is an automated

[GitHub] [spark] SparkQA commented on issue #25981: [SPARK-28420][SQL] Support the `INTERVAL` type in `date_part()`

2019-10-11 Thread GitBox
SparkQA commented on issue #25981: [SPARK-28420][SQL] Support the `INTERVAL` type in `date_part()` URL: https://github.com/apache/spark/pull/25981#issuecomment-541277428 **[Test build #111946 has

[GitHub] [spark] SparkQA commented on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread GitBox
SparkQA commented on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType URL: https://github.com/apache/spark/pull/26092#issuecomment-541277424 **[Test build #111944 has

[GitHub] [spark] AmplabJenkins removed a comment on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType URL: https://github.com/apache/spark/pull/26092#issuecomment-541236680 Merged build finished. Test PASSed.

[GitHub] [spark] SparkQA commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
SparkQA commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541277421 **[Test build #111945 has

[GitHub] [spark] SparkQA commented on issue #25929: [SPARK-29116][PYTHON][ML] Refactor py classes related to DecisionTree

2019-10-11 Thread GitBox
SparkQA commented on issue #25929: [SPARK-29116][PYTHON][ML] Refactor py classes related to DecisionTree URL: https://github.com/apache/spark/pull/25929#issuecomment-541277432 **[Test build #111947 has

[GitHub] [spark] AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541236879 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType URL: https://github.com/apache/spark/pull/26092#issuecomment-541236686 Test PASSed. Refer to this link for build results (access rights to CI server

[GitHub] [spark] zhengruifeng commented on a change in pull request #26064: [SPARK-23578][ML][PYSPARK] Binarizer support multi-column

2019-10-11 Thread GitBox
zhengruifeng commented on a change in pull request #26064: [SPARK-23578][ML][PYSPARK] Binarizer support multi-column URL: https://github.com/apache/spark/pull/26064#discussion_r334217681 ## File path: mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala ## @@

[GitHub] [spark] merrily01 commented on issue #25920: [SPARK-29233][K8S] Add regex expression checks for executorEnv…

2019-10-11 Thread GitBox
merrily01 commented on issue #25920: [SPARK-29233][K8S] Add regex expression checks for executorEnv… URL: https://github.com/apache/spark/pull/25920#issuecomment-541279733 Forget to say thank you. Thanks a lot @srowen @dongjoon-hyun

[GitHub] [spark] AmplabJenkins commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' URL: https://github.com/apache/spark/pull/26053#issuecomment-541281609 Merged build finished. Test PASSed. This

[GitHub] [spark] AmplabJenkins commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' URL: https://github.com/apache/spark/pull/26053#issuecomment-541281610 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' URL: https://github.com/apache/spark/pull/26053#issuecomment-541281609 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins removed a comment on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' URL: https://github.com/apache/spark/pull/26053#issuecomment-541281610 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case'

2019-10-11 Thread GitBox
SparkQA commented on issue #26053: [SPARK-29379][SQL]SHOW FUNCTIONS show '!=', '<>' , 'between', 'case' URL: https://github.com/apache/spark/pull/26053#issuecomment-541281523 **[Test build #111948 has

[GitHub] [spark] LantaoJin commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
LantaoJin commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541281628 Retest this please.

[GitHub] [spark] AmplabJenkins removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541283455 Merged build finished. Test FAILed.

[GitHub] [spark] AmplabJenkins commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541283455 Merged build finished. Test FAILed.

[GitHub] [spark] SparkQA removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
SparkQA removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541282120 **[Test build #111949 has

[GitHub] [spark] SparkQA commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
SparkQA commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541283451 **[Test build #111949 has

[GitHub] [spark] LantaoJin opened a new pull request #26097: [SPARK-29421][SQL] Supporting Create Table Like Stored as/Using FileFormat

2019-10-11 Thread GitBox
LantaoJin opened a new pull request #26097: [SPARK-29421][SQL] Supporting Create Table Like Stored as/Using FileFormat URL: https://github.com/apache/spark/pull/26097 ### What changes were proposed in this pull request? Hive support STORED AS new file format syntax: ```sql CREATE

[GitHub] [spark] AmplabJenkins commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541283456 Test FAILed. Refer to this link for build results (access rights to CI

[GitHub] [spark] AmplabJenkins removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #25960: [SPARK-29283][SQL] Error message is hidden when query from JDBC, especially enabled adaptive execution URL: https://github.com/apache/spark/pull/25960#issuecomment-541283456 Test FAILed. Refer to this link for build results (access rights

[GitHub] [spark] AmplabJenkins commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values URL: https://github.com/apache/spark/pull/26098#issuecomment-541284613 Can one of the admins verify this patch?

[GitHub] [spark] stczwd commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values

2019-10-11 Thread GitBox
stczwd commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values URL: https://github.com/apache/spark/pull/26098#issuecomment-541284599 Previous discussion about this: [SPARK-23773](https://github.com/apache/spark/pull/20884)

[GitHub] [spark] AmplabJenkins commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26098: Add configuration to support JacksonGenrator to keep fields with null values URL: https://github.com/apache/spark/pull/26098#issuecomment-541284526 Can one of the admins verify this patch?

[GitHub] [spark] AmplabJenkins commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541285306 Merged build finished. Test FAILed.

[GitHub] [spark] SparkQA removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
SparkQA removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541277421 **[Test build #111945 has

[GitHub] [spark] AmplabJenkins commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541285308 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541285306 Merged build finished. Test FAILed.

[GitHub] [spark] AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541285308 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix

2019-10-11 Thread GitBox
SparkQA commented on issue #26079: [SPARK-29369][SQL] Support string intervals without the `interval` prefix URL: https://github.com/apache/spark/pull/26079#issuecomment-541285275 **[Test build #111945 has

[GitHub] [spark] AmplabJenkins commented on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership URL: https://github.com/apache/spark/pull/26068#issuecomment-541288489 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite URL: https://github.com/apache/spark/pull/26028#issuecomment-541288483 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins removed a comment on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite URL: https://github.com/apache/spark/pull/26028#issuecomment-541288485 Test PASSed. Refer to this link for build results (access rights to CI server

[GitHub] [spark] AmplabJenkins removed a comment on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership URL: https://github.com/apache/spark/pull/26068#issuecomment-541288489 Test PASSed. Refer to this link for build results (access rights to CI server

[GitHub] [spark] AmplabJenkins removed a comment on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership

2019-10-11 Thread GitBox
AmplabJenkins removed a comment on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership URL: https://github.com/apache/spark/pull/26068#issuecomment-541288486 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins commented on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26068: [SPARK-29405][SQL] Alter table / Insert statements should not change a table's ownership URL: https://github.com/apache/spark/pull/26068#issuecomment-541288486 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins commented on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite URL: https://github.com/apache/spark/pull/26028#issuecomment-541288483 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins commented on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite

2019-10-11 Thread GitBox
AmplabJenkins commented on issue #26028: [SPARK-29359][SQL][TESTS] Better exception handling in (SQL|ThriftServer)QueryTestSuite URL: https://github.com/apache/spark/pull/26028#issuecomment-541288485 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333916849 ## File path:

[GitHub] [spark] sandeep-katta commented on a change in pull request #25977: [SPARK-29268][SQL]isolationOn value is wrong in case of spark.sql.hive.metastore.jars != builtin

2019-10-11 Thread GitBox
sandeep-katta commented on a change in pull request #25977: [SPARK-29268][SQL]isolationOn value is wrong in case of spark.sql.hive.metastore.jars != builtin URL: https://github.com/apache/spark/pull/25977#discussion_r333926916 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333927414 ## File path:

[GitHub] [spark] Clarkkkkk opened a new pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution

2019-10-11 Thread GitBox
Clark opened a new pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution URL: https://github.com/apache/spark/pull/26090 ### What changes were proposed in this pull request? When inserting into a partitioned

[GitHub] [spark] skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time

2019-10-11 Thread GitBox
skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time URL: https://github.com/apache/spark/pull/25609#discussion_r333990738 ## File path:

[GitHub] [spark] HeartSaVioR commented on issue #26089: [SPARK-29423][SQL] lazily initialize StreamingQueryManager in SessionState

2019-10-11 Thread GitBox
HeartSaVioR commented on issue #26089: [SPARK-29423][SQL] lazily initialize StreamingQueryManager in SessionState URL: https://github.com/apache/spark/pull/26089#issuecomment-541065149 I expect the code change would work, but it would be even better if you could attach some result of

[GitHub] [spark] MaxGekk commented on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread GitBox
MaxGekk commented on issue #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType URL: https://github.com/apache/spark/pull/26092#issuecomment-541086770 ping @cloud-fan @hvanhovell

[GitHub] [spark] skonto commented on issue #25870: [SPARK-27936][K8S] support python deps

2019-10-11 Thread GitBox
skonto commented on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-541001902 @holdenk this is because spark-submit add the resource in sparks.jars: ``` 19/10/11 13:01:30 WARN Utils: Your hostname, universe resolves to

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333917844 ## File path:

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333917844 ## File path:

[GitHub] [spark] turboFei edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite be unique

2019-10-11 Thread GitBox
turboFei edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite be unique URL: https://github.com/apache/spark/pull/26086#issuecomment-540982545 Oh, It seems that this issue is related with https://github.com/apache/spark/pull/24142.

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333943901 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown URL: https://github.com/apache/spark/pull/25955#discussion_r333962848 ## File path:

[GitHub] [spark] skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time

2019-10-11 Thread GitBox
skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time URL: https://github.com/apache/spark/pull/25609#discussion_r333990738 ## File path:

[GitHub] [spark] cloud-fan commented on issue #26091: [SPARK-29439][SQL] DDL commands should not use DataSourceV2Relation

2019-10-11 Thread GitBox
cloud-fan commented on issue #26091: [SPARK-29439][SQL] DDL commands should not use DataSourceV2Relation URL: https://github.com/apache/spark/pull/26091#issuecomment-541064323 @brkyvz @rdblue This is an automated message

[GitHub] [spark] nonsleepr commented on a change in pull request #26000: [SPARK-29330][CORE][YARN] Allow users to chose the name of Spark Shuffle service

2019-10-11 Thread GitBox
nonsleepr commented on a change in pull request #26000: [SPARK-29330][CORE][YARN] Allow users to chose the name of Spark Shuffle service URL: https://github.com/apache/spark/pull/26000#discussion_r334009122 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333926473 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333926303 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333934690 ## File path:

[GitHub] [spark] hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#discussion_r333938918 ## File path:

[GitHub] [spark] hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#discussion_r333936540 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

[GitHub] [spark] hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#discussion_r333938397 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333940553 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333940553 ## File path:

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333916849 ## File path:

[GitHub] [spark] s1ck commented on issue #24851: [SPARK-27303][GRAPH] Add Spark Graph API

2019-10-11 Thread GitBox
s1ck commented on issue #24851: [SPARK-27303][GRAPH] Add Spark Graph API URL: https://github.com/apache/spark/pull/24851#issuecomment-541041551 @dongjoon-hyun Thanks for the extra rounds. We made the `DataFrame` to `Dataset[Row]` changes and added some clarifying docs.

[GitHub] [spark] skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps

2019-10-11 Thread GitBox
skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-541001902 @holdenk this is because spark-submit adds the resource to the `sparks.jars` property by default, check bellow: ``` 19/10/11

[GitHub] [spark] turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server

2019-10-11 Thread GitBox
turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#discussion_r333935746 ## File path:

[GitHub] [spark] turboFei commented on a change in pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution

2019-10-11 Thread GitBox
turboFei commented on a change in pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution URL: https://github.com/apache/spark/pull/26090#discussion_r333941272 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333941162 ## File path:

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333917844 ## File path:

[GitHub] [spark] hagerf commented on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#issuecomment-541051711 I added PR with these changes and a new test for the typed datasets. 

[GitHub] [spark] hagerf edited a comment on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf edited a comment on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#issuecomment-541016870 Thank you for creating a PR so fast after my submitting the jira ticket! 

[GitHub] [spark] skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time

2019-10-11 Thread GitBox
skonto commented on a change in pull request #25609: [SPARK-28896][K8S] Support defining HADOOP_CONF_DIR and config map at the same time URL: https://github.com/apache/spark/pull/25609#discussion_r333985107 ## File path: docs/security.md ## @@ -845,8 +845,13 @@ When

[GitHub] [spark] Ngone51 commented on a change in pull request #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener

2019-10-11 Thread GitBox
Ngone51 commented on a change in pull request #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener URL: https://github.com/apache/spark/pull/25943#discussion_r334019895 ## File path:

[GitHub] [spark] skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps

2019-10-11 Thread GitBox
skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-541001902 @holdenk this is because spark-submit adds the resource to the `sparks.jars` property by default, check bellow: ``` 19/10/11

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333943188 ## File path:

[GitHub] [spark] hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#discussion_r333938397 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

[GitHub] [spark] Ngone51 commented on issue #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener

2019-10-11 Thread GitBox
Ngone51 commented on issue #25943: [WIP][SPARK-29261][SQL][CORE] Support recover live entities from KVStore for (SQL)AppStatusListener URL: https://github.com/apache/spark/pull/25943#issuecomment-541080685 > So you're saying the KVStore already has enough info, this PR just repopulates

[GitHub] [spark] MaxGekk opened a new pull request #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType

2019-10-11 Thread GitBox
MaxGekk opened a new pull request #26092: [SPARK-29440][SQL] Support java.time.Duration as an external type of CalendarIntervalType URL: https://github.com/apache/spark/pull/26092 ### What changes were proposed in this pull request? In the PR, I propose to convert values of the

[GitHub] [spark] hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on a change in pull request #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#discussion_r333936540 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333939935 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-10-11 Thread GitBox
cloud-fan commented on a change in pull request #25295: [SPARK-28560][SQL] Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution URL: https://github.com/apache/spark/pull/25295#discussion_r333946016 ## File path:

[GitHub] [spark] Clarkkkkk commented on a change in pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution

2019-10-11 Thread GitBox
Clark commented on a change in pull request #26090: [SPARK-29302]Fix writing file collision in dynamic partition overwrite mode within speculative execution URL: https://github.com/apache/spark/pull/26090#discussion_r333954828 ## File path:

[GitHub] [spark] tgravescs commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance

2019-10-11 Thread GitBox
tgravescs commented on issue #26085: [SPARK-29434] [Core] Improve the MapStatuses Serialization Performance URL: https://github.com/apache/spark/pull/26085#issuecomment-541067929 I haven't looked at the code yet, can you clarify what ops/ms is measuring here? >> For smaller

[GitHub] [spark] HeartSaVioR commented on a change in pull request #26018: [SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState

2019-10-11 Thread GitBox
HeartSaVioR commented on a change in pull request #26018: [SPARK-29352][SQL][SS] Track active streaming queries in the SparkSession.sharedState URL: https://github.com/apache/spark/pull/26018#discussion_r333997947 ## File path:

[GitHub] [spark] turboFei edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite and specified abs path be unique

2019-10-11 Thread GitBox
turboFei edited a comment on issue #26086: [SPARK-29302] Make the file name of a task for dynamic partition overwrite and specified abs path be unique URL: https://github.com/apache/spark/pull/26086#issuecomment-540982545 Oh, It seems that this issue is related with

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333938629 ## File path:

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333942317 ## File path:

[GitHub] [spark] beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number.

2019-10-11 Thread GitBox
beliefer commented on a change in pull request #25963: [SPARK-28137][SQL] Add Postgresql function to_number. URL: https://github.com/apache/spark/pull/25963#discussion_r333942529 ## File path:

[GitHub] [spark] skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps

2019-10-11 Thread GitBox
skonto edited a comment on issue #25870: [SPARK-27936][K8S] support python deps URL: https://github.com/apache/spark/pull/25870#issuecomment-541001902 @holdenk this is because spark-submit adds the resource to the `sparks.jars` property by default, check bellow: ``` 19/10/11

[GitHub] [spark] hagerf commented on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame

2019-10-11 Thread GitBox
hagerf commented on issue #26087: [SPARK-29427][SQL] Create KeyValueGroupedDataset from existing columns in DataFrame URL: https://github.com/apache/spark/pull/26087#issuecomment-541016870 Thank you for creating a MR so fast after my submitting the jira ticket! 

[GitHub] [spark] turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server

2019-10-11 Thread GitBox
turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#discussion_r333935746 ## File path:

[GitHub] [spark] turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server

2019-10-11 Thread GitBox
turboFei commented on a change in pull request #25797: [SPARK-29043][Core] Improve the concurrent performance of History Server URL: https://github.com/apache/spark/pull/25797#discussion_r333935746 ## File path:

<    1   2   3   4   5   6   7   >