[GitHub] [spark] shahidki31 commented on issue #25460: [SPARK-25474][SQL][Followup] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
shahidki31 commented on issue #25460: [SPARK-25474][SQL][Followup] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#issuecomment-521534572 cc @cloud-fan @dongjoon-hyun

[GitHub] [spark] dilipbiswal edited a comment on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore

2019-08-15 Thread GitBox
dilipbiswal edited a comment on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore URL: https://github.com/apache/spark/pull/25448#issuecomment-521533557 @cloud-fan @dongjoon-hyun @HyukjinKwon Was just checking the db2 definition of a identifier

[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-521533643 Test FAILed. Refer to this link for build results (access rights to CI

[GitHub] [spark] shahidki31 opened a new pull request #25460: [SPARK-25474][SQL][Followup] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
shahidki31 opened a new pull request #25460: [SPARK-25474][SQL][Followup] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460 …ts not available ## What changes were proposed in this pull request? When the table relation

[GitHub] [spark] dilipbiswal edited a comment on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore

2019-08-15 Thread GitBox
dilipbiswal edited a comment on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore URL: https://github.com/apache/spark/pull/25448#issuecomment-521533557 @cloud-fan @dongjoon-hyun Was just checking the db2 definition of a identifier in

[GitHub] [spark] cloud-fan commented on issue #25418: [SPARK-28695][SS] Use CaseInsensitiveMap in KafkaSourceProvider to make source param handling more robust

2019-08-15 Thread GitBox
cloud-fan commented on issue #25418: [SPARK-28695][SS] Use CaseInsensitiveMap in KafkaSourceProvider to make source param handling more robust URL: https://github.com/apache/spark/pull/25418#issuecomment-521534340 thanks, merging to master!

[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type. URL: https://github.com/apache/spark/pull/25458#issuecomment-521533637 Merged build finished. Test FAILed.

[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates URL: https://github.com/apache/spark/pull/25410#issuecomment-521542919 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates URL: https://github.com/apache/spark/pull/25410#issuecomment-521542912 Merged build finished. Test PASSed.

[GitHub] [spark] AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-521552668 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-521552426 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-521552661 Merged build finished. Test PASSed. This is an

[GitHub] [spark] AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-521552668 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-521552661 Merged build finished. Test PASSed. This is an automated

[GitHub] [spark] SparkQA removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-521551264 **[Test build #109151 has

[GitHub] [spark] AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-521552424 Merged build finished. Test FAILed.

[GitHub] [spark] gengliangwang commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

2019-08-15 Thread GitBox
gengliangwang commented on issue #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type URL: https://github.com/apache/spark/pull/25300#issuecomment-521557035 After consideration, I decide to close this one and open

[GitHub] [spark] gengliangwang closed pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type

2019-08-15 Thread GitBox
gengliangwang closed pull request #25300: [SPARK-28503][SQL] Return null result on cast an out-of-range value to a integral type URL: https://github.com/apache/spark/pull/25300 This is an automated message from the Apache

[GitHub] [spark] giamo commented on a change in pull request #24405: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas

2019-08-15 Thread GitBox
giamo commented on a change in pull request #24405: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/24405#discussion_r314221371 ## File path:

[GitHub] [spark] SparkQA commented on issue #24486: [SPARK-27592][SQL] Set the bucketed data source table SerDe correctly

2019-08-15 Thread GitBox
SparkQA commented on issue #24486: [SPARK-27592][SQL] Set the bucketed data source table SerDe correctly URL: https://github.com/apache/spark/pull/24486#issuecomment-521568889 **[Test build #109149 has

[GitHub] [spark] SparkQA removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates URL: https://github.com/apache/spark/pull/25410#issuecomment-521543496 **[Test build #109150 has

[GitHub] [spark] AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates URL: https://github.com/apache/spark/pull/25410#issuecomment-521599617 Merged build finished. Test PASSed. This is an

[GitHub] [spark] ajithme commented on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore

2019-08-15 Thread GitBox
ajithme commented on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore URL: https://github.com/apache/spark/pull/25448#issuecomment-521682625 > Wait, does table name starting with `_` work in Spark currently? From SPARK-19059 it seems supported, but

[GitHub] [spark] srowen closed pull request #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py

2019-08-15 Thread GitBox
srowen closed pull request #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py URL: https://github.com/apache/spark/pull/25406 This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314363301 ## File path:

[GitHub] [spark] squito commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files

2019-08-15 Thread GitBox
squito commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24892#issuecomment-521687011 thanks for the looking into this @vanzin and @cloud-fan . I agree its unnecessarily

[GitHub] [spark] SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521687136 **[Test build #4832 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4832/testReport)** for PR

[GitHub] [spark] brkyvz commented on issue #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

2019-08-15 Thread GitBox
brkyvz commented on issue #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog URL: https://github.com/apache/spark/pull/25402#issuecomment-521691626 Thanks @cloud-fan! This is an automated message

[GitHub] [spark] vanzin commented on issue #24970: [SPARK-23977][SQL] Support High Performance S3A committers [test-hadoop3.2]

2019-08-15 Thread GitBox
vanzin commented on issue #24970: [SPARK-23977][SQL] Support High Performance S3A committers [test-hadoop3.2] URL: https://github.com/apache/spark/pull/24970#issuecomment-521709157 Merging to master. This is an automated

[GitHub] [spark] abellina commented on issue #25403: [SPARK-28679][YARN] changes to setResourceInformation to handle empty resources and reflection error handling

2019-08-15 Thread GitBox
abellina commented on issue #25403: [SPARK-28679][YARN] changes to setResourceInformation to handle empty resources and reflection error handling URL: https://github.com/apache/spark/pull/25403#issuecomment-521714176 test this please

[GitHub] [spark] dongjoon-hyun opened a new pull request #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment

2019-08-15 Thread GitBox
dongjoon-hyun opened a new pull request #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment URL: https://github.com/apache/spark/pull/25466 ## What changes were proposed in this pull request? This PR aims to remove `+` from `Java 8+` in the documentation

[GitHub] [spark] dongjoon-hyun commented on issue #25462: [SPARK-28745][SQL][TEST] Add benchmarks for `extract()`

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25462: [SPARK-28745][SQL][TEST] Add benchmarks for `extract()` URL: https://github.com/apache/spark/pull/25462#issuecomment-521729821 Hi, @MaxGekk . I regenerated the result on EC2 and made a PR to you. Could you review and merge that? -

[GitHub] [spark] srowen commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
srowen commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521733832 Many things going on here. So, we build with Java 8 and test with 11. This should be true for Maven and

[GitHub] [spark] shaneknapp edited a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
shaneknapp edited a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521738315 > Many things going on here. So, we build with Java 8 and test with 11. This should be true for

[GitHub] [spark] brkyvz commented on a change in pull request #25348: [SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths

2019-08-15 Thread GitBox
brkyvz commented on a change in pull request #25348: [SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#discussion_r314377292 ## File path:

[GitHub] [spark] attilapiros commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#discussion_r314388565 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#discussion_r314392192 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

[GitHub] [spark] cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#discussion_r314392192 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

[GitHub] [spark] SparkQA commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
SparkQA commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521709690 **[Test build #4831 has

[GitHub] [spark] attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
attilapiros commented on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521709371 ok to test This is an

[GitHub] [spark] dongjoon-hyun commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521723310 Hi, @shaneknapp . We need to have `\` at the end of `java.version=11`. -

[GitHub] [spark] vanzin closed pull request #24970: [SPARK-23977][SQL] Support High Performance S3A committers [test-hadoop3.2]

2019-08-15 Thread GitBox
vanzin closed pull request #24970: [SPARK-23977][SQL] Support High Performance S3A committers [test-hadoop3.2] URL: https://github.com/apache/spark/pull/24970 This is an automated message from the Apache Git Service. To

[GitHub] [spark] dongjoon-hyun edited a comment on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment URL: https://github.com/apache/spark/pull/25466#issuecomment-521732815 cc @srowen , @kiszk , @dbtsai , @HyukjinKwon , @gatorsmile , @felixcheung

[GitHub] [spark] ulysses-you opened a new pull request #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries

2019-08-15 Thread GitBox
ulysses-you opened a new pull request #25464: [SPARK-28746][SQL] Add partitionby hint for sql queries URL: https://github.com/apache/spark/pull/25464 ## What changes were proposed in this pull request? Now, `RepartitionByExpression` is allowed at Dataset method

[GitHub] [spark] srowen commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py

2019-08-15 Thread GitBox
srowen commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py URL: https://github.com/apache/spark/pull/25406#issuecomment-521681492 Merged to master This is an

[GitHub] [spark] SparkQA removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521649568 **[Test build #4828 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4828/testReport)**

[GitHub] [spark] wangyum commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-521683954 > @wangyum do you mean CommandUtils.getSizeInBytesFallBackToHdfs is very slow if there are many

[GitHub] [spark] ajithme commented on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore

2019-08-15 Thread GitBox
ajithme commented on issue #25448: [SPARK-28697][SQL] Invalidate Database/Table names starting with underscore URL: https://github.com/apache/spark/pull/25448#issuecomment-521683571 I think spark allowed "_" to be used as start char for table/database to fall inline with PostGre, but this

[GitHub] [spark] gengliangwang commented on issue #24405: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas

2019-08-15 Thread GitBox
gengliangwang commented on issue #24405: [SPARK-27506][SQL] Allow deserialization of Avro data using compatible schemas URL: https://github.com/apache/spark/pull/24405#issuecomment-521688184 > cc @dbtsai @gengliangwang do we have this feature in the avro data source? e.g. can we read avro

[GitHub] [spark] dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#issuecomment-521691115 Retest this please.

[GitHub] [spark] cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#discussion_r314391382 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

[GitHub] [spark] cloud-fan commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
cloud-fan commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs URL: https://github.com/apache/spark/pull/25368#issuecomment-521708305 retest this please This is an automated

[GitHub] [spark] SparkQA removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521687136 **[Test build #4832 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4832/testReport)**

[GitHub] [spark] dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#issuecomment-521731966 Thank you for update, @shahidki31 . Hi, @cloud-fan , @maropu , @wangyum ,

[GitHub] [spark] dongjoon-hyun commented on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment URL: https://github.com/apache/spark/pull/25466#issuecomment-521732815 cc @srowen , @kiszk , @dbtsai , @gatorsmile

[GitHub] [spark] shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
shaneknapp commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds URL: https://github.com/apache/spark/pull/25423#issuecomment-521738315 > Many things going on here. So, we build with Java 8 and test with 11. This should be true for Maven

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314357768 ## File path:

[GitHub] [spark] cloud-fan commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314357473 ## File path:

[GitHub] [spark] dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins URL: https://github.com/apache/spark/pull/25443#issuecomment-521680571 @wangyum . Since [I verified Python UT failures are irrelevant to this

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314362751 ## File path:

[GitHub] [spark] squito commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
squito commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#discussion_r314365447 ## File path:

[GitHub] [spark] squito commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
squito commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#discussion_r314353253 ## File path:

[GitHub] [spark] gengliangwang commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow

2019-08-15 Thread GitBox
gengliangwang commented on issue #25461: [SPARK-28741][SQL]Throw exceptions when casting to integers causes overflow URL: https://github.com/apache/spark/pull/25461#issuecomment-521706706 retest this please. This is an

[GitHub] [spark] cloud-fan commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan commented on issue #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#issuecomment-521706641 cc @brkyvz @rdblue @gengliangwang This is an

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25462: [SPARK-28745][SQL][TEST] Add benchmarks for `extract()`

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25462: [SPARK-28745][SQL][TEST] Add benchmarks for `extract()` URL: https://github.com/apache/spark/pull/25462#discussion_r314417211 ## File path: sql/core/benchmarks/ExtractBenchmark-results.txt ## @@ -0,0 +1,145 @@

[GitHub] [spark] dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#issuecomment-521731966 Thank you for update, @shahidki31 . Hi, @cloud-fan , @maropu , @wangyum , @srowen

[GitHub] [spark] tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone

2019-08-15 Thread GitBox
tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#discussion_r314356398 ## File path: core/src/main/scala/org/apache/spark/resource/ResourceInformation.scala

[GitHub] [spark] tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone

2019-08-15 Thread GitBox
tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#discussion_r314355991 ## File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314357247 ## File path:

[GitHub] [spark] ajithme commented on issue #24076: [SPARK-27142] Provide REST API for SQL level information

2019-08-15 Thread GitBox
ajithme commented on issue #24076: [SPARK-27142] Provide REST API for SQL level information URL: https://github.com/apache/spark/pull/24076#issuecomment-521677980 > > Returning plan and metrics can significantly increase the payload. > > Yes, but it's also useful information and

[GitHub] [spark] tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone

2019-08-15 Thread GitBox
tgravescs commented on a change in pull request #25409: [SPARK-28414][WEBUI] UI updates to show resource info in Standalone URL: https://github.com/apache/spark/pull/25409#discussion_r314348119 ## File path: core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala

[GitHub] [spark] SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521681390 **[Test build #4828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4828/testReport)** for

[GitHub] [spark] cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465#discussion_r314389935 ## File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala

[GitHub] [spark] vanzin commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-15 Thread GitBox
vanzin commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r314393643 ## File path:

[GitHub] [spark] BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-15 Thread GitBox
BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r314396389 ## File path:

[GitHub] [spark] SparkQA removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#issuecomment-521674863 **[Test build #4831 has

[GitHub] [spark] BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-15 Thread GitBox
BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r314396389 ## File path:

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession URL: https://github.com/apache/spark/pull/25463#discussion_r314355321 ## File path:

[GitHub] [spark] huaxingao commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py

2019-08-15 Thread GitBox
huaxingao commented on issue #25406: [SPARK-28243][PYSPARK][ML][FOLLOW-UP] Move Python DecisionTreeParams to regression.py URL: https://github.com/apache/spark/pull/25406#issuecomment-521694313 Thanks @srowen @zhengruifeng I will be more careful next time.

[GitHub] [spark] dongjoon-hyun commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29 URL: https://github.com/apache/spark/pull/25455#issuecomment-521694203 Retest this please. This is an automated message from the Apache Git

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#discussion_r314386382 ## File path:

[GitHub] [spark] cloud-fan opened a new pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs

2019-08-15 Thread GitBox
cloud-fan opened a new pull request #25465: [SPARK-28747][SQL] merge the two data source v2 fallback configs URL: https://github.com/apache/spark/pull/25465 ## What changes were proposed in this pull request? Currently we have 2 configs to specify which v2 sources should fallback to

[GitHub] [spark] dongjoon-hyun commented on issue #25462: [SPARK-28745][SQL] Add benchmarks for `extract()`

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25462: [SPARK-28745][SQL] Add benchmarks for `extract()` URL: https://github.com/apache/spark/pull/25462#issuecomment-521710204 Thank you for pinging me, @MaxGekk . Sure! This is an

[GitHub] [spark] cloud-fan commented on a change in pull request #25348: [SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths

2019-08-15 Thread GitBox
cloud-fan commented on a change in pull request #25348: [SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#discussion_r314397529 ## File path:

[GitHub] [spark] vanzin commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-15 Thread GitBox
vanzin commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter URL: https://github.com/apache/spark/pull/25342#discussion_r314393643 ## File path:

[GitHub] [spark] dongjoon-hyun commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25229: [SPARK-27900][K8s] Add jvm oom flag URL: https://github.com/apache/spark/pull/25229#issuecomment-521714839 Although there was a request, but this is a kind of nice-to-have. I mean this is not a blocker because this is not a regression from 2.4.3. We

[GitHub] [spark] shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#discussion_r314404815 ## File path:

[GitHub] [spark] SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521717583 **[Test build #4832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4832/testReport)** for

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#discussion_r314407842 ## File path:

[GitHub] [spark] srowen commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
srowen commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521724613 Hm.. that failure looks unrelated, but I don't see it in master, and I see it failed twice the same way. I don't see any

[GitHub] [spark] shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
shahidki31 commented on a change in pull request #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available URL: https://github.com/apache/spark/pull/25460#discussion_r314416215 ## File path:

[GitHub] [spark] dongjoon-hyun commented on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25466: [MINOR][DOC] Use `Java 8` instead of `Java 8+` as a running environment URL: https://github.com/apache/spark/pull/25466#issuecomment-521744281 Thank you for review and approval, @srowen . I tested the doc generation and attached the screenshot.

[GitHub] [spark] shivsood commented on a change in pull request #25344: [WIP][SPARK-28151][SQL] Mapped ByteType to TinyINT for MsSQLServerDialect

2019-08-15 Thread GitBox
shivsood commented on a change in pull request #25344: [WIP][SPARK-28151][SQL] Mapped ByteType to TinyINT for MsSQLServerDialect URL: https://github.com/apache/spark/pull/25344#discussion_r314445675 ## File path:

[GitHub] [spark] huaxingao commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
huaxingao commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base URL: https://github.com/apache/spark/pull/25371#issuecomment-521759948 retest this please This is an

[GitHub] [spark] zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#discussion_r314471377 ## File path:

[GitHub] [spark] zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#discussion_r314471177 ## File path:

[GitHub] [spark] zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-08-15 Thread GitBox
zsxwing commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#discussion_r314471440 ## File path:

[GitHub] [spark] dilipbiswal commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-15 Thread GitBox
dilipbiswal commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc URL: https://github.com/apache/spark/pull/25459#issuecomment-521807472 @gatorsmile This is after change.. I have not captured all the screen shots..

[GitHub] [spark] wangyum commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
wangyum commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521810612 retest this please This is an automated message from the Apache

[GitHub] [spark] wangyum commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
wangyum commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3 URL: https://github.com/apache/spark/pull/25451#issuecomment-521810580 ![image](https://user-images.githubusercontent.com/5399861/63129354-e2828300-bfe9-11e9-8d21-710236971f81.png)

<    1   2   3   4   5   6   7   8   >