[GitHub] [spark] MaxGekk commented on issue #25329: [WIP][SPARK-28596][SQL] Use Java 8 time API in the date_trunc() function
MaxGekk commented on issue #25329: [WIP][SPARK-28596][SQL] Use Java 8 time API in the date_trunc() function URL: https://github.com/apache/spark/pull/25329#issuecomment-517976281 Unfortunately, benchmarks show that this implementation is 2-3 times slower than current one. Most of the time, it spends inside of resolving zone ids to zone offsets. For example, the `truncTimestamp` method is bounded by the binary search by historical zone infos in `DateTimeBenchmark`: https://user-images.githubusercontent.com/1580697/62419955-2221a480-b6a5-11e9-88bd-5da028160c88.png;> I have changed the benchmark to avoid the search on zone history but `truncTimestamp` still spends most of the time in zone conversions. For recent year - `2019`, it is bound by getting zone info from an internal cache: https://user-images.githubusercontent.com/1580697/62420057-fce16600-b6a5-11e9-8e07-ad5d3d7b6d84.png;> I will close the PR since I don't know how to avoid the zone conversions. Maybe in newer versions of JDK, this will be faster, and we will come back to the changes again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25253: [SPARK-28470][SQL] Cast to decimal throws ArithemticException on overflow
viirya commented on a change in pull request #25253: [SPARK-28470][SQL] Cast to
decimal throws ArithemticException on overflow
URL: https://github.com/apache/spark/pull/25253#discussion_r310371161
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
##
@@ -498,22 +499,36 @@ case class Cast(child: Expression, dataType: DataType,
timeZoneId: Option[String
b => x.numeric.asInstanceOf[Numeric[Any]].toInt(b).toByte
}
+ private val nullOnOverflow = SQLConf.get.decimalOperationsNullOnOverflow
+
/**
* Change the precision / scale in a given decimal to those set in
`decimalType` (if any),
- * returning null if it overflows or modifying `value` in-place and
returning it if successful.
+ * modifying `value` in-place and returning it if successful. If an overflow
occurs, it
+ * either returns null or throws an exception according to the value set for
+ * `spark.sql.decimalOperations.nullOnOverflow`.
*
* NOTE: this modifies `value` in-place, so don't call it on external data.
*/
private[this] def changePrecision(value: Decimal, decimalType: DecimalType):
Decimal = {
-if (value.changePrecision(decimalType.precision, decimalType.scale)) value
else null
+if (value.changePrecision(decimalType.precision, decimalType.scale)) {
+ value
+} else {
+ if (nullOnOverflow) {
+null
+ } else {
+throw new ArithmeticException(s"${value.toDebugString} cannot be
represented as " +
+ s"Decimal(${decimalType.precision}, ${decimalType.scale}).")
+ }
+}
}
/**
* Create new `Decimal` with precision and scale given in `decimalType` (if
any),
* returning null if it overflows or creating a new `value` and returning it
if successful.
Review comment:
Change this comment too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC
MaxGekk commented on issue #25336: [SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC URL: https://github.com/apache/spark/pull/25336#issuecomment-517975645 @dongjoon-hyun @srowen @cloud-fan May I ask you to review the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517974031 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517974036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108619/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517974031 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517974036 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108619/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
SparkQA commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517973983 **[Test build #108619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108619/testReport)** for PR 25324 at commit [`f5c5291`](https://github.com/apache/spark/commit/f5c5291123b97fb6807081a2c3a8e2abe04a8735). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
SparkQA removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971687 **[Test build #108619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108619/testReport)** for PR 25324 at commit [`f5c5291`](https://github.com/apache/spark/commit/f5c5291123b97fb6807081a2c3a8e2abe04a8735). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API
AmplabJenkins commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517972515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108614/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API
AmplabJenkins commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517972514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Datafram
AmplabJenkins removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517972514 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Datafram
AmplabJenkins removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517972515 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108614/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API
SparkQA commented on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517972460 **[Test build #108614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108614/testReport)** for PR 25334 at commit [`550246e`](https://github.com/apache/spark/commit/550246e6b6aa98f0c98ab06a87df9f94070c91f6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API
SparkQA removed a comment on issue #25334: [SPARK-28598][SQL][WIP] Few date time manipulation functions does not provide versions supporting Column as input through the Dataframe API URL: https://github.com/apache/spark/pull/25334#issuecomment-517965195 **[Test build #108614 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108614/testReport)** for PR 25334 at commit [`550246e`](https://github.com/apache/spark/commit/550246e6b6aa98f0c98ab06a87df9f94070c91f6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971959 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins removed a comment on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971963 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13706/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
AmplabJenkins commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971959 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971658 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108618/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations
SparkQA commented on issue #25324: [SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup and Tiny Optimizations URL: https://github.com/apache/spark/pull/25324#issuecomment-517971687 **[Test build #108619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108619/testReport)** for PR 25324 at commit [`f5c5291`](https://github.com/apache/spark/commit/f5c5291123b97fb6807081a2c3a8e2abe04a8735). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971656 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
SparkQA removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971404 **[Test build #108618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108618/testReport)** for PR 25256 at commit [`6c61f60`](https://github.com/apache/spark/commit/6c61f60863c2f946b45b64f1e78eebc0797e47aa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13705/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
SparkQA commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971654 **[Test build #108618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108618/testReport)** for PR 25256 at commit [`6c61f60`](https://github.com/apache/spark/commit/6c61f60863c2f946b45b64f1e78eebc0797e47aa). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971656 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971658 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108618/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971636 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13705/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971635 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
SparkQA commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971404 **[Test build #108618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108618/testReport)** for PR 25256 at commit [`6c61f60`](https://github.com/apache/spark/commit/6c61f60863c2f946b45b64f1e78eebc0797e47aa). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971372 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971372 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins removed a comment on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971374 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
AmplabJenkins commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517971374 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13704/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517970877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108600/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517970876 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517970876 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
AmplabJenkins removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517970877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108600/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
SparkQA commented on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517970788 **[Test build #108600 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108600/testReport)** for PR 25214 at commit [`a617812`](https://github.com/apache/spark/commit/a6178122d2284b1302dbed586e1575c483e55b72). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column
SparkQA removed a comment on issue #25214: [SPARK-28461][SQL] Pad Decimal numbers with trailing zeros to the scale of the column URL: https://github.com/apache/spark/pull/25214#issuecomment-517961438 **[Test build #108600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108600/testReport)** for PR 25214 at commit [`a617812`](https://github.com/apache/spark/commit/a6178122d2284b1302dbed586e1575c483e55b72). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT
zhengruifeng commented on issue #25256: [SPARK-28514][ML] Remove the redundant transformImpl method in RF & GBT URL: https://github.com/apache/spark/pull/25256#issuecomment-517970601 good idea, It is reasonable to restrict the `transformImpl` in `ProbabilisticClassificationModel` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #25319: [SPARK-28585][WebUI] Improve WebUI DAG information: Add extra info to rdd from spark plan
xuanyuanking commented on issue #25319: [SPARK-28585][WebUI] Improve WebUI DAG information: Add extra info to rdd from spark plan URL: https://github.com/apache/spark/pull/25319#issuecomment-517970544 Yep, it's a little duplication of SQL tab, thanks for the contribution. For the WholeStageCodeGen, maybe it makes some means to have the extra message, you can try more scenarios. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier()
WeichenXu123 commented on a change in pull request #25235: [SPARK-28483][Core]
Fix canceling a spark job using barrier mode but barrier tasks blocking on
BarrierTaskContext.barrier()
URL: https://github.com/apache/spark/pull/25235#discussion_r310368754
##
File path: core/src/main/scala/org/apache/spark/rpc/RpcEndpointRef.scala
##
@@ -46,6 +46,17 @@ private[spark] abstract class RpcEndpointRef(conf:
SparkConf)
*/
def send(message: Any): Unit
+ /**
+ * Send a message to the corresponding [[RpcEndpoint.receiveAndReply)]] and
return a [[Future]] to
+ * receive the reply within the specified timeout.
+ * Return a `CancelableFuture` instance which wrap `Future` but with
additional `cancel` method.
+ *
+ * This method only sends the message once and never retries.
+ */
+ def askCancelable[T: ClassTag](message: Any, timeout: RpcTimeout):
CancelableFuture[T] = {
Review comment:
@zsxwing Any thoughts ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517969468 **[Test build #108617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108617/testReport)** for PR 25247 at commit [`1c68429`](https://github.com/apache/spark/commit/1c68429e9863250851f50b882fbd80c1352711a3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517969407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13703/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517969406 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517969407 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13703/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517969406 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517968939 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13702/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins removed a comment on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517968938 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
SparkQA commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517968964 **[Test build #108616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108616/testReport)** for PR 25247 at commit [`a8e4004`](https://github.com/apache/spark/commit/a8e4004457492268ae4714c5f7dd4c66ba4f9645). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517968939 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13702/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
AmplabJenkins commented on issue #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables URL: https://github.com/apache/spark/pull/25247#issuecomment-517968938 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r310368022
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.ShowTablesSchema
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing tables.
+ */
+case class ShowTablesExec(catalog: TableCatalog, ident: Identifier, pattern:
Option[String])
+ extends LeafExecNode {
+ override def output: Seq[AttributeReference] = ShowTablesSchema.attributes()
Review comment:
Fixed this and `ShowTablesStatement` -> `ShowTables` as well. Now the
expression ids are the same across the plans:
```
== Analyzed Logical Plan ==
database: string, tableName: string, isTemporary: boolean
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
== Optimized Logical Plan ==
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
== Physical Plan ==
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r310366894
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ShowTablesExec.scala
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalog.v2.{Identifier, TableCatalog}
+import org.apache.spark.sql.catalog.v2.CatalogV2Implicits.NamespaceHelper
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.encoders.RowEncoder
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference,
GenericRowWithSchema}
+import org.apache.spark.sql.catalyst.plans.ShowTablesSchema
+import org.apache.spark.sql.catalyst.util.StringUtils
+import org.apache.spark.sql.execution.LeafExecNode
+
+/**
+ * Physical plan node for showing tables.
+ */
+case class ShowTablesExec(catalog: TableCatalog, ident: Identifier, pattern:
Option[String])
+ extends LeafExecNode {
+ override def output: Seq[AttributeReference] = ShowTablesSchema.attributes()
Review comment:
Fixed this and `ShowTablesStatement` -> `ShowTables` as well. Now the
`explain` displays the plans correctly (before, outputs were not displayed
because of the mismatches):
```
== Analyzed Logical Plan ==
database: string, tableName: string, isTemporary: boolean
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
== Optimized Logical Plan ==
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
== Physical Plan ==
ShowTables [database#19, tableName#20, isTemporary#21],
org.apache.spark.sql.sources.v2.TestInMemoryTableCatalog@3ce7394f, `db`
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of Unsupport
AmplabJenkins removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517968146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodi
AmplabJenkins commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517968145 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodi
AmplabJenkins commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517968146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of Unsupport
AmplabJenkins removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517968145 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEnco
SparkQA removed a comment on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517961440 **[Test build #108612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108612/testReport)** for PR 25335 at commit [`94bb1d6`](https://github.com/apache/spark/commit/94bb1d6b9caadb15b5e50c1710a488ca386597b7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingExce
SparkQA commented on issue #25335: [SPARK-28601][CORE][SQL][test-hadoop3.2] Use StandardCharsets.UTF_8 instead of "UTF-8" string representation, and get rid of UnsupportedEncodingException URL: https://github.com/apache/spark/pull/25335#issuecomment-517968079 **[Test build #108612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108612/testReport)** for PR 25335 at commit [`94bb1d6`](https://github.com/apache/spark/commit/94bb1d6b9caadb15b5e50c1710a488ca386597b7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517967699 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517967700 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108607/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517967700 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108607/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517967699 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517961468 **[Test build #108607 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108607/testReport)** for PR 25310 at commit [`abe9701`](https://github.com/apache/spark/commit/abe9701cca5ff4dd00ede5bc3fec1bd5fa9154cd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517967623 **[Test build #108607 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108607/testReport)** for PR 25310 at commit [`abe9701`](https://github.com/apache/spark/commit/abe9701cca5ff4dd00ede5bc3fec1bd5fa9154cd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967420 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108615/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967419 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
SparkQA commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967418 **[Test build #108615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108615/testReport)** for PR 25348 at commit [`59094e9`](https://github.com/apache/spark/commit/59094e93c45ec2d2ad9e629d364715d430149fb7). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `sealed trait SupportsV1Write extends V2TableWriteExec ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967419 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
SparkQA removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967030 **[Test build #108615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108615/testReport)** for PR 25348 at commit [`59094e9`](https://github.com/apache/spark/commit/59094e93c45ec2d2ad9e629d364715d430149fb7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967420 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108615/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #25247: [SPARK-28319][SQL] Implement SHOW TABLES for Data Source V2 Tables
imback82 commented on a change in pull request #25247: [SPARK-28319][SQL]
Implement SHOW TABLES for Data Source V2 Tables
URL: https://github.com/apache/spark/pull/25247#discussion_r310367491
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
##
@@ -1646,4 +1651,70 @@ class DataSourceV2SQLSuite extends QueryTest with
SharedSQLContext with BeforeAn
}
}
}
+
+ test("ShowTables using v2 catalog") {
+spark.sql("CREATE TABLE testcat.db.table_name (id bigint, data string)
USING foo")
+spark.sql("CREATE TABLE testcat.n1.n2.db.table_name (id bigint, data
string) USING foo")
+
+{
Review comment:
Fixed as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967231 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins removed a comment on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13701/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #25333: [SPARK-28597][SS]Spark streaming terminated when close meta data log error
xuanyuanking commented on a change in pull request #25333:
[SPARK-28597][SS]Spark streaming terminated when close meta data log error
URL: https://github.com/apache/spark/pull/25333#discussion_r310367408
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala
##
@@ -114,12 +116,16 @@ class HDFSMetadataLog[T <: AnyRef :
ClassTag](sparkSession: SparkSession, path:
}
}
- /** Write a batch to a temp file then rename it to the batch file.
- *
- * There may be multiple [[HDFSMetadataLog]] using the same metadata path.
Although it is not a
- * valid behavior, we still need to prevent it from destroying the files.
- */
private def writeBatchToFile(metadata: T, path: Path): Unit = {
+writeBatchToFileWithRetries(metadata, path, 1)
+ }
+
+ /** Write a batch to a temp file then rename it to the batch file.
+*
Review comment:
nit: indent, use the java style comment here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967232 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13701/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
AmplabJenkins commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967231 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths
SparkQA commented on issue #25348: [RFC][SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths URL: https://github.com/apache/spark/pull/25348#issuecomment-517967030 **[Test build #108615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108615/testReport)** for PR 25348 at commit [`59094e9`](https://github.com/apache/spark/commit/59094e93c45ec2d2ad9e629d364715d430149fb7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966119 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108608/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
AmplabJenkins removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517966102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108610/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966117 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
AmplabJenkins removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517966101 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966119 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108608/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966117 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
AmplabJenkins commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517966102 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108610/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517961439 **[Test build #108608 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108608/testReport)** for PR 25310 at commit [`3b675ff`](https://github.com/apache/spark/commit/3b675ff8a686f82cc7d56183a0a0c1b7caba7de1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966083 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108609/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966085 **[Test build #108608 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108608/testReport)** for PR 25310 at commit [`3b675ff`](https://github.com/apache/spark/commit/3b675ff8a686f82cc7d56183a0a0c1b7caba7de1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
AmplabJenkins commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517966101 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-517966027 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108597/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
SparkQA removed a comment on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517961456 **[Test build #108610 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108610/testReport)** for PR 25338 at commit [`4b0f608`](https://github.com/apache/spark/commit/4b0f60864e611fce063ffd733a4e53689435f422). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966081 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <>
SparkQA commented on issue #25338: [MINOR] Fix typos in comments and replace an explicit type with <> URL: https://github.com/apache/spark/pull/25338#issuecomment-517966069 **[Test build #108610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108610/testReport)** for PR 25338 at commit [`4b0f608`](https://github.com/apache/spark/commit/4b0f60864e611fce063ffd733a4e53689435f422). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966083 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108609/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517966081 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template
SparkQA removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template URL: https://github.com/apache/spark/pull/25310#issuecomment-517961454 **[Test build #108609 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108609/testReport)** for PR 25310 at commit [`b4d56b2`](https://github.com/apache/spark/commit/b4d56b20f9b83edea918914221c35b32805495b5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION
AmplabJenkins removed a comment on issue #25294: [WIP][SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION URL: https://github.com/apache/spark/pull/25294#issuecomment-517966026 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
