[GitHub] [spark] rrusso2007 commented on a change in pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
rrusso2007 commented on a change in pull request #24672: [SPARK-27801]
InMemoryFileIndex.listLeafFiles should use listLocatedStatus for
DistributedFileSystem
URL: https://github.com/apache/spark/pull/24672#discussion_r286323325
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##
@@ -274,7 +275,21 @@ object InMemoryFileIndex extends Logging {
// [SPARK-17599] Prevent InMemoryFileIndex from failing if path doesn't
exist
// Note that statuses only include FileStatus for the files and dirs
directly under path,
// and does not include anything else recursively.
-val statuses = try fs.listStatus(path) catch {
+val statuses: Array[FileStatus] = try {
+ fs match {
+// DistributedFileSystem overrides listLocatedStatus to make 1 single
call to namenode
+// to retrieve the file status with the file block location. The
reason to still fallback
+// to listStatus is because the default implementation would
potentially throw a
+// FileNotFoundException which is better handled by doing the lookups
manually below.
Review comment:
I believe that should not matter here as I tried to preserve the original
logic (which would be your new logic now) when using listStatus which has that
issue. The only real change here is using the
DistributedFileSystem.listLocatedStatus and there should be no case where
FileNotFoundException happens when using that as they will all be
LocatedFileStatus that are returned in the array.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24632: [SPARK-27755][BUILD] Update zstd-jni to 1.4.0-1
dongjoon-hyun edited a comment on issue #24632: [SPARK-27755][BUILD] Update zstd-jni to 1.4.0-1 URL: https://github.com/apache/spark/pull/24632#issuecomment-494662915 Hi, @melin . Could you write in English? This is mainly for Hadoop 3 profile in Apache Spark. Spark 3.0 will support Hadoop 3.2.0+. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
JoshRosen commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494662954 This is an awesome find . Fellow readers / reviewers, definitely check out the JIRA description, too, since it lists some impressive perf. numbers: https://issues.apache.org/jira/browse/SPARK-27801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24632: [SPARK-27755][BUILD] Update zstd-jni to 1.4.0-1
dongjoon-hyun commented on issue #24632: [SPARK-27755][BUILD] Update zstd-jni to 1.4.0-1 URL: https://github.com/apache/spark/pull/24632#issuecomment-494662915 Hi, @melin . Could you write in English? This is mainly for Hadoop 3 profile in Apache Spark. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #24673: [SPARK-27737][FOLLOW-UP][SQL][test-hadoop3.2] Update Hive test jars from 2.3.4 to 2.3.5
wangyum opened a new pull request #24673: [SPARK-27737][FOLLOW-UP][SQL][test-hadoop3.2] Update Hive test jars from 2.3.4 to 2.3.5 URL: https://github.com/apache/spark/pull/24673 ## What changes were proposed in this pull request? This pr update `hive-contrib-2.3.4.jar` to `hive-contrib-2.3.5.jar` and `hive-hcatalog-core-2.3.4.jar` to `hive-hcatalog-core-2.3.5.jar`. ## How was this patch tested? Existing test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on a change in pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
JoshRosen commented on a change in pull request #24672: [SPARK-27801]
InMemoryFileIndex.listLeafFiles should use listLocatedStatus for
DistributedFileSystem
URL: https://github.com/apache/spark/pull/24672#discussion_r286322473
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##
@@ -274,7 +275,21 @@ object InMemoryFileIndex extends Logging {
// [SPARK-17599] Prevent InMemoryFileIndex from failing if path doesn't
exist
// Note that statuses only include FileStatus for the files and dirs
directly under path,
// and does not include anything else recursively.
-val statuses = try fs.listStatus(path) catch {
+val statuses: Array[FileStatus] = try {
+ fs match {
+// DistributedFileSystem overrides listLocatedStatus to make 1 single
call to namenode
+// to retrieve the file status with the file block location. The
reason to still fallback
+// to listStatus is because the default implementation would
potentially throw a
+// FileNotFoundException which is better handled by doing the lookups
manually below.
Review comment:
FYI, over at https://github.com/apache/spark/pull/24668 I'm proposing a
change to the `FileNotFoundException` handling logic here: after my patch's
changes, we'll only ignore `FileNotFoundException` if we are not listing the
root directory of the table (i.e. we are listing a subfolder) _and_ the
`ignoreMissingFiles` setting has not been set.
Does that change have any implications for this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on a change in pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
JoshRosen commented on a change in pull request #24672: [SPARK-27801]
InMemoryFileIndex.listLeafFiles should use listLocatedStatus for
DistributedFileSystem
URL: https://github.com/apache/spark/pull/24672#discussion_r286322473
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##
@@ -274,7 +275,21 @@ object InMemoryFileIndex extends Logging {
// [SPARK-17599] Prevent InMemoryFileIndex from failing if path doesn't
exist
// Note that statuses only include FileStatus for the files and dirs
directly under path,
// and does not include anything else recursively.
-val statuses = try fs.listStatus(path) catch {
+val statuses: Array[FileStatus] = try {
+ fs match {
+// DistributedFileSystem overrides listLocatedStatus to make 1 single
call to namenode
+// to retrieve the file status with the file block location. The
reason to still fallback
+// to listStatus is because the default implementation would
potentially throw a
+// FileNotFoundException which is better handled by doing the lookups
manually below.
Review comment:
FYI, over at https://github.com/apache/spark/pull/24668 I'm proposing a
change to the `FileNotFoundException` handling logic here: after my patch's
changes, we'll only ignore `FileNotFoundException` if we are not listing the
root directory of the table (i.e. we are listing a subfolder) _and_ the
`ignoreFile` setting has not been set.
Does that change have any implications for this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
WangGuangxin commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494662341 The build failure was because the type of `option` has been changed in lastest master. Fixed by rebase with master. Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] JoshRosen commented on a change in pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
JoshRosen commented on a change in pull request #24672: [SPARK-27801]
InMemoryFileIndex.listLeafFiles should use listLocatedStatus for
DistributedFileSystem
URL: https://github.com/apache/spark/pull/24672#discussion_r286322473
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
##
@@ -274,7 +275,21 @@ object InMemoryFileIndex extends Logging {
// [SPARK-17599] Prevent InMemoryFileIndex from failing if path doesn't
exist
// Note that statuses only include FileStatus for the files and dirs
directly under path,
// and does not include anything else recursively.
-val statuses = try fs.listStatus(path) catch {
+val statuses: Array[FileStatus] = try {
+ fs match {
+// DistributedFileSystem overrides listLocatedStatus to make 1 single
call to namenode
+// to retrieve the file status with the file block location. The
reason to still fallback
+// to listStatus is because the default implementation would
potentially throw a
+// FileNotFoundException which is better handled by doing the lookups
manually below.
Review comment:
FYI, over at https://github.com/apache/spark/pull/24668 I'm proposing a
change to the `FileNotFoundException` handling logic here: after my patch's
changes, we'll only ignore `FileNotFoundException` if (a) we are not listing
the root directory of the table (i.e. we are listing a subfolder) and (b) the
`ignoreFile` setting has not been set.
Does that change have any implications for this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494661207 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
AmplabJenkins removed a comment on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494660785 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
AmplabJenkins removed a comment on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494660640 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494660785 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
AmplabJenkins commented on issue #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672#issuecomment-494660640 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rrusso2007 opened a new pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem
rrusso2007 opened a new pull request #24672: [SPARK-27801] InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem URL: https://github.com/apache/spark/pull/24672 ## What changes were proposed in this pull request? InMemoryFileIndex.listLeafFiles should use listLocatedStatus for DistributedFileSystem. DistributedFileSystem overrides the listLocatedStatus method in order to do it with 1 single namenode call thus saving thousands of calls to getBlockLocations. ## How was this patch tested? test suite ran Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
SparkQA removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494626419 **[Test build #105651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105651/testReport)** for PR 24647 at commit [`e3014cb`](https://github.com/apache/spark/commit/e3014cb98163ba837e5101790e62afe30246142e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494657061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105651/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins removed a comment on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494657054 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494657054 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
AmplabJenkins commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494657061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105651/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource.
SparkQA commented on issue #24647: [SPARK-27776][SQL]Avoid duplicate Java reflection in DataSource. URL: https://github.com/apache/spark/pull/24647#issuecomment-494656635 **[Test build #105651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105651/testReport)** for PR 24647 at commit [`e3014cb`](https://github.com/apache/spark/commit/e3014cb98163ba837e5101790e62afe30246142e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494655860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105665/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494655855 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494655841 **[Test build #105665 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105665/testReport)** for PR 24043 at commit [`e84cdba`](https://github.com/apache/spark/commit/e84cdba59c36a37589792a0ea0e5e21542978f0f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494655860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105665/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
SparkQA removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494654247 **[Test build #105665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105665/testReport)** for PR 24043 at commit [`e84cdba`](https://github.com/apache/spark/commit/e84cdba59c36a37589792a0ea0e5e21542978f0f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494655855 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494655205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494655209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10919/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494655209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10919/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494655205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494654263 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
AmplabJenkins removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494654266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105650/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494654263 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494654231 **[Test build #105664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105664/testReport)** for PR 24344 at commit [`9060b0c`](https://github.com/apache/spark/commit/9060b0c36806aeb4c823306e0284cc412476d7c3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
AmplabJenkins commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494654266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105650/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494654247 **[Test build #105665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105665/testReport)** for PR 24043 at commit [`e84cdba`](https://github.com/apache/spark/commit/e84cdba59c36a37589792a0ea0e5e21542978f0f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
SparkQA removed a comment on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494622177 **[Test build #105650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105650/testReport)** for PR 24637 at commit [`7a790ff`](https://github.com/apache/spark/commit/7a790ff3bfb1acf0835d1a0f4513df4c02ca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494653875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494653881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10918/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode
SparkQA commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-494653959 **[Test build #105650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105650/testReport)** for PR 24637 at commit [`7a790ff`](https://github.com/apache/spark/commit/7a790ff3bfb1acf0835d1a0f4513df4c02ca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494653881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10918/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494653875 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494653052 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494653058 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105648/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
dongjoon-hyun commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-494653101 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494653052 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494653058 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105648/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
SparkQA removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494619439 **[Test build #105648 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105648/testReport)** for PR 24668 at commit [`24ad834`](https://github.com/apache/spark/commit/24ad834e262bdb5453379937400e76f0bbec4d85). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on a change in pull request #24628:
[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
URL: https://github.com/apache/spark/pull/24628#discussion_r286313546
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##
@@ -551,7 +557,11 @@ class HiveThriftBinaryServerSuite extends
HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ if (HiveUtils.isHive23) {
+assert(conf.get("spark.sql.hive.version") === Some("2.3.4"))
Review comment:
Yep. `2.3.4` causes UT failure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
SparkQA commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494652725 **[Test build #105648 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105648/testReport)** for PR 24668 at commit [`24ad834`](https://github.com/apache/spark/commit/24ad834e262bdb5453379937400e76f0bbec4d85). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494651969 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494651969 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494651972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105649/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
AmplabJenkins removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494651972 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105649/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
dongjoon-hyun closed pull request #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
SparkQA removed a comment on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494620856 **[Test build #105649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105649/testReport)** for PR 24525 at commit [`3fd4cc5`](https://github.com/apache/spark/commit/3fd4cc5aae6ea01514cbd35267c0f412d5ee4f8a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing
SparkQA commented on issue #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing URL: https://github.com/apache/spark/pull/24525#issuecomment-494651574 **[Test build #105649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105649/testReport)** for PR 24525 at commit [`3fd4cc5`](https://github.com/apache/spark/commit/3fd4cc5aae6ea01514cbd35267c0f412d5ee4f8a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494651201 Merged to `master/2.4/2.3`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494651166 This is a documentation change and I verified manually. ``` spark-sql> DESCRIBE FUNCTION EXTENDED ^; Function: ^ Class: org.apache.spark.sql.catalyst.expressions.BitwiseXor Usage: expr1 ^ expr2 - Returns the result of bitwise exclusive OR of `expr1` and `expr2`. Extended Usage: Examples: > SELECT 3 ^ 5; 6 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494650258 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105656/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494650252 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
SparkQA commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494650414 **[Test build #105663 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105663/testReport)** for PR 24669 at commit [`82e6528`](https://github.com/apache/spark/commit/82e6528076dd8d8259d3dc22baf0f14f054a595d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494650258 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105656/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
SparkQA removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494630568 **[Test build #105656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105656/testReport)** for PR 24628 at commit [`38ae34e`](https://github.com/apache/spark/commit/38ae34e1daf8fd543a8bf9eef568ba5d0541a8f4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494650092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10917/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494650088 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494650252 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
AmplabJenkins commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494650088 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
AmplabJenkins commented on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494650092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10917/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494650085 **[Test build #105656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105656/testReport)** for PR 24628 at commit [`38ae34e`](https://github.com/apache/spark/commit/38ae34e1daf8fd543a8bf9eef568ba5d0541a8f4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor
AmplabJenkins removed a comment on issue #24669: [SPARK-27800][SQL][DOC] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494627280 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL] Fix wrong answer of example for BitwiseXor
dongjoon-hyun commented on issue #24669: [SPARK-27800][SQL] Fix wrong answer of example for BitwiseXor URL: https://github.com/apache/spark/pull/24669#issuecomment-494649846 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-494648867 So, the files under `v2.3.5` are from Hive `2.3.5`, not Hive `2.3.4`, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on a change in pull request #24628:
[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
URL: https://github.com/apache/spark/pull/24628#discussion_r286309454
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##
@@ -551,7 +557,11 @@ class HiveThriftBinaryServerSuite extends
HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ if (HiveUtils.isHive23) {
+assert(conf.get("spark.sql.hive.version") === Some("2.3.4"))
Review comment:
`Some("2.3.5")`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on a change in pull request #24628:
[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
URL: https://github.com/apache/spark/pull/24628#discussion_r286309339
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##
@@ -484,10 +483,13 @@ class HiveThriftBinaryServerSuite extends
HiveThriftJdbcTest {
withMultipleConnectionJdbcStatement("smallKV", "addJar")(
{
statement =>
+ val jarName = if (HiveUtils.isHive23) {
+"hive-hcatalog-core-2.3.4.jar"
Review comment:
ditto. `"hive-hcatalog-core-2.3.5.jar"`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on a change in pull request #24628:
[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
URL: https://github.com/apache/spark/pull/24628#discussion_r286309409
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
##
@@ -538,7 +540,11 @@ class HiveThriftBinaryServerSuite extends
HiveThriftJdbcTest {
conf += resultSet.getString(1) -> resultSet.getString(2)
}
- assert(conf.get("spark.sql.hive.version") === Some("1.2.1"))
+ if (HiveUtils.isHive23) {
+assert(conf.get("spark.sql.hive.version") === Some("2.3.4"))
Review comment:
`Some("2.3.5")`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
dongjoon-hyun commented on a change in pull request #24628:
[SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
URL: https://github.com/apache/spark/pull/24628#discussion_r286309260
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
##
@@ -200,10 +201,13 @@ class CliSuite extends SparkFunSuite with
BeforeAndAfterAll with Logging {
}
test("Commands using SerDe provided in --jars") {
+val jarName = if (HiveUtils.isHive23) {
+ "hive-hcatalog-core-2.3.4.jar"
Review comment:
`"hive-hcatalog-core-2.3.5.jar"`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647691 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10916/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types
dongjoon-hyun commented on issue #21586: [SPARK-24586][SQL] Upcast should not allow casting from string to other types URL: https://github.com/apache/spark/pull/21586#issuecomment-494647942 cc @dbtsai This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647681 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647691 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/10916/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647681 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24664: [SPARK-27794][R][DOCS] Use https URL for CRAN repo
dongjoon-hyun commented on a change in pull request #24664:
[SPARK-27794][R][DOCS] Use https URL for CRAN repo
URL: https://github.com/apache/spark/pull/24664#discussion_r286308360
##
File path: dev/create-release/spark-rm/Dockerfile
##
@@ -42,8 +42,8 @@ ARG PIP_PKGS="pyopenssl pypandoc numpy pygments sphinx"
#
# This is all in a single "RUN" command so that if anything changes, "apt
update" is run to fetch
# the most current package versions (instead of potentially using old versions
cached by docker).
-RUN apt-get clean && apt-get update && $APT_INSTALL gnupg && \
- echo 'deb http://cran.cnr.Berkeley.edu/bin/linux/ubuntu bionic-cran35/' >>
/etc/apt/sources.list && \
+RUN apt-get clean && apt-get update && $APT_INSTALL gnupg && $APT_INSTALL
ca-certificates \
Review comment:
@srowen . This fails because `&&` is missing at the end of the line.
```
E: Unable to locate package echo
E: Unable to locate package deb [trusted=yes]
https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35
E: Couldn't find any package by glob 'deb [trusted=yes]
https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35'
E: Couldn't find any package by regex 'deb [trusted=yes]
https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35'
The command '/bin/sh -c apt-get clean && apt-get update && $APT_INSTALL
gnupg && $APT_INSTALL ca-certificates echo 'deb [trusted=yes]
https://cloud.r-project.org/bin/linux/ubuntu bionic-cran35/' >>
/etc/apt/sources.list && gpg --keyserver keyserver.ubuntu.com --recv-key
E084DAB9 && gpg -a --export E084DAB9 | apt-key add - && apt-get clean &&
rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get update &&
$APT_INSTALL software-properties-common && apt-add-repository -y
ppa:brightbox/ruby-ng && apt-get update && $APT_INSTALL openjdk-8-jdk &&
update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
&& $APT_INSTALL curl wget git maven ivy subversion make gcc lsof libffi-dev
pandoc pandoc-citeproc libssl-dev libcurl4-openssl-dev libxml2-dev && curl
-sL https://deb.nodesource.com/setup_11.x | bash && $APT_INSTALL nodejs &&
$APT_INSTALL libpython2.7-dev libpython3-dev python-pip python3-pip && pip
install $BASE_PIP_PKGS && pip install $PIP_PKGS && cd && virtualenv -p
python3 /opt/p35 && . /opt/p35/bin/activate && pip install $BASE_PIP_PKGS
&& pip install $PIP_PKGS && $APT_INSTALL r-base r-base-dev &&
$APT_INSTALL texlive-latex-base texlive texlive-fonts-extra texinfo qpdf &&
Rscript -e "install.packages(c('curl', 'xml2', 'httr', 'devtools', 'testthat',
'knitr', 'rmarkdown', 'roxygen2', 'e1071', 'survival'),
repos='https://cloud.r-project.org/')" && Rscript -e
"devtools::install_github('jimhester/lintr')" && $APT_INSTALL ruby2.3
ruby2.3-dev mkdocs && gem install jekyll --no-rdoc --no-ri && gem install
jekyll-redirect-from && gem install pygments.rb' returned a non-zero code: 100
```
After fixing like `$APT_INSTALL ca-certificates \` -> `$APT_INSTALL
ca-certificates && \`, you can test with the following command.
```
$ cd dev/create-release
$ docker build -t "test" spark-rm
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647087 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105662/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647087 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105662/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647082 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647077 **[Test build #105662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105662/testReport)** for PR 24344 at commit [`c481c6e`](https://github.com/apache/spark/commit/c481c6e9b5c5eb1f438d1e6387c2f2900d644e3c). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA removed a comment on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494646768 **[Test build #105662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105662/testReport)** for PR 24344 at commit [`c481c6e`](https://github.com/apache/spark/commit/c481c6e9b5c5eb1f438d1e6387c2f2900d644e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
AmplabJenkins commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494647082 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled
HyukjinKwon closed pull request #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled URL: https://github.com/apache/spark/pull/24650 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery
SparkQA commented on issue #24344: [SPARK-27440][SQL] Optimize uncorrelated predicate subquery URL: https://github.com/apache/spark/pull/24344#issuecomment-494646768 **[Test build #105662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105662/testReport)** for PR 24344 at commit [`c481c6e`](https://github.com/apache/spark/commit/c481c6e9b5c5eb1f438d1e6387c2f2900d644e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ben-manes commented on issue #24585: [Spark-27664][SQL] Performance issue while listing large number of files on an object store.
ben-manes commented on issue #24585: [Spark-27664][SQL] Performance issue while listing large number of files on an object store. URL: https://github.com/apache/spark/pull/24585#issuecomment-494646554 fyi, [Caffeine](https://github.com/ben-manes/caffeine) is a rewrite of Guava's cache that does not have this limitation due to decoupling the eviction policy from the hash table. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled
HyukjinKwon commented on issue #24650: [SPARK-27778][PYTHON] Fix toPandas conversion of empty DataFrame with Arrow enabled URL: https://github.com/apache/spark/pull/24650#issuecomment-494646437 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494642642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105655/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
SparkQA removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494629294 **[Test build #105655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105655/testReport)** for PR 24668 at commit [`0c1eba3`](https://github.com/apache/spark/commit/0c1eba3ff813586973e6bfbb661f9cbe6686d5a0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins removed a comment on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494642639 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494642642 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105655/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
AmplabJenkins commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494642639 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles
SparkQA commented on issue #24668: [SPARK-27676][SQL][SS] InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles URL: https://github.com/apache/spark/pull/24668#issuecomment-494642536 **[Test build #105655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105655/testReport)** for PR 24668 at commit [`0c1eba3`](https://github.com/apache/spark/commit/0c1eba3ff813586973e6bfbb661f9cbe6686d5a0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
