[GitHub] [spark] yaooqinn commented on a change in pull request #26942: [SPARK-30301][SQL] Fix wrong results when datetimes as fields of complex types
yaooqinn commented on a change in pull request #26942: [SPARK-30301][SQL] Fix
wrong results when datetimes as fields of complex types
URL: https://github.com/apache/spark/pull/26942#discussion_r397624096
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##
@@ -56,78 +56,41 @@ object HiveResult {
// We need the types so we can output struct field names
val types = executedPlan.output.map(_.dataType)
// Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(toHiveString)).map(_.mkString("\t"))
+ result.map(_.zip(types).map(e => toHiveString(e)))
+.map(_.mkString("\t"))
}
- private val primitiveTypes = Seq(
-StringType,
-IntegerType,
-LongType,
-DoubleType,
-FloatType,
-BooleanType,
-ByteType,
-ShortType,
-DateType,
-TimestampType,
-BinaryType)
-
private lazy val zoneId =
DateTimeUtils.getZoneId(SQLConf.get.sessionLocalTimeZone)
private lazy val dateFormatter = DateFormatter(zoneId)
private lazy val timestampFormatter =
TimestampFormatter.getFractionFormatter(zoneId)
- /** Hive outputs fields of structs slightly differently than top level
attributes. */
- private def toHiveStructString(a: (Any, DataType)): String = a match {
-case (struct: Row, StructType(fields)) =>
- struct.toSeq.zip(fields).map {
-case (v, t) => s${t.name}":${toHiveStructString((v,
t.dataType))}"""
- }.mkString("{", ",", "}")
-case (seq: Seq[_], ArrayType(typ, _)) =>
- seq.map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
-case (map: Map[_, _], MapType(kType, vType, _)) =>
- map.map {
-case (key, value) =>
- toHiveStructString((key, kType)) + ":" + toHiveStructString((value,
vType))
- }.toSeq.sorted.mkString("{", ",", "}")
-case (null, _) => "null"
-case (s: String, StringType) => "\"" + s + "\""
-case (decimal, DecimalType()) => decimal.toString
-case (interval: CalendarInterval, CalendarIntervalType) =>
- SQLConf.get.intervalOutputStyle match {
-case SQL_STANDARD => toSqlStandardString(interval)
-case ISO_8601 => toIso8601String(interval)
-case MULTI_UNITS => toMultiUnitsString(interval)
- }
-case (other, tpe) if primitiveTypes contains tpe => other.toString
- }
-
/** Formats a datum (based on the given data type) and returns the string
representation. */
- def toHiveString(a: (Any, DataType)): String = a match {
-case (struct: Row, StructType(fields)) =>
- struct.toSeq.zip(fields).map {
-case (v, t) => s${t.name}":${toHiveStructString((v,
t.dataType))}"""
- }.mkString("{", ",", "}")
-case (seq: Seq[_], ArrayType(typ, _)) =>
- seq.map(v => (v, typ)).map(toHiveStructString).mkString("[", ",", "]")
-case (map: Map[_, _], MapType(kType, vType, _)) =>
- map.map {
-case (key, value) =>
- toHiveStructString((key, kType)) + ":" + toHiveStructString((value,
vType))
- }.toSeq.sorted.mkString("{", ",", "}")
-case (null, _) => "NULL"
+ def toHiveString(a: (Any, DataType), nested: Boolean = false): String = a
match {
+case (null, _) => if (nested) "null" else "NULL"
+case (b, BooleanType) => b.toString
case (d: Date, DateType) =>
dateFormatter.format(DateTimeUtils.fromJavaDate(d))
case (t: Timestamp, TimestampType) =>
- DateTimeUtils.timestampToString(timestampFormatter,
DateTimeUtils.fromJavaTimestamp(t))
+ timestampFormatter.format(DateTimeUtils.fromJavaTimestamp(t))
case (bin: Array[Byte], BinaryType) => new String(bin,
StandardCharsets.UTF_8)
Review comment:
```sql
hive> create table bt(k binary);
OK
Time taken: 0.637 seconds
hive> insert into bt values ("spark"), ("hello"), ("3"), ("."), ("0");
Query ID = root_2020032505_1c227d63-47dd-4899-a879-0b6a98269908
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1585115622286_0001, Tracking URL =
http://quickstart.cloudera:8088/proxy/application_1585115622286_0001/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1585115622286_0001
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2020-03-25 05:57:03,557 Stage-1 map = 0%, reduce = 0%
2020-03-25 05:57:09,766 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
1.86 sec
MapReduce Total cumulative CPU time: 1 seconds 860 msec
Ended Job = job_1585115622286_0001
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to:
hdfs://quickstart.cloudera:8020/user/hive/warehouse/bt/.hive-staging_hive_2020-03-25_05-56-54_007_226069574359185383-1/-ext-1
Loading data to table default.bt
Table default.bt stats: [numFiles=1, numRows=5, totalSize=33,
[GitHub] [spark] turboFei commented on a change in pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
turboFei commented on a change in pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#discussion_r397623831 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java ## @@ -112,6 +114,7 @@ public TransportClientFactory( } this.metrics = new NettyMemoryMetrics( this.pooledAllocator, conf.getModuleName() + "-client", conf); +fastFailTimeWindow = conf.ioRetryWaitTimeMs() * 0.95; Review comment: thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603654880 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603654887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25021/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603654887 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25021/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603654880 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603654493 **[Test build #120311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120311/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
AmplabJenkins commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603653204 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
AmplabJenkins removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603653210 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120296/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
AmplabJenkins commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603653210 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120296/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
AmplabJenkins removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603653204 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397621197
##
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
+
Integer.parseInt(conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent"));
int threads =
this.serverThreads() > 0 ? this.serverThreads() : 2 *
NettyRuntime.availableProcessors();
return (int) Math.ceil(threads * (chunkFetchHandlerThreadsPercent /
100.0));
}
+ /**
+ * Whether to use a separate EventLoopGroup to process ChunkFetchRequest
messages, it is decided
+ * by the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is
set or not.
+ */
+ public boolean separateChunkFetchRequest() {
+try {
+ conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent");
Review comment:
Thanks, done in 4f42083.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
SparkQA removed a comment on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603585031 **[Test build #120296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120296/testReport)** for PR 27690 at commit [`073b2e5`](https://github.com/apache/spark/commit/073b2e5aa299225dae830542247be0f15488eba9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage
SparkQA commented on issue #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage URL: https://github.com/apache/spark/pull/27690#issuecomment-603652551 **[Test build #120296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120296/testReport)** for PR 27690 at commit [`073b2e5`](https://github.com/apache/spark/commit/073b2e5aa299225dae830542247be0f15488eba9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603650851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603650859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25020/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603650859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25020/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
Ngone51 commented on a change in pull request #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#discussion_r397618892 ## File path: common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java ## @@ -112,6 +114,7 @@ public TransportClientFactory( } this.metrics = new NettyMemoryMetrics( this.pooledAllocator, conf.getModuleName() + "-client", conf); +fastFailTimeWindow = conf.ioRetryWaitTimeMs() * 0.95; Review comment: I just have another idea to make code a bit more simpler: ```suggestion fastFailTimeWindow = if (conf.maxIORetries() > 0) conf.ioRetryWaitTimeMs() * 0.95 else 0; ``` Then, we can simply the if condition below and call `conf.maxIORetries()` only once. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603650851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile edited a comment on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable
gatorsmile edited a comment on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable URL: https://github.com/apache/spark/pull/27815#issuecomment-603650408 There are three major reasons why I think we should keep HiveContext: - SQLContext will not be removed in the near future, because it is part of [DSV1 APIs](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala). HiveContext is just the corresponding parts with Hive metastore support. - SQLContext/HiveContext is being used as the function parameter in many internal/public libraries. Replace it by SparkSession is not straightforward. It requires massive updates when end users upgrade Spark from 2.x to 3.x. - SQLContext/HiveContext is being documented in various tutorials and books. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603650519 **[Test build #120310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120310/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile edited a comment on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable
gatorsmile edited a comment on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable URL: https://github.com/apache/spark/pull/27815#issuecomment-603650408 There are three major reasons why I think we should keep HiveContext: - SQLContext will not be removed in the near future, because it is part of DSV1 APIs. HiveContext is just the corresponding parts with Hive metastore support. - SQLContext/HiveContext is being used as the function parameter in many internal/public libraries. Replace it by SparkSession is not straightforward. It requires massive updates when end users upgrade Spark from 2.x to 3.x. - SQLContext/HiveContext is being documented in various tutorials and books. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable
gatorsmile commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable URL: https://github.com/apache/spark/pull/27815#issuecomment-603650408 There are two major reasons why I think we should keep HiveContext: - SQLContext will not be removed in the near future, because it is part of DSV1 APIs. HiveContext is just the corresponding parts with Hive metastore support. - SQLContext/HiveContext is being used as the function parameter in many internal/public libraries. Replace it by SparkSession is not straightforward. It requires massive updates when end users upgrade Spark from 2.x to 3.x. - SQLContext/HiveContext is being documented in various tutorials and books. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL]
ResetCommand should reset config to sc.conf only
URL: https://github.com/apache/spark/pull/28003#discussion_r397615611
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -3258,7 +3258,9 @@ class SQLConf extends Serializable with Logging {
}
def clear(): Unit = {
Review comment:
This API has been exposed to users since 2.1, and makes the SQLConf hard to
be restored to its initial state after calling `spark.sessionState.conf.clear`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-603649303 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds`
jiangxb1987 commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds` URL: https://github.com/apache/spark/pull/28007#issuecomment-603647844 Thanks @HyukjinKwon @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable
dongjoon-hyun commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable URL: https://github.com/apache/spark/pull/27815#issuecomment-603647771 Hi, All. Although I have been against this reverting PR until now, I'll not be against any longer if @gatorsmile has his reason to support this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks
jiangxb1987 commented on issue #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks URL: https://github.com/apache/spark/pull/27972#issuecomment-603647722 Thanks @cloud-fan @dongjoon-hyun @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL]
ResetCommand should reset config to sc.conf only
URL: https://github.com/apache/spark/pull/28003#discussion_r397615611
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -3258,7 +3258,9 @@ class SQLConf extends Serializable with Logging {
}
def clear(): Unit = {
Review comment:
This API is exposed to users, and makes the SQLConf hard to be restored to
its initial state after calling `spark.sessionState.conf.clear`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
yaooqinn commented on a change in pull request #28003: [SPARK-31234][SQL]
ResetCommand should reset config to sc.conf only
URL: https://github.com/apache/spark/pull/28003#discussion_r397615611
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -3258,7 +3258,9 @@ class SQLConf extends Serializable with Logging {
}
def clear(): Unit = {
Review comment:
This API is exposed to users, and make the SQLConf hard to restore to its
initial state after calling `spark.sessionState.conf.clear`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
AmplabJenkins commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603647001 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25019/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
AmplabJenkins commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603646994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
AmplabJenkins removed a comment on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603646994 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
AmplabJenkins removed a comment on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603647001 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25019/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
SparkQA commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603646664 **[Test build #120309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120309/testReport)** for PR 28003 at commit [`435e411`](https://github.com/apache/spark/commit/435e4112355a31c158619dd2ca4f05d6d71242f9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #27935: [SPARK-25121][SQL] Supports multi-part table names for broadcast hint resolution
maropu commented on issue #27935: [SPARK-25121][SQL] Supports multi-part table names for broadcast hint resolution URL: https://github.com/apache/spark/pull/27935#issuecomment-603646852 Sure, I'll do that. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
yaooqinn commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-603646632 also, cc @cloud-fan @gatorsmile thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603646163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks
cloud-fan commented on issue #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks URL: https://github.com/apache/spark/pull/27972#issuecomment-603646303 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603646158 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
AmplabJenkins commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603646163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120292/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks
cloud-fan closed pull request #27972: [SPARK-31207][CORE] Ensure the total number of blocks to fetch equals to the sum of local/hostLocal/remote blocks URL: https://github.com/apache/spark/pull/27972 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
AmplabJenkins removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603646158 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
SparkQA removed a comment on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603572691 **[Test build #120292 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120292/testReport)** for PR 27991 at commit [`05dc916`](https://github.com/apache/spark/commit/05dc91695139b1e5f5c94eb117ba34c76bd3a3ff). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type
SparkQA commented on issue #27991: [SPARK-31227][SQL] Non-nullable null type in complex types should not coerce to nullable type URL: https://github.com/apache/spark/pull/27991#issuecomment-603645528 **[Test build #120292 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120292/testReport)** for PR 27991 at commit [`05dc916`](https://github.com/apache/spark/commit/05dc91695139b1e5f5c94eb117ba34c76bd3a3ff). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603644867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603644860 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
SparkQA commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603644822 **[Test build #120308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120308/testReport)** for PR 28005 at commit [`6999fa7`](https://github.com/apache/spark/commit/6999fa79bf7b5aed9878e461d86e8dcdeaf69523). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603644860 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603644867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
SparkQA removed a comment on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603610979 **[Test build #120301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120301/testReport)** for PR 28006 at commit [`384d1e3`](https://github.com/apache/spark/commit/384d1e32f1d3dabc1297ad1cb288fe998ddbbef7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document
HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document URL: https://github.com/apache/spark/pull/27956#discussion_r397613125 ## File path: python/pyspark/sql/streaming.py ## @@ -491,6 +490,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, disables `partition discovery`_. .. _partition discovery: /sql-data-sources-parquet.html#partition-discovery Review comment: Oh, right. The root directory is different when it's released. Sure, seems like it's best to stick to the absolute links then. The syntax itself is fine as a relative link This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
SparkQA commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603644413 **[Test build #120301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120301/testReport)** for PR 28006 at commit [`384d1e3`](https://github.com/apache/spark/commit/384d1e32f1d3dabc1297ad1cb288fe998ddbbef7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397612491
##
File path:
common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java
##
@@ -39,7 +39,7 @@
public class TransportRequestHandlerSuite {
@Test
- public void handleStreamRequest() {
+ public void handleStreamRequest() throws Exception {
Review comment:
Yes, because of the changes for `TransportRequestHandler.handler` here:
https://github.com/apache/spark/pull/27665/files#diff-0e3429029d3f8d49e94ef11e4e3051a2R105
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603643247 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25018/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603643240 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603643247 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25018/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603643240 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document
yaooqinn commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document URL: https://github.com/apache/spark/pull/27956#discussion_r397611490 ## File path: python/pyspark/sql/streaming.py ## @@ -491,6 +490,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, disables `partition discovery`_. .. _partition discovery: /sql-data-sources-parquet.html#partition-discovery Review comment: the links are malformed, please check http://spark.apache.org/docs/3.0.0-preview2/api/python/pyspark.sql.html#module-pyspark.sql.streaming with `partition discovery` content This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
HyukjinKwon commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603642856 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603642160 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120302/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603642154 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
SparkQA removed a comment on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603610962 **[Test build #120302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120302/testReport)** for PR 28005 at commit [`6999fa7`](https://github.com/apache/spark/commit/6999fa79bf7b5aed9878e461d86e8dcdeaf69523). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603642160 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120302/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
AmplabJenkins commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603642154 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document
HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document URL: https://github.com/apache/spark/pull/27956#discussion_r397610671 ## File path: python/pyspark/sql/streaming.py ## @@ -491,6 +490,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, disables `partition discovery`_. .. _partition discovery: /sql-data-sources-parquet.html#partition-discovery Review comment: Sorry for my late response. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document
HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document URL: https://github.com/apache/spark/pull/27956#discussion_r397610591 ## File path: python/pyspark/sql/streaming.py ## @@ -491,6 +490,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, disables `partition discovery`_. .. _partition discovery: /sql-data-sources-parquet.html#partition-discovery Review comment: This is fine - it's a way of reusing a link. It's a relative link This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
SparkQA commented on issue #28005: [SPARK-31231][BUILD][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28005#issuecomment-603641928 **[Test build #120302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120302/testReport)** for PR 28005 at commit [`6999fa7`](https://github.com/apache/spark/commit/6999fa79bf7b5aed9878e461d86e8dcdeaf69523). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document
HyukjinKwon commented on a change in pull request #27956: [SPARK-31189][SQL][DOCS] Fix errors and missing parts for datetime pattern document URL: https://github.com/apache/spark/pull/27956#discussion_r397610591 ## File path: python/pyspark/sql/streaming.py ## @@ -491,6 +490,7 @@ def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None, disables `partition discovery`_. .. _partition discovery: /sql-data-sources-parquet.html#partition-discovery Review comment: This is fine - it's a way of reusing a link This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables
viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid
CHAR type in non-Hive-Serde tables
URL: https://github.com/apache/spark/pull/27902#discussion_r397610314
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -329,4 +331,19 @@ private[sql] object CatalogV2Util {
.getOrElse(catalogManager.v2SessionCatalog)
.asTableCatalog
}
+
+ def failCharType(dt: DataType): Unit = {
+if (HiveStringType.containsCharType(dt)) {
+ throw new AnalysisException(
+"Cannot use CHAR type in non-Hive-Serde tables, please use STRING type
instead.")
+}
+ }
+
+ def assertNoCharTypeInSchema(schema: StructType): Unit = {
+schema.foreach { f =>
+ if (f.metadata.contains(HIVE_TYPE_STRING)) {
+
failCharType(CatalystSqlParser.parseRawDataType(f.metadata.getString(HIVE_TYPE_STRING)))
+ }
Review comment:
So we don't need to detect char type in so many cases?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables
viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid
CHAR type in non-Hive-Serde tables
URL: https://github.com/apache/spark/pull/27902#discussion_r397610129
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala
##
@@ -329,4 +331,19 @@ private[sql] object CatalogV2Util {
.getOrElse(catalogManager.v2SessionCatalog)
.asTableCatalog
}
+
+ def failCharType(dt: DataType): Unit = {
+if (HiveStringType.containsCharType(dt)) {
+ throw new AnalysisException(
+"Cannot use CHAR type in non-Hive-Serde tables, please use STRING type
instead.")
+}
+ }
+
+ def assertNoCharTypeInSchema(schema: StructType): Unit = {
+schema.foreach { f =>
+ if (f.metadata.contains(HIVE_TYPE_STRING)) {
+
failCharType(CatalystSqlParser.parseRawDataType(f.metadata.getString(HIVE_TYPE_STRING)))
+ }
Review comment:
Can we just disallow char type at parser? Like throwing exception in
`visitColType` when we find it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
dongjoon-hyun edited a comment on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27997#issuecomment-603640642 Also, cc @gatorsmile and @cloud-fan since the situation is different from `master` and `branch-3.0`. ``` - commons-compiler/3.0.9//commons-compiler-3.0.9.jar + commons-compiler/3.0.16//commons-compiler-3.0.16.jar ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
dongjoon-hyun commented on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27997#issuecomment-603640642 Also, cc @gatorsmile and @cloud-fan since the situation is different from `master` and `branch-3.0`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables
viirya commented on a change in pull request #27902: [SPARK-31147][SQL] Forbid CHAR type in non-Hive-Serde tables URL: https://github.com/apache/spark/pull/27902#discussion_r397606686 ## File path: docs/sql-migration-guide.md ## @@ -111,6 +111,8 @@ license: | - Since Spark 3.0, `SHOW CREATE TABLE` will always return Spark DDL, even when the given table is a Hive serde table. For generating Hive DDL, please use `SHOW CREATE TABLE AS SERDE` command instead. + - Since Spark 3.0, columns of CHAR type is not allowed in non-Hive-Serde tables, and CREATE/ALTER TABLE commands will fail if CHAR type is detected. Please use STRING type instead. In Spark version 2.4 and earlier, CHAR type is treated as STRING type and the length parameter is simply ignored. Review comment: nit: is -> are This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603640008 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16
dongjoon-hyun commented on issue #27997: [SPARK-31101][BUILD][2.4] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27997#issuecomment-603640263 Hi, @HeartSaVioR . Could you update the PR description, please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
AmplabJenkins removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603640018 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603640018 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120298/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27996: [SPARK-31101][BUILD][3.0] Upgrade Janino to 3.0.16
dongjoon-hyun closed pull request #27996: [SPARK-31101][BUILD][3.0] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27996 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
AmplabJenkins commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603640008 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
SparkQA removed a comment on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603598937 **[Test build #120298 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120298/testReport)** for PR 27943 at commit [`9ff5ca2`](https://github.com/apache/spark/commit/9ff5ca20be633c17ceeabbaad46693da001ed943). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait
SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in the last retry IO wait URL: https://github.com/apache/spark/pull/27943#issuecomment-603639340 **[Test build #120298 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120298/testReport)** for PR 27943 at commit [`9ff5ca2`](https://github.com/apache/spark/commit/9ff5ca20be633c17ceeabbaad46693da001ed943). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27996: [SPARK-31101][BUILD][3.0] Upgrade Janino to 3.0.16
dongjoon-hyun commented on issue #27996: [SPARK-31101][BUILD][3.0] Upgrade Janino to 3.0.16 URL: https://github.com/apache/spark/pull/27996#issuecomment-603639297 Thank you so much, @HeartSaVioR ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-603639160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25017/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-603639154 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603639123 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins removed a comment on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603639129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25016/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-603639154 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-603639160 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25017/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds`
dongjoon-hyun commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds` URL: https://github.com/apache/spark/pull/28007#issuecomment-603638943 +1, LGTM. Thank you, @jiangxb1987 and @HyukjinKwon . cc @holdenk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603639123 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test
dongjoon-hyun commented on issue #28006: [DO-NOT-MERGE][SPARK-31231][BUILD][2.4][FOLLOW-UP] Set the upper bound (before 46.1.0) for setuptools in pip package test URL: https://github.com/apache/spark/pull/28006#issuecomment-603639137 Thank you for working on this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
AmplabJenkins commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603639129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25016/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-603638824 **[Test build #120307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120307/testReport)** for PR 27728 at commit [`44b310e`](https://github.com/apache/spark/commit/44b310e423250c4635943c8e66323f37f03ef75b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
SparkQA commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603638807 **[Test build #120306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120306/testReport)** for PR 27861 at commit [`d369cbc`](https://github.com/apache/spark/commit/d369cbc61811f4334f511fcda9e191a89c040e3c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty
HyukjinKwon commented on issue #27861: [SPARK-30707][SQL]Window function set partitionSpec as order spec when orderSpec is empty URL: https://github.com/apache/spark/pull/27861#issuecomment-603638120 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable
gatorsmile commented on issue #27815: [SPARK-31088][SQL] Add back HiveContext and createExternalTable URL: https://github.com/apache/spark/pull/27815#issuecomment-603638140 Let me correct the typo and rewrite my original reply to make my point clear. Based on the latest discussion in the community, I strongly disagree that we removed these APIs in Spark 3.0 initially. I think we need to add them back. If we want to remove them, we need more public discussions in the community. This PR is just to add the removed APIs back and delay the related discussions to the future releases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28010: [SPARK-26329][CORE][FOLLOW_UP] Fix some small typo
HyukjinKwon commented on issue #28010: [SPARK-26329][CORE][FOLLOW_UP] Fix some small typo URL: https://github.com/apache/spark/pull/28010#issuecomment-603637680 Let's don't fix a couple of nits in a separate PR. If we should, let's do it in a batch as a minor PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds`
HyukjinKwon commented on issue #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds` URL: https://github.com/apache/spark/pull/28007#issuecomment-603637402 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds`
HyukjinKwon closed pull request #28007: [SPARK-31239][CORE][TEST] Increase await duration in `WorkerDecommissionSuite`.`verify a task with all workers decommissioned succeeds` URL: https://github.com/apache/spark/pull/28007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
