[GitHub] [spark] AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588712511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
AmplabJenkins removed a comment on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588712461 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588712511 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
AmplabJenkins commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588712461 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
dongjoon-hyun commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588710588 cc @cloud-fan and @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
SparkQA commented on issue #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643#issuecomment-588710457 **[Test build #118702 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118702/testReport)** for PR 27643 at commit [`442d5da`](https://github.com/apache/spark/commit/442d5daddf3cb24bec728ae0cf862676cd1e38e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
dongjoon-hyun commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588708800 Hi, All. Please review the following together. - https://github.com/apache/spark/pull/27643 SPARK-30886 Warn two-parameter TRIM/LTRIM/RTRIM function usages This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages
dongjoon-hyun opened a new pull request #27643: [SPARK-30886][SQL] Warn two-parameter TRIM/LTRIM/RTRIM function usages URL: https://github.com/apache/spark/pull/27643 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
maropu commented on a change in pull request #27638: [SPARK-30878][SQL][DOC]
Improve the CREATE TABLE document
URL: https://github.com/apache/spark/pull/27638#discussion_r381828881
##
File path: docs/sql-ref-syntax-ddl-create-table-hiveformat.md
##
@@ -96,21 +97,37 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ]
table_identifier
### Examples
{% highlight sql %}
---Using Comment and loading data from another table into the created table
-CREATE TABLE StudentInfo
Review comment:
yea, ok. thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381826963
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
Good point, let me explore this route. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27499: [SPARK-30590][SQL] Untyped select API cannot take typed column expression
cloud-fan commented on a change in pull request #27499: [SPARK-30590][SQL]
Untyped select API cannot take typed column expression
URL: https://github.com/apache/spark/pull/27499#discussion_r381826270
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala
##
@@ -394,4 +403,21 @@ class DatasetAggregatorSuite extends QueryTest with
SharedSparkSession {
checkAnswer(group, Row("bob", Row(true, 3)) :: Nil)
checkDataset(group.as[OptionBooleanIntData], OptionBooleanIntData("bob",
Some((true, 3
}
+
+ test("SPARK-30590: select multiple typed column expressions") {
+val df = Seq((1, 2, 3, 4, 5, 6)).toDF("a", "b", "c", "d", "e", "f")
+val fooAgg = (i: Int) => FooAgg(i).toColumn.name(s"foo_agg_$i")
+
+val agg1 = df.select(fooAgg(1), fooAgg(2), fooAgg(3), fooAgg(4), fooAgg(5))
+checkDataset(agg1, (3, 5, 7, 9, 11))
+
+val agg2 = df.selectUntyped(fooAgg(1), fooAgg(2), fooAgg(3), fooAgg(4),
fooAgg(5), fooAgg(6))
+ .asInstanceOf[Dataset[(Int, Int, Int, Int, Int, Int)]]
+checkDataset(agg2, (3, 5, 7, 9, 11, 13))
+
+val err = intercept[AnalysisException] {
+ df.select(fooAgg(1), fooAgg(2), fooAgg(3), fooAgg(4), fooAgg(5),
fooAgg(6))
Review comment:
We can open a JIRA first.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381825279
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
I think `V2SessionCatalog` should stop filling the default database. It
should assume the input identifier is the final identifier like other catalogs,
and fail if identifier doesn't have database part. Then we don't need to do the
check here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381825279
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
I think `V2SessionCatalog` should stop filling the default database. It
should assume the input identifier is the final identifier and fail it doesn't
have database part. Then we don't need to do the check here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381825279
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
I think `V2SessionCatalog` should stop filling the default database. It
should assume the input identifier is the final identifier like other catalogs,
and fail it doesn't have database part. Then we don't need to do the check here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381824585
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
##
@@ -128,7 +128,7 @@ class PlanResolutionSuite extends AnalysisTest {
}
})
when(manager.currentCatalog).thenReturn(v2SessionCatalog)
-when(manager.currentNamespace).thenReturn(Array.empty[String])
+when(manager.currentNamespace).thenReturn(Array("default"))
Review comment:
why is this needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381823564
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
##
@@ -2121,23 +2120,40 @@ class DataSourceV2SQLSuite
withTable("t") {
sql("CREATE TABLE t USING json AS SELECT 1 AS i")
checkAnswer(sql("select * from t"), Row(1))
- checkAnswer(sql("select * from spark_catalog.t"), Row(1))
checkAnswer(sql("select * from spark_catalog.default.t"), Row(1))
}
}
+ test("SPARK-30885: v1 table name should be fully qualified") {
+def run(): Unit = {
Review comment:
nit: `assertWrongTableIdent`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381823606
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
The reason I put here is that `CatalogAndIdentifier` is also used here:
https://github.com/apache/spark/blob/7c4ad6316e3fe4a58629771a99faaa986c866c0b/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala#L53-L57,
which means I have to put the same check everywhere `CatalogAndIdentifier` is
used?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381822779
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
`CatalogAndIdentifier` should just focus on extracting catalog and
identifier.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381822407
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
or inside `ResolveSessionCatalog`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
cloud-fan commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381821285
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
shall we do this check inside `SessionCatalogAndIdentifier`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
AmplabJenkins removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588661698 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
AmplabJenkins removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588661667 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
AmplabJenkins commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588661698 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
AmplabJenkins commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588661667 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
SparkQA removed a comment on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588585126 **[Test build #118693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118693/testReport)** for PR 27632 at commit [`780fc35`](https://github.com/apache/spark/commit/780fc3531d39c9a0ef1143b8c49472fe22db9875). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes
SparkQA commented on issue #27632: [WIP][SPARK-30872][SQL] Constraints inferred from inferred attributes URL: https://github.com/apache/spark/pull/27632#issuecomment-588658774 **[Test build #118693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118693/testReport)** for PR 27632 at commit [`780fc35`](https://github.com/apache/spark/commit/780fc3531d39c9a0ef1143b8c49472fe22db9875). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
beliefer commented on a change in pull request #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#discussion_r381818519 ## File path: sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4 ## @@ -1797,11 +1797,11 @@ SIMPLE_COMMENT ; BRACKETED_EMPTY_COMMENT -: '/**/' -> channel(HIDDEN) +: '/*' BRACKETED_EMPTY_COMMENT* '*/' -> channel(HIDDEN) ; BRACKETED_COMMENT -: '/*' ~[+] .*? '*/' -> channel(HIDDEN) +: '/*' (BRACKETED_COMMENT .*? | ~[+] (BRACKETED_COMMENT|.)*?)*? '*/' -> channel(HIDDEN) Review comment: @gengliangwang The lexer rule executed before parser rule. So we must treat it in the lexer rule. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
imback82 commented on a change in pull request #27642: [SPARK-30885][SQL] V1
table name should be fully qualified if catalog name is provided
URL: https://github.com/apache/spark/pull/27642#discussion_r381817006
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/LookupCatalog.scala
##
@@ -106,24 +107,29 @@ private[sql] trait LookupCatalog extends Logging {
def unapply(nameParts: Seq[String]): Option[(CatalogPlugin, Identifier)] =
{
assert(nameParts.nonEmpty)
- if (nameParts.length == 1) {
-Some((currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head)))
+ val (catalog, ident) = if (nameParts.length == 1) {
+(currentCatalog, Identifier.of(catalogManager.currentNamespace,
nameParts.head))
} else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
// Conceptually global temp views are in a special reserved catalog.
However, the v2 catalog
// API does not support view yet, and we have to use v1 commands to
deal with global temp
// views. To simplify the implementation, we put global temp views in
a special namespace
// in the session catalog. The special namespace has higher priority
during name resolution.
// For example, if the name of a custom catalog is the same with
`GLOBAL_TEMP_DATABASE`,
// this custom catalog can't be accessed.
-Some((catalogManager.v2SessionCatalog, nameParts.asIdentifier))
+(catalogManager.v2SessionCatalog, nameParts.asIdentifier)
} else {
try {
- Some((catalogManager.catalog(nameParts.head),
nameParts.tail.asIdentifier))
+ (catalogManager.catalog(nameParts.head), nameParts.tail.asIdentifier)
} catch {
case _: CatalogNotFoundException =>
-Some((currentCatalog, nameParts.asIdentifier))
+(currentCatalog, nameParts.asIdentifier)
}
}
+ if (CatalogV2Util.isSessionCatalog(catalog) && ident.namespace.isEmpty) {
Review comment:
@cloud-fan For a session catalog, I could make this assumption that the
namespace is required, right? I looked at `CatalogManager` that uses
`v1SessionCatalog` for setting current namespace if the current catalog is a
session catalog; and `v1SessionCatalog` requires the namespace (database) to
already exist.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder
cloud-fan commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder URL: https://github.com/apache/spark/pull/27592#issuecomment-588652142 If we merge it to 3.0 but can't finish adding version info, then users would just see a lot of "unknown" since version which is harmful. BTW what's the drawback of merging it to master before RC1? AFAIK creating an RC just uses the target branch and it's nothing to do with the master branch. And one thing we missed when cutting the branch: we should bump up the version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
dongjoon-hyun commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588650942 Yep. +1 for reverting. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27615: [SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when it's disabled
HyukjinKwon commented on issue #27615: [SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when it's disabled URL: https://github.com/apache/spark/pull/27615#issuecomment-588650369 Thanks all! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
dongjoon-hyun commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588650135 Thank you for review and approval, @HyukjinKwon . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27615: [SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when it's disabled
dongjoon-hyun closed pull request #27615: [SPARK-29148][CORE][FOLLOW-UP] Don't dynamic allocation warning when it's disabled URL: https://github.com/apache/spark/pull/27615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
AmplabJenkins removed a comment on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642#issuecomment-588645554 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
AmplabJenkins removed a comment on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642#issuecomment-588645583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23452/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
AmplabJenkins commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642#issuecomment-588645583 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23452/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
AmplabJenkins commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642#issuecomment-588645554 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
SparkQA commented on issue #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642#issuecomment-588643848 **[Test build #118701 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118701/testReport)** for PR 27642 at commit [`018e6f6`](https://github.com/apache/spark/commit/018e6f6da7c698e52a219cd520a811e2a1e8d3c1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 opened a new pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided
imback82 opened a new pull request #27642: [SPARK-30885][SQL] V1 table name should be fully qualified if catalog name is provided URL: https://github.com/apache/spark/pull/27642 ### What changes were proposed in this pull request? For the following: ``` CREATE TABLE t USING json AS SELECT 1 AS i SELECT * FROM spark_catalog.t ``` `spark_catalog.t` is resolved to `spark_catalog.default.t` assuming the current namespace is `default`. However, this is not consistent with V2 behavior where the namespace must be specified if the catalog name is provided. This PR proposes to fix this inconsistency. ### Why are the changes needed? To be consistent with V2 table naming scheme in SQL commands. ### Does this PR introduce any user-facing change? Yes, now the user has to specify the namespace if the catalog name is provided. For example, ``` SELECT * FROM spark_catalog.t # Will throw AnalysisException with 'Session catalog cannot have an empty namespace: spark_catalog.t' SELECT * FROM spark_catalog.default.t # OK ``` ### How was this patch tested? Added new tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
dongjoon-hyun commented on issue #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector URL: https://github.com/apache/spark/pull/27637#issuecomment-588637593 Thank you for doing this, @gaborgsomogyi ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL]
Support Postgres Kerberos login in JDBC connector
URL: https://github.com/apache/spark/pull/27637#discussion_r381801540
##
File path: core/src/main/scala/org/apache/spark/util/SecurityUtils.scala
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import org.apache.spark.internal.Logging
+
+/**
+ * Various utility methods used by Spark Security.
+ */
+private[spark] object SecurityUtils extends Logging {
+ private val JAVA_VENDOR = "java.vendor"
+ private val IBM_KRB_DEBUG_CONFIG = "com.ibm.security.krb5.Krb5Debug"
+ private val SUN_KRB_DEBUG_CONFIG = "sun.security.krb5.debug"
+
+ def setGlobalKrbDebug(enabled: Boolean): Unit = {
+if (enabled) {
+ if (isIBMVendor()) {
+System.setProperty(IBM_KRB_DEBUG_CONFIG, "all")
+ } else {
+System.setProperty(SUN_KRB_DEBUG_CONFIG, "true")
+ }
+} else {
+ if (isIBMVendor()) {
+System.clearProperty(IBM_KRB_DEBUG_CONFIG)
+ } else {
+System.clearProperty(SUN_KRB_DEBUG_CONFIG)
+ }
+}
+ }
+
+ def isGlobalKrbDebugEnabled(): Boolean = {
+if (isIBMVendor()) {
+ val debug = System.getenv(IBM_KRB_DEBUG_CONFIG)
+ debug != null && debug.equalsIgnoreCase("all")
+} else {
+ val debug = System.getenv(SUN_KRB_DEBUG_CONFIG)
+ debug != null && debug.equalsIgnoreCase("true")
+}
+ }
+
+ /**
+ * Krb5LoginModule package vary in different JVMs.
Review comment:
`vary` -> `varies`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL]
Support Postgres Kerberos login in JDBC connector
URL: https://github.com/apache/spark/pull/27637#discussion_r381800481
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/ConnectionProvider.scala
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.jdbc.connection
+
+import java.sql.{Connection, Driver}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions
+
+/**
+ * Connection provider which opens connection toward various databases
(database specific instance
+ * needed). If authentication required then it's the provider's responsibility
to set all the
+ * parameters.
+ */
+private[jdbc] trait ConnectionProvider {
+ def getConnection(): Connection
+}
+
+private[jdbc] object ConnectionProvider extends Logging {
+ def create(driver: Driver, options: JDBCOptions): ConnectionProvider = {
+if (options.keytab == null || options.principal == null) {
+ logInfo("No authentication configuration found, using basic connection
provider")
+ new BasicConnectionProvider(driver, options)
+} else {
+ logInfo("Authentication configuration found, using database specific
connection provider")
+ options.driverClass match {
+case PostgresConnectionProvider.driverClass =>
+ logInfo("Postgres connection provider found")
Review comment:
Could you lower the log level for this line and line 40?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL]
Support Postgres Kerberos login in JDBC connector
URL: https://github.com/apache/spark/pull/27637#discussion_r381799498
##
File path:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala
##
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import com.spotify.docker.client.messages.{ContainerConfig, HostConfig}
+import java.io.{File, FileInputStream, FileOutputStream}
+import java.sql.Connection
+import java.util.Properties
+import javax.security.auth.login.Configuration
+import org.apache.hadoop.minikdc.MiniKdc
+import scala.io.Source
+
+import org.apache.spark.sql.Row
+import
org.apache.spark.sql.execution.datasources.jdbc.connection.PostgresConnectionProvider
+import org.apache.spark.sql.types.StringType
+import org.apache.spark.tags.DockerTest
+import org.apache.spark.util.{SecurityUtils, Utils}
+
+@DockerTest
+class PostgresKrbIntegrationSuite extends DockerJDBCIntegrationSuite {
Review comment:
It seems that we are going to add `OracleKrbIntegrationSuite` and
`MsSQLServerKrbIntegrationSuite`. Could you make a base class for them from the
beginning?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
AmplabJenkins removed a comment on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588631927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
AmplabJenkins removed a comment on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588631921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588631892 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588631887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588631887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
AmplabJenkins commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588631921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588631892 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
AmplabJenkins commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588631927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
SparkQA commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588631631 **[Test build #118699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118699/testReport)** for PR 27540 at commit [`7a0e2ef`](https://github.com/apache/spark/commit/7a0e2ef4d2116d5e02cd9ef280b965679f820369). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
SparkQA commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588631635 **[Test build #118700 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118700/testReport)** for PR 27495 at commit [`0a42a59`](https://github.com/apache/spark/commit/0a42a59df69ab262f78a9f6158eb8fad593b5579). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27640: [SPARK-30667][CORE] Add all gather method to BarrierTaskContext
mengxr commented on issue #27640: [SPARK-30667][CORE] Add all gather method to BarrierTaskContext URL: https://github.com/apache/spark/pull/27640#issuecomment-588631042 """ /home/runner/work/spark/spark/python/pyspark/taskcontext.py:docstring of pyspark.BarrierTaskContext.getTaskInfos:2:Explicit markup ends without a blank line; unexpected unindent. """ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
cloud-fan commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588630463 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
cloud-fan commented on issue #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540#issuecomment-588630427 according to the dev list discussion, we want to continue the revert. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL] Support Postgres Kerberos login in JDBC connector
dongjoon-hyun commented on a change in pull request #27637: [SPARK-30874][SQL]
Support Postgres Kerberos login in JDBC connector
URL: https://github.com/apache/spark/pull/27637#discussion_r381795251
##
File path:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
##
@@ -62,10 +62,17 @@ abstract class DatabaseOnDocker {
* Optional process to run when container starts
*/
def getStartupProcessName: Option[String]
+
+ /**
+ * Optional step before container starts
+ */
+ def beforeContainerStart(hostConfigBuilder: HostConfig.Builder,
+ containerConfigBuilder: ContainerConfig.Builder): Unit = {}
Review comment:
nit. Could you put the first argument on the next line?
```scala
def beforeContainerStart(
hostConfigBuilder: HostConfig.Builder,
containerConfigBuilder: ContainerConfig.Builder): Unit = {}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 opened a new pull request #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue"
Ngone51 opened a new pull request #27540: Revert "[SPARK-28093][SQL] Fix TRIM/LTRIM/RTRIM function parameter order issue" URL: https://github.com/apache/spark/pull/27540 ### What changes were proposed in this pull request? This reverts commit bef5d9d6c348e390f99b2cd781a2471d635e55f8. ### Why are the changes needed? Revert it according to https://github.com/apache/spark/pull/24902#issuecomment-584511167. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Gschiavon commented on issue #27578: [WIP][SPARK-30828][SQL] Improving insertInto behaviour
Gschiavon commented on issue #27578: [WIP][SPARK-30828][SQL] Improving insertInto behaviour URL: https://github.com/apache/spark/pull/27578#issuecomment-588630041 ping @cloud-fan @HyukjinKwon Let me know what do you think about this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
AmplabJenkins removed a comment on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588622012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
AmplabJenkins commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588622012 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
AmplabJenkins removed a comment on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588622007 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
AmplabJenkins commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588622007 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder
gatorsmile commented on issue #27592: [SPARK-30840][CORE][SQL] Add version property for ConfigEntry and ConfigBuilder URL: https://github.com/apache/spark/pull/27592#issuecomment-588621574 Thanks for doing this! Just my idea. First merge it to master. If we can finish the version info before the release, we can then cherry-pick this PR with the other follow-up PRs to 3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
SparkQA commented on issue #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641#issuecomment-588621590 **[Test build #118698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118698/testReport)** for PR 27641 at commit [`43da2e2`](https://github.com/apache/spark/commit/43da2e265754efd524a0cbfe107ac0daf7ec393a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9
dongjoon-hyun opened a new pull request #27641: [SPARK-30884][PYSPARK] Upgrade to Py4J 0.10.9 URL: https://github.com/apache/spark/pull/27641 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce any user-facing change? ### How was this patch tested? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
cloud-fan commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588612231 thanks for review, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
cloud-fan closed pull request #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yma11 edited a comment on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
yma11 edited a comment on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-588608946 Hi @srowen and @mengxr, I tested vector size 256 and nativeBLAS shows obvious perf gain compared with f2jBLAS in methods axpy(~1.7X), dot(~2.8X) and scal(double, dense)(>1.5X). For MKL, I can confirm it uses AVX in the methods from output but for OpenBLAS, seems it haven't used AVX in level-1 routines as based on the info from [https://github.com/xianyi/OpenBLAS/blob/develop/README.md](url). As to the MKL_NUM_THREADS and OPENBLAS_NUM_THREADS, limiting the threads to 1 doesn't always mean the best end-2-end performance. We used to test Kmeans using HiBench in a 1+4 cluster with MKL configured. Set the threads number to 1 or use default setting have no obvious performance change. By the way, using default MKL threads setting, MKL will bring 1.09X perf gain than java implementation in end-2-end. I also have updated this PR to revert back to use java implementation for scal(sparse) and dspmv(). Please take a further review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
AmplabJenkins commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#issuecomment-588609512 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23448/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
AmplabJenkins removed a comment on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#issuecomment-588609509 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
AmplabJenkins commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#issuecomment-588609509 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
AmplabJenkins removed a comment on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#issuecomment-588609512 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23448/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
SparkQA commented on issue #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#issuecomment-588609201 **[Test build #118697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118697/testReport)** for PR 27621 at commit [`ec9f494`](https://github.com/apache/spark/commit/ec9f4942f57ac03057e78ac5ce87758a4de45098). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yma11 commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines
yma11 commented on issue #27546: [SPARK-30773][ML]Support NativeBlas for level-1 routines URL: https://github.com/apache/spark/pull/27546#issuecomment-588608946 Hi @srowen and @mengxr, I tested vector size 256 and nativeBLAS shows obvious perf gain compared with f2jBLAS in methods axpy(~1.7X), dot(~2.8X) and scal(double, dense)(>1.5X). For MKL, I can confirm it uses AVX in the methods from output but for OpenBLAS, seems it haven't used AVX in level-1 routines as based on the info from [https://github.com/xianyi/OpenBLAS/blob/develop/README.md](url). As to the MKL_NUM_THREADS and OPENBLAS_NUM_THREADS, limiting the threads to 1 doesn't always mean the best end-to-end performance. We used to test Kmeans using HiBench in a 1+4 cluster with MKL configured. Set the threads number to 1 or use default setting have no obvious performance change. By the way, using default MKL threads setting, MKL will bring 1.09X perf gain than java implementation in end-to-end. I also have updated this PR to revert back to use java implementation for scal(sparse) and dspmv(). Please take a further review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
AmplabJenkins removed a comment on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588607748 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23447/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
AmplabJenkins removed a comment on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588607747 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
AmplabJenkins commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588607747 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
AmplabJenkins commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588607748 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23447/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
SparkQA commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588607440 **[Test build #118696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118696/testReport)** for PR 27639 at commit [`4adae15`](https://github.com/apache/spark/commit/4adae1553282ff3037e521539cfc16ffc55328d7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588606486 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588606481 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588606486 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
SparkQA removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604150 **[Test build #118695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118695/testReport)** for PR 27638 at commit [`b340a54`](https://github.com/apache/spark/commit/b340a5480f38d3823e045eedc92f597be1cdd353). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
SparkQA commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588606405 **[Test build #118695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118695/testReport)** for PR 27638 at commit [`b340a54`](https://github.com/apache/spark/commit/b340a5480f38d3823e045eedc92f597be1cdd353). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588606481 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold
gengliangwang commented on issue #27639: [SPARK-30881][SQL][DOCS]Revise the doc of spark.sql.sources.parallelPartitionDiscovery.threshold URL: https://github.com/apache/spark/pull/27639#issuecomment-588605957 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604454 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins removed a comment on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23446/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27619: revert SPARK-29663 and SPARK-29688
cloud-fan commented on a change in pull request #27619: revert SPARK-29663 and
SPARK-29688
URL: https://github.com/apache/spark/pull/27619#discussion_r381743821
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala
##
@@ -957,17 +957,4 @@ class DataFrameAggregateSuite extends QueryTest
assert(error.message.contains("function count_if requires boolean type"))
}
}
-
- test("calendar interval agg support hash aggregate") {
Review comment:
Added to the PR description.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604458 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23446/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
AmplabJenkins commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604454 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document
SparkQA commented on issue #27638: [SPARK-30878][SQL][DOC] Improve the CREATE TABLE document URL: https://github.com/apache/spark/pull/27638#issuecomment-588604150 **[Test build #118695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118695/testReport)** for PR 27638 at commit [`b340a54`](https://github.com/apache/spark/commit/b340a5480f38d3823e045eedc92f597be1cdd353). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key
gatorsmile commented on a change in pull request #27621: [SPARK-26071][followup] Improve migration guide of disallowing map type map key URL: https://github.com/apache/spark/pull/27621#discussion_r381740913 ## File path: docs/sql-migration-guide.md ## @@ -43,7 +43,7 @@ license: | - The `ADD JAR` command previously returned a result set with the single value 0. It now returns an empty result set. - - In Spark version 2.4 and earlier, users can create map values with map type key via built-in function like `CreateMap`, `MapFromArrays`, etc. Since Spark 3.0, it's not allowed to create map values with map type key with these built-in functions. Users can still read map values with map type key from data source or Java/Scala collections, though they are not very useful. + - In Spark version 2.4 and earlier, users can create map values with map type key via built-in function such as `CreateMap`, `MapFromArrays`, etc. Since Spark 3.0, it's not allowed to create map values with map type key with these built-in functions. Users can use array> as map keys instead as a workaround. In addition, users can still read map values with map type key from data source or Java/Scala collections, though it is discouraged. Review comment: Could we give a very simple example, how to use array> as map keys as a workaround? Also, do we have such a test case? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27638: [SPARK-30878][SQL] Improve the CREATE TABLE document
cloud-fan commented on a change in pull request #27638: [SPARK-30878][SQL]
Improve the CREATE TABLE document
URL: https://github.com/apache/spark/pull/27638#discussion_r381740742
##
File path: docs/sql-ref-syntax-ddl-create-table-hiveformat.md
##
@@ -96,21 +97,37 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ]
table_identifier
### Examples
{% highlight sql %}
---Using Comment and loading data from another table into the created table
-CREATE TABLE StudentInfo
Review comment:
yes we have. Search `CREATE TABLE` in
https://github.com/apache/spark/blob/master/docs/sql-migration-guide.md
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs
xuanyuanking commented on issue #27560: [SPARK-30809][SQL] Review and fix issues in SQL API docs URL: https://github.com/apache/spark/pull/27560#issuecomment-588603571 Yes, will address comments today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #27638: [SPARK-30878][SQL] Improve the CREATE TABLE document
viirya commented on a change in pull request #27638: [SPARK-30878][SQL] Improve
the CREATE TABLE document
URL: https://github.com/apache/spark/pull/27638#discussion_r381737898
##
File path: docs/sql-ref-syntax-ddl-create-table-hiveformat.md
##
@@ -96,21 +97,37 @@ CREATE [ EXTERNAL ] TABLE [ IF NOT EXISTS ]
table_identifier
### Examples
{% highlight sql %}
---Using Comment and loading data from another table into the created table
-CREATE TABLE StudentInfo
Review comment:
I remember we already add a note for this in migration guide.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588601284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins removed a comment on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588601296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23445/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments
AmplabJenkins commented on issue #27495: [SPARK-28880][SQL] Support ANSI nested bracketed comments URL: https://github.com/apache/spark/pull/27495#issuecomment-588601284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
