[GitHub] [spark] dongjoon-hyun commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496809787 I left a few comments. The others look good. Thank you for updating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288414881
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala
##
@@ -19,26 +19,35 @@ package org.apache.spark.sql.execution
import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{DataFrame, QueryTest}
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
/**
* Suite that tests the redaction of DataSourceScanExec
*/
-class DataSourceScanExecRedactionSuite extends QueryTest with SharedSQLContext
{
+abstract class DataSourceScanRedactionTest extends QueryTest with
SharedSQLContext {
override protected def sparkConf: SparkConf = super.sparkConf
-.set("spark.redaction.string.regex", "file:/[\\w_]+")
+.set("spark.redaction.string.regex", "file:/[\\w_/]+")
Review comment:
It looks correct fix, but do we need to change the pattern to pass the tests
by adding '/'?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288415208
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala
##
@@ -53,18 +62,21 @@ class DataSourceScanExecRedactionSuite extends QueryTest
with SharedSQLContext {
assert(df.queryExecution.simpleString.contains(replacement))
}
}
+}
- private def isIncluded(queryExecution: QueryExecution, msg: String): Boolean
= {
-queryExecution.toString.contains(msg) ||
-queryExecution.simpleString.contains(msg) ||
-queryExecution.stringWithStats.contains(msg)
- }
Review comment:
Could you add a document like
https://github.com/apache/spark/pull/24719/files#diff-0c515221ed6e6eadcec71b3b9ad3a3e1R116
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288414881
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala
##
@@ -19,26 +19,35 @@ package org.apache.spark.sql.execution
import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{DataFrame, QueryTest}
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
/**
* Suite that tests the redaction of DataSourceScanExec
*/
-class DataSourceScanExecRedactionSuite extends QueryTest with SharedSQLContext
{
+abstract class DataSourceScanRedactionTest extends QueryTest with
SharedSQLContext {
override protected def sparkConf: SparkConf = super.sparkConf
-.set("spark.redaction.string.regex", "file:/[\\w_]+")
+.set("spark.redaction.string.regex", "file:/[\\w_/]+")
Review comment:
Do we need to change the pattern by adding '/'?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288413825
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/DataSourceScanExecRedactionSuite.scala
##
@@ -19,26 +19,35 @@ package org.apache.spark.sql.execution
import org.apache.hadoop.fs.Path
import org.apache.spark.SparkConf
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{DataFrame, QueryTest}
+import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.datasources.v2.orc.OrcScan
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSQLContext
/**
* Suite that tests the redaction of DataSourceScanExec
Review comment:
Could you update the description, too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288413517
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
##
@@ -42,6 +44,21 @@ abstract class FileScan(
false
}
+ override def description(): String = {
+def seqToString(seq: Seq[Any]) = seq.mkString("[", ", ", "]")
+val locationDesc = fileIndex.getClass.getSimpleName +
seqToString(fileIndex.rootPaths)
+val metadata: Map[String, String] = Map(
+ "ReadSchema" -> readDataSchema.catalogString,
+ "Location" -> locationDesc)
+val metadataStr = metadata.toSeq.sorted.map {
+ case (key, value) =>
+val redactedValue =
+ Utils.redact(sparkSession.sessionState.conf.stringRedactionPattern,
value)
+key + ": " + StringUtils.abbreviate(redactedValue, 100)
+}.mkString(",")
Review comment:
`","` -> `", "`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do
not forcibly add cast when inserting table
URL: https://github.com/apache/spark/pull/24721#discussion_r288408930
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveOutputRelation.scala
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
NamedExpression, UpCast}
+import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan,
OverwriteByExpression, OverwritePartitionsDynamic, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DataType
+
+/**
+ * Resolves columns of an output table from the data in a logical plan. This
rule will:
+ *
+ * - Reorder columns when the write is by name
+ * - Insert safe casts when data types do not match
+ * - Insert aliases when column names do not match
+ * - Detect plans that are not compatible with the output table and throw
AnalysisException
+ */
+object ResolveOutputRelation extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperators {
+case append @ AppendData(table, query, isByName)
+ if table.resolved && query.resolved && !append.outputResolved =>
+ val projection = resolveOutputColumns(table.name, table.output, query,
isByName)
+
+ if (projection != query) {
+append.copy(query = projection)
+ } else {
+append
+ }
+
+case overwrite @ OverwriteByExpression(table, _, query, isByName)
+ if table.resolved && query.resolved && !overwrite.outputResolved =>
Review comment:
Nit: two extra spaces
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do
not forcibly add cast when inserting table
URL: https://github.com/apache/spark/pull/24721#discussion_r288412910
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1777,6 +1777,12 @@ object SQLConf {
.doc("When true, the upcast will be loose and allows string to atomic
types.")
.booleanConf
.createWithDefault(false)
+
+ val LEGACY_INSERT_TABLE_FORCIBLE_CAST =
buildConf("spark.sql.legacy.insertTable.forcibleCast")
+.doc("When inserting data to a table, Spark will cast the data type of
input query to " +
Review comment:
How about CTAS?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#discussion_r288411977 ## File path: sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala ## @@ -175,6 +175,7 @@ abstract class BaseSessionStateBuilder( override val postHocResolutionRules: Seq[Rule[LogicalPlan]] = PreprocessTableCreation(session) +: PreprocessTableInsertion(conf) +: +ResolveUpCast +: Review comment: Write a comment and explain this rule is to resolve the upcast added in PreprocessTableInsertion? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do
not forcibly add cast when inserting table
URL: https://github.com/apache/spark/pull/24721#discussion_r288408951
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveOutputRelation.scala
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
NamedExpression, UpCast}
+import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan,
OverwriteByExpression, OverwritePartitionsDynamic, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DataType
+
+/**
+ * Resolves columns of an output table from the data in a logical plan. This
rule will:
+ *
+ * - Reorder columns when the write is by name
+ * - Insert safe casts when data types do not match
+ * - Insert aliases when column names do not match
+ * - Detect plans that are not compatible with the output table and throw
AnalysisException
+ */
+object ResolveOutputRelation extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperators {
+case append @ AppendData(table, query, isByName)
+ if table.resolved && query.resolved && !append.outputResolved =>
+ val projection = resolveOutputColumns(table.name, table.output, query,
isByName)
+
+ if (projection != query) {
+append.copy(query = projection)
+ } else {
+append
+ }
+
+case overwrite @ OverwriteByExpression(table, _, query, isByName)
+ if table.resolved && query.resolved && !overwrite.outputResolved =>
+ val projection = resolveOutputColumns(table.name, table.output, query,
isByName)
+
+ if (projection != query) {
+overwrite.copy(query = projection)
+ } else {
+overwrite
+ }
+
+case overwrite @ OverwritePartitionsDynamic(table, query, isByName)
+ if table.resolved && query.resolved && !overwrite.outputResolved =>
Review comment:
Nit: two extra spaces
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do
not forcibly add cast when inserting table
URL: https://github.com/apache/spark/pull/24721#discussion_r288408857
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveOutputRelation.scala
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
NamedExpression, UpCast}
+import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan,
OverwriteByExpression, OverwritePartitionsDynamic, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DataType
+
+/**
+ * Resolves columns of an output table from the data in a logical plan. This
rule will:
+ *
+ * - Reorder columns when the write is by name
+ * - Insert safe casts when data types do not match
+ * - Insert aliases when column names do not match
+ * - Detect plans that are not compatible with the output table and throw
AnalysisException
+ */
+object ResolveOutputRelation extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperators {
+case append @ AppendData(table, query, isByName)
+ if table.resolved && query.resolved && !append.outputResolved =>
Review comment:
Nit: two extra spaces
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
gatorsmile commented on a change in pull request #24721: [SPARK-27856][SQL] do
not forcibly add cast when inserting table
URL: https://github.com/apache/spark/pull/24721#discussion_r288410459
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1777,6 +1777,12 @@ object SQLConf {
.doc("When true, the upcast will be loose and allows string to atomic
types.")
.booleanConf
.createWithDefault(false)
+
+ val LEGACY_INSERT_TABLE_FORCIBLE_CAST =
buildConf("spark.sql.legacy.insertTable.forcibleCast")
+.doc("When inserting data to a table, Spark will cast the data type of
input query to " +
+ "the data type of target table forcibly if this config is true.")
Review comment:
We need to add it to the migration guide.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288412832
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
##
@@ -42,6 +44,21 @@ abstract class FileScan(
false
}
+ override def description(): String = {
+def seqToString(seq: Seq[Any]) = seq.mkString("[", ", ", "]")
+val locationDesc = fileIndex.getClass.getSimpleName +
seqToString(fileIndex.rootPaths)
Review comment:
`seqToString(fileIndex.rootPaths)` -> `fileIndex.rootPaths.mkString("[", ",
", "]")`. And, this may need new line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288412667
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScan.scala
##
@@ -42,6 +44,21 @@ abstract class FileScan(
false
}
+ override def description(): String = {
+def seqToString(seq: Seq[Any]) = seq.mkString("[", ", ", "]")
Review comment:
It seems that we don't need to declare this one-time function. It's
different from `FileSourceScanExec` which reuses this function multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
HeartSaVioR commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-496804491 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR edited a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
HeartSaVioR edited a comment on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-496804372 I'm not sure I understand the mechanism of Amplab Jenkins. I guess either author or PR should have to be registered as OK to run the job. Normally committers and PMC members have privilege to trigger Jenkins and both of I and Gabor are not a committer. But let me just try out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
HeartSaVioR commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-496804372 I'm not sure I understand the mechanism of Amplab Jenkins. I guess either author or PR should have to be registered as OK to run the job. Normally committers and PMC members have privilege to trigger Jenkins (and both of I and Gabor are not.) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288411021
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala
##
@@ -49,6 +51,13 @@ trait FileDataSourceV2 extends TableProvider with
DataSourceRegister {
}
protected def getTableName(paths: Seq[String]): String = {
-shortName() + ":" + paths.mkString(";")
+val name = shortName().capitalize + " " +
paths.map(qualifiedPathName).mkString(",")
Review comment:
`shortName().capitalize` looks like beyond the initial goal(`redact`) of
this PR. Is this capitalization is required to pass the tests? Otherwise, shall
we keep the existing behavior?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on a change in pull request #24719: [SPARK-27849][SQL]
Redact treeString of FileTable and DataSourceV2ScanExecBase
URL: https://github.com/apache/spark/pull/24719#discussion_r288411021
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala
##
@@ -49,6 +51,13 @@ trait FileDataSourceV2 extends TableProvider with
DataSourceRegister {
}
protected def getTableName(paths: Seq[String]): String = {
-shortName() + ":" + paths.mkString(";")
+val name = shortName().capitalize + " " +
paths.map(qualifiedPathName).mkString(",")
Review comment:
`shortName().capitalize` looks like beyond the initial goal of this PR. Is
this capitalization is required to pass the tests? Otherwise, shall we keep the
existing behavior?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
dongjoon-hyun commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496803276 Thank you for redesigning, @gengliangwang . I'll take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly.
beliefer commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly. URL: https://github.com/apache/spark/pull/24372#issuecomment-496800369 > We are trying our best to reduce coupling with Hive. Having a native support of Default in schema specification in Spark is what we need. AFAIK, `CatalogTable` depend on `Hive` metastore. What you mean is to say support `DEFAULT` value not depend on `Hive` metastore? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
SparkQA commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-496799860 **[Test build #105893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105893/testReport)** for PR 24628 at commit [`cee5636`](https://github.com/apache/spark/commit/cee56365f1f2a589dde0eb0b602f1db1c912da9f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496799276 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105891/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496799273 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-496799412 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-496799412 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins removed a comment on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-496799416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11149/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver
AmplabJenkins commented on issue #24628: [SPARK-27749][SQL][test-hadoop3.2] hadoop-3.2 support hive-thriftserver URL: https://github.com/apache/spark/pull/24628#issuecomment-496799416 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11149/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496799143 **[Test build #105891 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105891/testReport)** for PR 24735 at commit [`55677c0`](https://github.com/apache/spark/commit/55677c04eb7e0782efc756073e5bf85b59b1aa1a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DummyExpressionHolder(exprs: Seq[Expression]) extends LeafNode ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496799273 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496799276 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105891/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496782273 **[Test build #105891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105891/testReport)** for PR 24735 at commit [`55677c0`](https://github.com/apache/spark/commit/55677c04eb7e0782efc756073e5bf85b59b1aa1a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
AmplabJenkins commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496798851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
AmplabJenkins removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496798859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105887/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
AmplabJenkins removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496798851 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
AmplabJenkins commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496798859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105887/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
SparkQA removed a comment on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496766399 **[Test build #105887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105887/testReport)** for PR 24734 at commit [`4d58419`](https://github.com/apache/spark/commit/4d58419df523ebc5427287a4c84bfb28968ad32c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline)
SparkQA commented on issue #24734: [SPARK-27870][SQL][PySpark] Flush each batch for pandas UDF (for improving pandas UDFs pipeline) URL: https://github.com/apache/spark/pull/24734#issuecomment-496798438 **[Test build #105887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105887/testReport)** for PR 24734 at commit [`4d58419`](https://github.com/apache/spark/commit/4d58419df523ebc5427287a4c84bfb28968ad32c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] eyalzit commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer
eyalzit commented on issue #24382: [SPARK-27330][SS] support task abort in foreach writer URL: https://github.com/apache/spark/pull/24382#issuecomment-496796715 @gaborgsomogyi @HeartSaVioR what else is required in order to trigger the Jenkins job? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5
AmplabJenkins removed a comment on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736#issuecomment-496794227 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5
AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736#issuecomment-496794643 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5
AmplabJenkins removed a comment on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736#issuecomment-496794099 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5
AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736#issuecomment-496794227 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5
AmplabJenkins commented on issue #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736#issuecomment-496794099 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] igreenfield commented on issue #24729: [SPARK-27862][Build] Move to json4s 3.6.5
igreenfield commented on issue #24729: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24729#issuecomment-496793245 Created also for master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] igreenfield opened a new pull request #24736: [SPARK-27862][Build] Move to json4s 3.6.5
igreenfield opened a new pull request #24736: [SPARK-27862][Build] Move to json4s 3.6.5 URL: https://github.com/apache/spark/pull/24736 Add scala-xml 1.2.0 What changes were proposed in this pull request? Move to json4s version 3.6.5 How was this patch tested? run: build/mvn clean package This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #24727: [SPARK-27867][ML] RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation
zhengruifeng commented on issue #24727: [SPARK-27867][ML] RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation URL: https://github.com/apache/spark/pull/24727#issuecomment-496792285 @srowen But current `Evaluator` do not expose method to obtain the metrics other that set by `metricName`. If we want two metrics, we have to compute twice. or we modify `Evaluator` to support `setMetricNames` and return an array? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496790279 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496790283 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105889/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496790279 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
AmplabJenkins commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496790283 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105889/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
SparkQA commented on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496789836 **[Test build #105889 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105889/testReport)** for PR 24643 at commit [`7cc4a92`](https://github.com/apache/spark/commit/7cc4a92b5d7ebbf421421f1c5d8da0bd0e671a49). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series
SparkQA removed a comment on issue #24643: [SPARK-26412][PySpark][SQL][WIP] Allow Pandas UDF to take an iterator of pd.Series or an iterator of tuple of pd.Series URL: https://github.com/apache/spark/pull/24643#issuecomment-496766421 **[Test build #105889 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105889/testReport)** for PR 24643 at commit [`7cc4a92`](https://github.com/apache/spark/commit/7cc4a92b5d7ebbf421421f1c5d8da0bd0e671a49). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496787849 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105890/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496787847 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496787849 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105890/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496787847 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
SparkQA commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496787757 **[Test build #105890 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105890/testReport)** for PR 24721 at commit [`4258665`](https://github.com/apache/spark/commit/425866578f8f18c861e64666e2b454376b6594fc). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
SparkQA removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773836 **[Test build #105890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105890/testReport)** for PR 24721 at commit [`4258665`](https://github.com/apache/spark/commit/425866578f8f18c861e64666e2b454376b6594fc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
cloud-fan edited a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781526 cc @ueshin @viirya @rednaxelafx @gatorsmile @maropu This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
SparkQA commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496783581 **[Test build #105892 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105892/testReport)** for PR 24719 at commit [`509761b`](https://github.com/apache/spark/commit/509761b4473611d393598e545a5fe14daaab5326). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
AmplabJenkins removed a comment on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496783301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11148/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
AmplabJenkins commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496783299 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
AmplabJenkins commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496783301 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11148/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
AmplabJenkins removed a comment on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496783299 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase
gengliangwang commented on issue #24719: [SPARK-27849][SQL] Redact treeString of FileTable and DataSourceV2ScanExecBase URL: https://github.com/apache/spark/pull/24719#issuecomment-496782905 @dongjoon-hyun I think the test suites are much better now. Please review it again, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
AmplabJenkins removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496782716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105886/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
AmplabJenkins removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496782715 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781889 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
SparkQA removed a comment on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496759896 **[Test build #105886 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105886/testReport)** for PR 24671 at commit [`f23c1b7`](https://github.com/apache/spark/commit/f23c1b70b9dcf8e4dce43e5dc217ea8822ddfae3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins removed a comment on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11147/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
AmplabJenkins commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496782715 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
AmplabJenkins commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496782716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105886/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead.
SparkQA commented on issue #24671: [SPARK-27811][Core][Docs]Improve docs about spark.driver.memoryOverhead and spark.executor.memoryOverhead. URL: https://github.com/apache/spark/pull/24671#issuecomment-496782380 **[Test build #105886 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105886/testReport)** for PR 24671 at commit [`f23c1b7`](https://github.com/apache/spark/commit/f23c1b70b9dcf8e4dce43e5dc217ea8822ddfae3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
SparkQA commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496782273 **[Test build #105891 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105891/testReport)** for PR 24735 at commit [`55677c0`](https://github.com/apache/spark/commit/55677c04eb7e0782efc756073e5bf85b59b1aa1a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly.
gatorsmile commented on issue #24372: [SPARK-27462][SQL] Enhance insert into hive table that could choose some columns in target table flexibly. URL: https://github.com/apache/spark/pull/24372#issuecomment-496782130 We are trying our best to reduce coupling with Hive. Having a native support of Default in schema specification in Spark is what we need. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11147/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
AmplabJenkins commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781889 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
cloud-fan commented on issue #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735#issuecomment-496781526 cc @ueshin @viirya @rednaxelafx @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan opened a new pull request #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs
cloud-fan opened a new pull request #24735: [SPARK-27871][SQL] LambdaVariable should use per-query unique IDs instead of globally unique IDs URL: https://github.com/apache/spark/pull/24735 ## What changes were proposed in this pull request? For simplicity, all `LambdaVariable`s are globally unique, to avoid any potential conflicts. However, this causes a perf problem: we can never hit codegen cache for encoder expressions that deal with collections (which means they contain `LambdaVariable`). To overcome this problem, `LambdaVariable` should have per-query unique IDs. This PR does 2 things: 1. refactor `LambdaVariable` to carry an ID, so that it's easier to change the ID. 2. add an optimizer rule to reassign `LambdaVariable` IDs, which are per-query unique. ## How was this patch tested? new tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #24730: [SPARK-27835][Core] Resource Scheduling: change driver config from addresses
felixcheung commented on a change in pull request #24730: [SPARK-27835][Core]
Resource Scheduling: change driver config from addresses
URL: https://github.com/apache/spark/pull/24730#discussion_r288389257
##
File path: core/src/main/scala/org/apache/spark/ResourceDiscoverer.scala
##
@@ -132,4 +132,20 @@ private[spark] object ResourceDiscoverer extends Logging {
}
}
}
+
+ def parseAllocatedFromJsonFile(resourcesFile: String): Map[String,
ResourceInformation] = {
+implicit val formats = DefaultFormats
+// case class to make json4s parsing easy
+case class JsonResourceInformation(val name: String, val addresses:
Array[String])
+val resourceInput = new BufferedInputStream(new
FileInputStream(resourcesFile))
+val resources = try {
+ parse(resourceInput).extract[Seq[JsonResourceInformation]]
+} catch {
+ case e@(_: MappingException | _: MismatchedInputException | _:
ClassCastException) =>
+throw new SparkException(s"Exception parsing the resources in
$resourcesFile", e)
+} finally {
+ resourceInput.close()
+}
+resources.map(r => (r.name, new ResourceInformation(r.name,
r.addresses))).toMap
Review comment:
could `Seq[JsonResourceInformation]` contain duplicated name? might be (very
marginally) better to do
`resource.toMap.map(...)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page
cloud-fan commented on issue #22571: [SPARK-25392][Spark Job History]Inconsistent behaviour for pool details in spark web UI and history server page URL: https://github.com/apache/spark/pull/22571#issuecomment-496779646 I don't know this part well, and I have no idea why this problem occurs. Would be great to see more details/analysis. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #24732: [SPARK-27868][core] Better default value and documentation for socket server backlog.
felixcheung commented on a change in pull request #24732: [SPARK-27868][core]
Better default value and documentation for socket server backlog.
URL: https://github.com/apache/spark/pull/24732#discussion_r288388516
##
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##
@@ -108,8 +108,8 @@ public int numConnectionsPerPeer() {
return conf.getInt(SPARK_NETWORK_IO_NUMCONNECTIONSPERPEER_KEY, 1);
}
- /** Requested maximum length of the queue of incoming connections. Default
-1 for no backlog. */
- public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, -1);
}
+ /** Requested maximum length of the queue of incoming connections. Default
is 64. */
+ public int backLog() { return conf.getInt(SPARK_NETWORK_IO_BACKLOG_KEY, 64);
}
Review comment:
what's the different between setting to -1 or to 64 as a default? does this
change any existing behavior?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on a change in pull request #24709: [SPARK-27841][SQL] Improve UTF8String to/fromString()/numBytesForFirstByte() performance
felixcheung commented on a change in pull request #24709: [SPARK-27841][SQL]
Improve UTF8String to/fromString()/numBytesForFirstByte() performance
URL: https://github.com/apache/spark/pull/24709#discussion_r288388247
##
File path:
common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java
##
@@ -1217,6 +1264,20 @@ public boolean toByte(IntWrapper intWrapper) {
@Override
public String toString() {
+byte[] bytes = getBytes();
+// Optimization for ASCII characters: use deprecated string API which
+// skips charset encoder and simply casts each byte into a char.
+if (isAscii(bytes)) {
+ return new String(bytes, 0);
Review comment:
it does have to check all bytes though? isn't this grow linearly to the
length of bytes?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
AmplabJenkins removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-49684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
SparkQA removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-496766406 **[Test build #105888 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105888/testReport)** for PR 24717 at commit [`7d83ac3`](https://github.com/apache/spark/commit/7d83ac39df2a45fca2990e855802055f60804cc6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
AmplabJenkins removed a comment on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-49686 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105888/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
AmplabJenkins commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-49684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
AmplabJenkins commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-49686 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105888/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics
SparkQA commented on issue #24717: [SPARK-27847][ML] One-Pass MultilabelMetrics & MulticlassMetrics URL: https://github.com/apache/spark/pull/24717#issuecomment-496777624 **[Test build #105888 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105888/testReport)** for PR 24717 at commit [`7d83ac3`](https://github.com/apache/spark/commit/7d83ac39df2a45fca2990e855802055f60804cc6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] felixcheung commented on issue #24677: [SPARK-27805][PYTHON] Propagate SparkExceptions during toPandas with arrow enabled
felixcheung commented on issue #24677: [SPARK-27805][PYTHON] Propagate SparkExceptions during toPandas with arrow enabled URL: https://github.com/apache/spark/pull/24677#issuecomment-496776866 > Yes, you are right, this is the same issue as `toLocalIterator` in #24070 and needs to be fixed. This is a real problem for branch-2.4 which, like you said, could cause `toPandas` to return a partial result without raising the error. @HyukjinKwon do you think would it make sense to patch branch-2.4 with a manual fix? this sounds important... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make separate PySpark/SparkR vectorization configurations
viirya commented on issue #24700: [SPARK-27834][SQL][R][PYTHON] Make separate PySpark/SparkR vectorization configurations URL: https://github.com/apache/spark/pull/24700#issuecomment-496775395 `spark.sql.execution.arrow.pyspark.enabled` looks slightly better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
SparkQA commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773836 **[Test build #105890 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105890/testReport)** for PR 24721 at commit [`4258665`](https://github.com/apache/spark/commit/425866578f8f18c861e64666e2b454376b6594fc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773587 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11146/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins removed a comment on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773581 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773581 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table
AmplabJenkins commented on issue #24721: [SPARK-27856][SQL] do not forcibly add cast when inserting table URL: https://github.com/apache/spark/pull/24721#issuecomment-496773587 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11146/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] lu-wang-dl closed pull request #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local
lu-wang-dl closed pull request #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local URL: https://github.com/apache/spark/pull/24705 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] lu-wang-dl commented on issue #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local
lu-wang-dl commented on issue #24705: [SPARK-22340][PYTHON] Save localProperties in thread.local URL: https://github.com/apache/spark/pull/24705#issuecomment-496772401 Close this PR now. We will design this more carefully. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24689: [SPARK-26946][SQL][FOLLOWUP] Require lookup function
AmplabJenkins removed a comment on issue #24689: [SPARK-26946][SQL][FOLLOWUP] Require lookup function URL: https://github.com/apache/spark/pull/24689#issuecomment-496767946 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
