[GitHub] [spark] sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502555484 my JIRA Id: m.sandeep , Can you please add me as a contributor to the JIRA issue ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24043: [SPARK-11412][SQL] Support merge schema for ORC
gengliangwang commented on a change in pull request #24043: [SPARK-11412][SQL]
Support merge schema for ORC
URL: https://github.com/apache/spark/pull/24043#discussion_r294153497
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaMergeUtils.scala
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileStatus, Path}
+
+import org.apache.spark.SparkException
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.util.SerializableConfiguration
+
+object SchemaMergeUtils extends Logging {
+ /**
+ * Figures out a merged Parquet/ORC schema with a distributed Spark job.
+ */
+ def mergeSchemasInParallel(
+ sparkSession: SparkSession,
+ files: Seq[FileStatus],
+ schemaReader: (Seq[FileStatus], Configuration, Boolean) =>
Seq[StructType])
+ : Option[StructType] = {
+val serializedConf = new
SerializableConfiguration(sparkSession.sessionState.newHadoopConf())
+
+// !! HACK ALERT !!
+//
+// Parquet/ORC requires `FileStatus`es to read footers.
Review comment:
Sorry, I check the code again and I think only Parquet requires
`FileStatus`, while Orc requires only `Path`. So the comments here are not
accurate.
We should explain that the hack is for Parquet, but it can be used by ORC as
well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jerryshao commented on a change in pull request #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
jerryshao commented on a change in pull request #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#discussion_r294148376 ## File path: core/src/main/scala/org/apache/spark/deploy/master/Master.scala ## @@ -561,7 +561,7 @@ private[deploy] class Master( apps.filter(_.state == ApplicationState.UNKNOWN).foreach(finishApplication) // Update the state of recovered apps to RUNNING -apps.filter(_.state == ApplicationState.WAITING).foreach(_.state = ApplicationState.RUNNING) +apps.filter(_.coresGranted > 0).foreach(_.state = ApplicationState.RUNNING) Review comment: Shall we also check the `state` to guarantee only the `WAITING` state and `coresGranted > 0` can be shifted to `RUNNING` ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502548931 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106569/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502548930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502548931 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106569/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502548930 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
SparkQA removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502520033 **[Test build #106569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106569/testReport)** for PR 24887 at commit [`b0ca665`](https://github.com/apache/spark/commit/b0ca66552b7ca2ec8610b9ec6a1e087dcea8948a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
SparkQA commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502548511 **[Test build #106569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106569/testReport)** for PR 24887 at commit [`b0ca665`](https://github.com/apache/spark/commit/b0ca66552b7ca2ec8610b9ec6a1e087dcea8948a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zuotingbing commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
zuotingbing commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-502547480 1. submit tow apps called A and B to the HA standalone cluster. APP A requests all resources, so app B is on waiting state. 2. stop the active master(make sure APP A is still running after recovery completed) 3. after recovery completed, app B is on running state but granted 0 cores This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #24859: [SPARK-28033][SQL] String concatenation should low priority than other operators
wangyum commented on issue #24859: [SPARK-28033][SQL] String concatenation should low priority than other operators URL: https://github.com/apache/spark/pull/24859#issuecomment-502547155 @mgaido91 @dongjoon-hyun Should we update the migration guide? I think this is a bug fix, not a behavior change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502546228 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106570/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502546222 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
SparkQA commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502546093 **[Test build #106570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106570/testReport)** for PR 24830 at commit [`784e63b`](https://github.com/apache/spark/commit/784e63b7a65a205b98661a1215a114bc40ac76b5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502546228 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106570/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
SparkQA removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525429 **[Test build #106570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106570/testReport)** for PR 24830 at commit [`784e63b`](https://github.com/apache/spark/commit/784e63b7a65a205b98661a1215a114bc40ac76b5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502546222 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` codegen to fix ICCE in JDK9+
dongjoon-hyun edited a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` codegen to fix ICCE in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502543663 cc @MaxGekk and @cloud-fan Also, cc @srowen and @kiszk . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
dongjoon-hyun commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502545450 @sandeepvja . What is your JIRA ID? You will be an Apache Spark contributor as the `Assignee` of https://issues.apache.org/jira/browse/SPARK-24898 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
dongjoon-hyun closed pull request #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
viirya commented on a change in pull request #24846: [SPARK-28012][SQL] Hive
UDF supports literal struct type
URL: https://github.com/apache/spark/pull/24846#discussion_r294144110
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
##
@@ -847,6 +847,8 @@ private[hive] trait HiveInspectors {
ObjectInspectorFactory.getStandardConstantMapObjectInspector(keyOI,
valueOI, jmap)
}
+case Literal(_, dt: StructType) =>
+ toInspector(dt)
Review comment:
The fix looks correct.
However, to use `toInspector` here is to invoke `toInspector(dataType:
DataType)` for nested fields. For the fields, not constant object inspector is
used anymore.
Since it is literal, can we follow array and map literals logics, to use
constant object inspector for nested fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
viirya commented on a change in pull request #24846: [SPARK-28012][SQL] Hive
UDF supports literal struct type
URL: https://github.com/apache/spark/pull/24846#discussion_r294144110
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
##
@@ -847,6 +847,8 @@ private[hive] trait HiveInspectors {
ObjectInspectorFactory.getStandardConstantMapObjectInspector(keyOI,
valueOI, jmap)
}
+case Literal(_, dt: StructType) =>
+ toInspector(dt)
Review comment:
The fix looks correct.
However, to use `toInspector` here is to invoke `toInspector(dataType:
DataType)` for nested fields. For the fields, not constant object inspector is
used anymore.
Can we follow array and map literals logics, to use constant object
inspector for nested fields?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG
SparkQA commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG URL: https://github.com/apache/spark/pull/24861#issuecomment-502544368 **[Test build #106574 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106574/testReport)** for PR 24861 at commit [`42d02aa`](https://github.com/apache/spark/commit/42d02aa56b155f0e0001e957e845a2e15c1c1cc2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG
AmplabJenkins removed a comment on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG URL: https://github.com/apache/spark/pull/24861#issuecomment-502543918 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG
AmplabJenkins commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG URL: https://github.com/apache/spark/pull/24861#issuecomment-502543921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11814/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG
AmplabJenkins removed a comment on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG URL: https://github.com/apache/spark/pull/24861#issuecomment-502543921 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11814/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG
AmplabJenkins commented on issue #24861: [SPARK-26985][CORE] Fix "access only some column of the all of columns " for big endian architecture BUG URL: https://github.com/apache/spark/pull/24861#issuecomment-502543918 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` codegen to fix ICCE in JDK9+
dongjoon-hyun commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` codegen to fix ICCE in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502543663 cc @MaxGekk and @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24843: [SPARK-28004][UI] Update jquery to 3.4.1
gengliangwang commented on issue #24843: [SPARK-28004][UI] Update jquery to 3.4.1 URL: https://github.com/apache/spark/pull/24843#issuecomment-502543856 A late LGTM. I also tried `grep -r "jquery-1.12.4.min.js" *` and there is no matched results. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jerryshao commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed.
jerryshao commented on issue #24848: [SPARK-28014][core] All waiting apps will be changed to the wrong state of Running after master changed. URL: https://github.com/apache/spark/pull/24848#issuecomment-502542514 How to reproduce this issue? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
dongjoon-hyun commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502541397 Thank you for updating, @srowen . The sentence looks correct since it describes the content of the return value. However, can we explicitly describe both `return this` and `the memory copy` cases separately? `@return this string` seems to give an impression to reuse the underlying memory space. But, we do copy for the 10% of the input strings. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
SparkQA removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502538729 **[Test build #106572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106572/testReport)** for PR 24883 at commit [`a130625`](https://github.com/apache/spark/commit/a130625dd96f972b21593291924599d399c59f99). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502540543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106572/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502540541 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502540543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106572/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502540541 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
SparkQA commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502540489 **[Test build #106572 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106572/testReport)** for PR 24883 at commit [`a130625`](https://github.com/apache/spark/commit/a130625dd96f972b21593291924599d399c59f99). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
AmplabJenkins removed a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502539770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
AmplabJenkins removed a comment on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502539774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11812/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
SparkQA commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502540102 **[Test build #106573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106573/testReport)** for PR 24889 at commit [`8c5e1cd`](https://github.com/apache/spark/commit/8c5e1cd818347313f609a3632d324c13949c70b3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539790 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
AmplabJenkins commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502539770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins removed a comment on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539797 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11813/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539790 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
AmplabJenkins commented on issue #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889#issuecomment-502539774 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11812/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
AmplabJenkins commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539797 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11813/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539646 > Thank you for your first contribution, @sandeepvja . > > * I left a minor comment. > * Please attach the generated screenshot to the PR description. @dongjoon-hyun Thanks for reviewing my first PR. Attached the screenshot to the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
sandeepvja commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502539270  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+
dongjoon-hyun opened a new pull request #24889: [SPARK-28072][SQL] Use `Iso8601TimestampFormatter` in `FromUnixTime` to fix `IncompatibleClassChangeError` in JDK9+ URL: https://github.com/apache/spark/pull/24889 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeepvja commented on a change in pull request #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
sandeepvja commented on a change in pull request #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#discussion_r294139294 ## File path: docs/configuration.md ## @@ -1107,6 +1107,14 @@ Apart from these, the following properties are also available, and may be useful Compression will use spark.io.compression.codec. + + spark.checkpoint.compress + false + +Whether to compress RDD checkpoints. Generally a good idea. +Compression will use spark.io.compression.codec Review comment: @dongjoon-hyun added a `.` at the end of the sentence This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs
SparkQA commented on issue #24883: [SPARK-24898][DOC] Adding spark.checkpoint.compress to the docs URL: https://github.com/apache/spark/pull/24883#issuecomment-502538729 **[Test build #106572 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106572/testReport)** for PR 24883 at commit [`a130625`](https://github.com/apache/spark/commit/a130625dd96f972b21593291924599d399c59f99). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
AmplabJenkins removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502531920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
AmplabJenkins removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502531923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106566/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
AmplabJenkins commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502531923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106566/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
AmplabJenkins commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502531920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
SparkQA removed a comment on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502512170 **[Test build #106566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106566/testReport)** for PR 24884 at commit [`81edf56`](https://github.com/apache/spark/commit/81edf561309cae081d96e0171226c776314db90b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace
SparkQA commented on issue #24884: [SPARK-28066][CORE] Optimize UTF8String.trim() for common case of no whitespace URL: https://github.com/apache/spark/pull/24884#issuecomment-502531644 **[Test build #106566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106566/testReport)** for PR 24884 at commit [`81edf56`](https://github.com/apache/spark/commit/81edf561309cae081d96e0171226c776314db90b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
AmplabJenkins removed a comment on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888#issuecomment-502528207 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888#issuecomment-502528497 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
AmplabJenkins removed a comment on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888#issuecomment-502528128 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888#issuecomment-502528207 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
AmplabJenkins commented on issue #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888#issuecomment-502528128 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on issue #24885: [SPARK-28040][R] Add serialization for glue type
MichaelChirico commented on issue #24885: [SPARK-28040][R] Add serialization for glue type URL: https://github.com/apache/spark/pull/24885#issuecomment-502528014 Have also filed https://github.com/apache/spark/pull/24888; leaving this PR open for now since the latter is a lot more involved / may take a tad longer to work out all the kinks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico opened a new pull request #24888: [SPARK-28040][SPARK-28070][R] Write type object s3
MichaelChirico opened a new pull request #24888: [SPARK-28040][SPARK-28070][R] Write type object s3 URL: https://github.com/apache/spark/pull/24888 ## What changes were proposed in this pull request? Follow-on to https://github.com/apache/spark/pull/24885 -- implement `writeType` and `writeObject` as S3 methods ## How was this patch tested? Untested; please advise Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
AmplabJenkins removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502526888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106567/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
SparkQA removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502512169 **[Test build #106567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106567/testReport)** for PR 24846 at commit [`7f8fbe8`](https://github.com/apache/spark/commit/7f8fbe8e51b31abb104210e1c1d15af3c8f9c5fc). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
AmplabJenkins removed a comment on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502526887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
AmplabJenkins commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502526887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
AmplabJenkins commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502526888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106567/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
SparkQA commented on issue #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type URL: https://github.com/apache/spark/pull/24846#issuecomment-502526696 **[Test build #106567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106567/testReport)** for PR 24846 at commit [`7f8fbe8`](https://github.com/apache/spark/commit/7f8fbe8e51b31abb104210e1c1d15af3c8f9c5fc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-502526238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11811/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins removed a comment on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-502526234 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-502526238 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11811/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
AmplabJenkins commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-502526234 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on a change in pull request #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
WeichenXu123 commented on a change in pull request #24830:
[SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
URL: https://github.com/apache/spark/pull/24830#discussion_r294129122
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala
##
@@ -96,30 +102,36 @@ abstract class PartitioningAwareFileIndex(
def allFiles(): Seq[FileStatus] = {
val files = if (partitionSpec().partitionColumns.isEmpty) {
- // For each of the root input paths, get the list of files inside them
- rootPaths.flatMap { path =>
-// Make the path qualified (consistent with listLeafFiles and
bulkListLeafFiles).
-val fs = path.getFileSystem(hadoopConf)
-val qualifiedPathPre = fs.makeQualified(path)
-val qualifiedPath: Path = if (qualifiedPathPre.isRoot &&
!qualifiedPathPre.isAbsolute) {
- // SPARK-17613: Always append `Path.SEPARATOR` to the end of parent
directories,
- // because the `leafFile.getParent` would have returned an absolute
path with the
- // separator at the end.
- new Path(qualifiedPathPre, Path.SEPARATOR)
-} else {
- qualifiedPathPre
-}
-
-// There are three cases possible with each path
-// 1. The path is a directory and has children files in it. Then it
must be present in
-//leafDirToChildrenFiles as those children files will have been
found as leaf files.
-//Find its children files from leafDirToChildrenFiles and include
them.
-// 2. The path is a file, then it will be present in leafFiles.
Include this path.
-// 3. The path is a directory, but has no children files. Do not
include this path.
+ if (recursive) {
+leafFiles.values.toSeq
Review comment:
I think "leafFiles" represent all nodes in a tree except "internal nodes".
("internal nodes" is directories in the tree structure)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
SparkQA commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525429 **[Test build #106570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106570/testReport)** for PR 24830 at commit [`784e63b`](https://github.com/apache/spark/commit/784e63b7a65a205b98661a1215a114bc40ac76b5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC
SparkQA commented on issue #24043: [SPARK-11412][SQL] Support merge schema for ORC URL: https://github.com/apache/spark/pull/24043#issuecomment-502525431 **[Test build #106571 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106571/testReport)** for PR 24043 at commit [`9353214`](https://github.com/apache/spark/commit/93532141438815bf1f3de0bde3a18ce8b931a97e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11810/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins removed a comment on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525120 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource
AmplabJenkins commented on issue #24830: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource URL: https://github.com/apache/spark/pull/24830#issuecomment-502525123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11810/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on a change in pull request #24885: [SPARK-28040][R] Add serialization for glue type
MichaelChirico commented on a change in pull request #24885: [SPARK-28040][R]
Add serialization for glue type
URL: https://github.com/apache/spark/pull/24885#discussion_r294128553
##
File path: R/pkg/R/serialize.R
##
@@ -71,18 +69,19 @@ writeObject <- function(con, object, writeType = TRUE) {
switch(serdeType,
NULL = writeVoid(con),
integer = writeInt(con, object),
+ glue =,
character = writeString(con, object),
logical = writeBoolean(con, object),
- double = writeDouble(con, object),
+ double =,
Review comment:
readability is in the eye of the beholder 🙃
anyway this `switch` will go away after 28070
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on a change in pull request #24885: [SPARK-28040][R] Add serialization for glue type
MichaelChirico commented on a change in pull request #24885: [SPARK-28040][R]
Add serialization for glue type
URL: https://github.com/apache/spark/pull/24885#discussion_r294128470
##
File path: R/pkg/R/serialize.R
##
@@ -36,15 +36,15 @@
# nolint end
getSerdeType <- function(object) {
- type <- class(object)[[1]]
- if (is.atomic(object) & !is.raw(object) & length(object) > 1) {
+ type <- class(object)[[1L]]
Review comment:
no, just a microoptimization while i'm in this source
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24885: [SPARK-28040][R] Add serialization for glue type
HyukjinKwon commented on a change in pull request #24885: [SPARK-28040][R] Add
serialization for glue type
URL: https://github.com/apache/spark/pull/24885#discussion_r294127967
##
File path: R/pkg/R/serialize.R
##
@@ -36,15 +36,15 @@
# nolint end
getSerdeType <- function(object) {
- type <- class(object)[[1]]
- if (is.atomic(object) & !is.raw(object) & length(object) > 1) {
+ type <- class(object)[[1L]]
Review comment:
btw why did we change 1 -> 1L? Is it related with the fix?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24885: [SPARK-28040][R] Add serialization for glue type
HyukjinKwon commented on a change in pull request #24885: [SPARK-28040][R] Add
serialization for glue type
URL: https://github.com/apache/spark/pull/24885#discussion_r294127821
##
File path: R/pkg/R/serialize.R
##
@@ -71,18 +69,19 @@ writeObject <- function(con, object, writeType = TRUE) {
switch(serdeType,
NULL = writeVoid(con),
integer = writeInt(con, object),
+ glue =,
character = writeString(con, object),
logical = writeBoolean(con, object),
- double = writeDouble(con, object),
+ double =,
Review comment:
I think it's better to keep it anyway. Looks previous one is better to read.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502519752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins removed a comment on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502519754 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11809/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
SparkQA commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502520033 **[Test build #106569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106569/testReport)** for PR 24887 at commit [`b0ca665`](https://github.com/apache/spark/commit/b0ca66552b7ca2ec8610b9ec6a1e087dcea8948a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502519752 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
SparkQA removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502509867 **[Test build #106565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106565/testReport)** for PR 24868 at commit [`344784d`](https://github.com/apache/spark/commit/344784d992e51727adb82015c5ccb27b371ae72b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql
AmplabJenkins commented on issue #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887#issuecomment-502519754 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11809/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
AmplabJenkins removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502519665 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106565/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
AmplabJenkins removed a comment on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502519664 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
AmplabJenkins commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502519665 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106565/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
HyukjinKwon commented on a change in pull request #24846: [SPARK-28012][SQL]
Hive UDF supports literal struct type
URL: https://github.com/apache/spark/pull/24846#discussion_r294124007
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala
##
@@ -847,6 +847,8 @@ private[hive] trait HiveInspectors {
ObjectInspectorFactory.getStandardConstantMapObjectInspector(keyOI,
valueOI, jmap)
}
+case Literal(_, dt: StructType) =>
+ toInspector(dt)
Review comment:
I think technically `named_struct` isn't a literal in this case. It's
foldable expression in the example above. Can you fix the PR title and
description accordingly to prevent confusion?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #24846: [SPARK-28012][SQL] Hive UDF supports literal struct type
HyukjinKwon commented on a change in pull request #24846: [SPARK-28012][SQL]
Hive UDF supports literal struct type
URL: https://github.com/apache/spark/pull/24846#discussion_r294121586
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala
##
@@ -652,6 +652,17 @@ class HiveUDFSuite extends QueryTest with
TestHiveSingleton with SQLTestUtils {
}
}
}
+ test("SPARK-28012 Hive UDF supports literal struct type") {
+withUserDefinedFunction("testLiteralStructType" -> false) {
+ // Simulate a hive udf that supports struct parameters
+ sql(s"CREATE FUNCTION testLiteralStructType AS '" +
Review comment:
leading `s` seems not needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
AmplabJenkins commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502519664 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics
SparkQA commented on issue #24868: [SPARK-28044][ML][PYTHON] MulticlassClassificationEvaluator support more metrics URL: https://github.com/apache/spark/pull/24868#issuecomment-502519527 **[Test build #106565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106565/testReport)** for PR 24868 at commit [`344784d`](https://github.com/apache/spark/commit/344784d992e51727adb82015c5ccb27b371ae72b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request #24887: [SPARK-28039][SQL][TEST] Port float4.sql
wangyum opened a new pull request #24887: [SPARK-28039][SQL][TEST] Port float4.sql URL: https://github.com/apache/spark/pull/24887 ## What changes were proposed in this pull request? This PR is to port float4.sql from PostgreSQL regression tests. https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/sql/float4.sql The expected results can be found in the link: https://github.com/postgres/postgres/blob/REL_12_BETA1/src/test/regress/expected/float4.out When porting the test cases, found three PostgreSQL specific features that do not exist in Spark SQL: [SPARK-28060](https://issues.apache.org/jira/browse/SPARK-28060): Float type can not accept some special inputs [SPARK-28027](https://issues.apache.org/jira/browse/SPARK-28027): Spark SQL does not support prefix operator `@` [SPARK-28061](https://issues.apache.org/jira/browse/SPARK-28061): Support for converting float to binary format Also, found a bug: [SPARK-28024](https://issues.apache.org/jira/browse/SPARK-28024): Incorrect value when out of range Also, found three inconsistent behavior: [SPARK-27923](https://issues.apache.org/jira/browse/SPARK-27923): Spark SQL insert there bad inputs to NULL [SPARK-28028](https://issues.apache.org/jira/browse/SPARK-28028): Cast numeric to integral type need round [SPARK-27923](https://issues.apache.org/jira/browse/SPARK-27923): Spark SQL returns NULL when dividing by zero ## How was this patch tested? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24882: [SPARK-28063][SQL] Replace deprecated `.newInstance()` in DSv2 `Catalogs`
dongjoon-hyun closed pull request #24882: [SPARK-28063][SQL] Replace deprecated `.newInstance()` in DSv2 `Catalogs` URL: https://github.com/apache/spark/pull/24882 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
