[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21024 **[Test build #89145 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89145/testReport)** for PR 21024 at commit [`e082f00`](https://github.com/apache/spark/commit/e082f0017dc670441e96a9b7d2ffa527302db2e3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89134/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21026 **[Test build #89134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89134/testReport)** for PR 21026 at commit [`45d3ed8`](https://github.com/apache/spark/commit/45d3ed8cefa17858def1df94bf7ccdbdfe642c96). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21024 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21021 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21021 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2170/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21021 **[Test build #89144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89144/testReport)** for PR 21021 at commit [`0203920`](https://github.com/apache/spark/commit/020392024f19cbc1ce172051643e15dade7e7b19). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20981: [SPARK-23873][SQL] Use accessors in interpreted LambdaVa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20981 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2169/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21026 **[Test build #89143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89143/testReport)** for PR 21026 at commit [`b2c1601`](https://github.com/apache/spark/commit/b2c16012d253b3c5cbd1145b21dfb10017926026). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20981: [SPARK-23873][SQL] Use accessors in interpreted LambdaVa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20981 **[Test build #89123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89123/testReport)** for PR 20981 at commit [`54dd939`](https://github.com/apache/spark/commit/54dd939e4771ca1678a3c9e5ffb9fc56ee119c32). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21026 **[Test build #89129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89129/testReport)** for PR 21026 at commit [`0b194ca`](https://github.com/apache/spark/commit/0b194ca4c3ef6b2b6411e123c1153da63a111374). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21014: [SPARK-23941][Mesos] Mesos task failed on specific spark...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21014 > mainClass doesn't need the same because space is not allowed It may have dollar signs and other things that the shell might want to interpret. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20795: [SPARK-23486]cache the function name from the catalog fo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20795 **[Test build #89142 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89142/testReport)** for PR 20795 at commit [`d1ee9cb`](https://github.com/apache/spark/commit/d1ee9cb89b9a718fcedea22f19bd7661709fb5fd). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21031 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21031 **[Test build #89141 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89141/testReport)** for PR 21031 at commit [`8945854`](https://github.com/apache/spark/commit/89458544b564097f2fa6d34ae654c86f155410df). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Cardinality(child: Expression) extends UnaryExpression with ExpectsInputTypes ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20981: [SPARK-23873][SQL] Use accessors in interpreted LambdaVa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20981 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89123/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21031 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89129/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21031 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2168/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21031 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89141/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16578: [SPARK-4502][SQL] Parquet nested column pruning
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16578#discussion_r180495402
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -151,6 +151,9 @@ abstract class Optimizer(sessionCatalog: SessionCatalog)
// The following batch should be executed after batch "Join Reorder"
and "LocalRelation".
Batch("Check Cartesian Products", Once,
CheckCartesianProducts) :+
+Batch("Field Extraction Pushdown", fixedPoint,
+ AggregateFieldExtractionPushdown,
+ JoinFieldExtractionPushdown) :+
--- End diff --
@mallman Could you split these new optimizer rules to two PRs first?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21031: [SPARK-23923][SQL] Add cardinality function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21031 **[Test build #89141 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89141/testReport)** for PR 21031 at commit [`8945854`](https://github.com/apache/spark/commit/89458544b564097f2fa6d34ae654c86f155410df). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21031: [SPARK-23923][SQL] Add cardinality function
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/21031 [SPARK-23923][SQL] Add cardinality function ## What changes were proposed in this pull request? The PR adds the SQL function `cardinality`. The behavior of the function is based on Presto's one. The function returns the length of the array or map stored in the column as `BigInt`. ## How was this patch tested? Added UTs You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-23923 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21031.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21031 commit 89458544b564097f2fa6d34ae654c86f155410df Author: Kazuaki Ishizaki Date: 2018-04-10T16:52:36Z initial commit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2167/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20938 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20938 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89119/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21025 **[Test build #89140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89140/testReport)** for PR 21025 at commit [`fbb9dc1`](https://github.com/apache/spark/commit/fbb9dc104a0bf78fc25d7c060f38b5485f279c1c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20938 **[Test build #89119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89119/testReport)** for PR 20938 at commit [`b9d99f7`](https://github.com/apache/spark/commit/b9d99f70cabadfaae72102e1d3ca80ccd2a616df). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `sealed trait Node extends Serializable ` * `sealed trait ClassificationNode extends Node ` * `sealed trait RegressionNode extends Node ` * `sealed trait LeafNode extends Node ` * `sealed trait InternalNode extends Node ` * `case class ExprCode(var code: String, var isNull: ExprValue, var value: ExprValue)` * `case class SubExprEliminationState(isNull: ExprValue, value: ExprValue)` * `abstract class ExprValue ` * `class LiteralValue(val value: String, val javaType: String) extends ExprValue ` * `case class VariableValue(` * `case class StatementValue(` * `case class GlobalValue(val value: String, val javaType: String) extends ExprValue ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20983: [SPARK-23747][Structured Streaming] Add EpochCoor...
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/20983#discussion_r180490617
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/continuous/EpochCoordinatorSuite.scala
---
@@ -0,0 +1,225 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.streaming.continuous
+
+import org.mockito.InOrder
+import org.mockito.Matchers.{any, eq => eqTo}
+import org.mockito.Mockito._
+import org.scalatest.BeforeAndAfterEach
+import org.scalatest.mockito.MockitoSugar
+
+import org.apache.spark._
+import org.apache.spark.rpc.RpcEndpointRef
+import org.apache.spark.sql.execution.streaming.continuous._
+import org.apache.spark.sql.sources.v2.reader.streaming.{ContinuousReader,
PartitionOffset}
+import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage
+import org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
+import org.apache.spark.sql.test.SharedSparkSession
+
+class EpochCoordinatorSuite
+ extends SparkFunSuite
+with SharedSparkSession
+with MockitoSugar
+with BeforeAndAfterEach {
+
+ private var epochCoordinator: RpcEndpointRef = _
+
+ private var writer: StreamWriter = _
+ private var query: ContinuousExecution = _
+ private var orderVerifier: InOrder = _
+
+ override def beforeEach(): Unit = {
+val reader = mock[ContinuousReader]
+writer = mock[StreamWriter]
+query = mock[ContinuousExecution]
+orderVerifier = inOrder(writer, query)
+
+epochCoordinator
+ = EpochCoordinatorRef.create(writer, reader, query, "test", 1,
spark, SparkEnv.get)
+ }
+
+ override def afterEach(): Unit = {
+SparkEnv.get.rpcEnv.stop(epochCoordinator)
+ }
+
+ test("single epoch") {
+setWriterPartitions(3)
+setReaderPartitions(2)
+
+commitPartitionEpoch(0, 1)
+commitPartitionEpoch(1, 1)
+commitPartitionEpoch(2, 1)
+reportPartitionOffset(0, 1)
+reportPartitionOffset(1, 1)
+
+// Here and in subsequent tests this is called to make a synchronous
call to EpochCoordinator
+// so that mocks would have been acted upon by the time verification
happens
+makeSynchronousCall()
+
+verifyCommit(1)
+ }
+
+ test("single epoch, all but one writer partition has committed") {
+setWriterPartitions(3)
+setReaderPartitions(2)
+
+commitPartitionEpoch(0, 1)
+commitPartitionEpoch(1, 1)
+reportPartitionOffset(0, 1)
+reportPartitionOffset(1, 1)
+
+makeSynchronousCall()
+
+verifyCommitHasntHappened(1)
+ }
+
+ test("single epoch, all but one reader partition has reported an
offset") {
+setWriterPartitions(3)
+setReaderPartitions(2)
+
+commitPartitionEpoch(0, 1)
+commitPartitionEpoch(1, 1)
+commitPartitionEpoch(2, 1)
+reportPartitionOffset(0, 1)
+
+makeSynchronousCall()
+
+verifyCommitHasntHappened(1)
--- End diff --
nit: maybe `verifyNoCommitFor`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/21025 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21029: [WIP][SPARK-23952] remove type parameter in DataR...
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21029#discussion_r180489842
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -143,7 +144,7 @@ case class MemoryStream[A : Encoder](id: Int,
sqlContext: SQLContext)
logDebug(generateDebugString(newBlocks.flatten, startOrdinal,
endOrdinal))
newBlocks.map { block =>
-new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory[UnsafeRow]]
+new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory]
--- End diff --
I'd like that, but I don't know if that would make things harder for data
source implementers working in Java.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21029: [WIP][SPARK-23952] remove type parameter in DataR...
Github user jose-torres commented on a diff in the pull request:
https://github.com/apache/spark/pull/21029#discussion_r180489396
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExec.scala
---
@@ -95,21 +77,29 @@ case class DataSourceV2ScanExec(
sparkContext.getLocalProperty(ContinuousExecution.EPOCH_COORDINATOR_ID_KEY),
sparkContext.env)
.askSync[Unit](SetReaderPartitions(readerFactories.size))
- new ContinuousDataSourceRDD(sparkContext, sqlContext,
readerFactories)
-.asInstanceOf[RDD[InternalRow]]
-
-case r: SupportsScanColumnarBatch if r.enableBatchRead() =>
- new DataSourceRDD(sparkContext,
batchReaderFactories).asInstanceOf[RDD[InternalRow]]
-
+ if (readerFactories.exists(_.dataFormat() ==
DataFormat.COLUMNAR_BATCH)) {
+throw new IllegalArgumentException(
+ "continuous stream reader does not support columnar read yet.")
--- End diff --
I don't, because I'm not really sure how it works in the batch case. How
does it work to do
new DataSourceRDD(sparkContext,
batchReaderFactories).asInstanceOf[RDD[InternalRow]]
when the type parameter of batchReaderFactories doesn't match InternalRow?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89125/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21025 **[Test build #89125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89125/testReport)** for PR 21025 at commit [`fbb9dc1`](https://github.com/apache/spark/commit/fbb9dc104a0bf78fc25d7c060f38b5485f279c1c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21029: [WIP][SPARK-23952] remove type parameter in DataReaderFa...
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/21029 +1 From the PR #20933, we can see that there is a lot of common code between `DataReaderFactory[ColumnarBatch]` and `DataReaderFactory[UnsafeRow]`, if we use the current method factory pattern. This change makes data source implementation easier, and we don't need to do runtime type cast. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21001: [SPARK-19724][SQL][FOLLOW-UP]Check location of ma...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/21001 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21001: [SPARK-19724][SQL][FOLLOW-UP]Check location of managed t...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21001 LGTM Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21001: [SPARK-19724][SQL][FOLLOW-UP]Check location of managed t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21001 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21001: [SPARK-19724][SQL][FOLLOW-UP]Check location of managed t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21001 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89115/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20984 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2166/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20984 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21001: [SPARK-19724][SQL][FOLLOW-UP]Check location of managed t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21001 **[Test build #89115 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89115/testReport)** for PR 21001 at commit [`ffd0c66`](https://github.com/apache/spark/commit/ffd0c66c621191945d70c21c05f6f49020bffdd4). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20984 **[Test build #89139 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89139/testReport)** for PR 20984 at commit [`a77128f`](https://github.com/apache/spark/commit/a77128f910eca1e0ced20257fa94ddaef513eae1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21009: [SPARK-23905][SQL] Add UDF weekday
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21009 **[Test build #89138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89138/testReport)** for PR 21009 at commit [`2b5db56`](https://github.com/apache/spark/commit/2b5db564e73e7919a0ac9783c44f3378291a72b8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20984 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20984 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20984 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89122/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21009: [SPARK-23905][SQL] Add UDF weekday
Github user yucai commented on the issue: https://github.com/apache/spark/pull/21009 Jenkins, retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20984: [SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20984 **[Test build #89122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89122/testReport)** for PR 20984 at commit [`a77128f`](https://github.com/apache/spark/commit/a77128f910eca1e0ced20257fa94ddaef513eae1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20981: [SPARK-23873][SQL] Use accessors in interpreted LambdaVa...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/20981 Looks good. Two more comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2165/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21026 **[Test build #89136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89136/testReport)** for PR 21026 at commit [`2488007`](https://github.com/apache/spark/commit/2488007d67c9e05d1b632efba2b56f3d87a63b06). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20981: [SPARK-23873][SQL] Use accessors in interpreted L...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/20981#discussion_r180475548
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala ---
@@ -119,4 +119,28 @@ object InternalRow {
case v: MapData => v.copy()
case _ => value
}
+
+ /**
+ * Returns an accessor for an `InternalRow` with given data type. The
returned accessor
+ * actually takes a `SpecializedGetters` input because it can be
generalized to other classes
+ * that implements `SpecializedGetters` (e.g., `ArrayData`) too.
+ */
+ def getAccessor(dataType: DataType): (SpecializedGetters, Int) => Any =
dataType match {
--- End diff --
Perhaps we should move this to the companion object of `SpecializedGetters`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21009: [SPARK-23905][SQL] Add UDF weekday
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21009 **[Test build #89137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89137/testReport)** for PR 21009 at commit [`2b5db56`](https://github.com/apache/spark/commit/2b5db564e73e7919a0ac9783c44f3378291a72b8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21019 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21019 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89117/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21019: [SPARK-23948] Trigger mapstage's job listener in submitM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21019 **[Test build #89117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89117/testReport)** for PR 21019 at commit [`685124a`](https://github.com/apache/spark/commit/685124a11b789af2a42b4978e25ed404b2a15176). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20535: [SPARK-23341][SQL] define some standard options for data...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20535 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89121/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20535: [SPARK-23341][SQL] define some standard options for data...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20535 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89116/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21025 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21025 **[Test build #89121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89121/testReport)** for PR 21025 at commit [`626f8cd`](https://github.com/apache/spark/commit/626f8cd49018ccb631e493f4cb3565bdb1415d75). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20535: [SPARK-23341][SQL] define some standard options for data...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20535 **[Test build #89116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89116/testReport)** for PR 20535 at commit [`c5e403c`](https://github.com/apache/spark/commit/c5e403c960cdfb68755df754abf7aa96ac6d40bc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21009: [SPARK-23905][SQL] Add UDF weekday
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/21009#discussion_r180472754
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -426,15 +426,7 @@ case class DayOfMonth(child: Expression) extends
UnaryExpression with ImplicitCa
""",
since = "2.3.0")
// scalastyle:on line.size.limit
-case class DayOfWeek(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
-
- override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
-
- override def dataType: DataType = IntegerType
-
- @transient private lazy val c = {
-Calendar.getInstance(DateTimeUtils.getTimeZone("UTC"))
- }
+case class DayOfWeek(child: Expression) extends DayWeek {
--- End diff --
They are different, see:
WeekDay: 0 = Monday, 1 = Tuesday, ⦠6 = Sunday
DayOfWeek: 1 = Sunday, 2 = Monday, ..., 7 = Saturday
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/16578 I will review this huge PR. : ) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21030: typo rawPredicition changed to rawPrediction
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21030 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19373: [SPARK-22150][CORE] PeriodicCheckpointer fails in case o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19373 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21030: typo rawPredicition changed to rawPrediction
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21030 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21029: [WIP][SPARK-23952] remove type parameter in DataReaderFa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21029 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89133/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21029: [WIP][SPARK-23952] remove type parameter in DataReaderFa...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21029 **[Test build #89133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89133/testReport)** for PR 21029 at commit [`d44105d`](https://github.com/apache/spark/commit/d44105d88e3e5ca1e01ce96efe7019a47960d5be). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21029: [WIP][SPARK-23952] remove type parameter in DataReaderFa...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21029 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20560: [SPARK-23375][SQL] Eliminate unneeded Sort in Optimizer
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20560 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2164/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20560: [SPARK-23375][SQL] Eliminate unneeded Sort in Optimizer
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20560 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21026: [SPARK-23951][SQL] Use actual java class instead ...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/21026#discussion_r180468933
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala
---
@@ -144,7 +144,7 @@ object GenerateUnsafeProjection extends
CodeGenerator[Seq[Expression], UnsafePro
case _ => s"$rowWriter.write($index, ${input.value});"
}
-if (input.isNull == "false") {
+if (input.isNull == FalseLiteral) {
--- End diff --
This fixes an existing bug :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21021: [SPARK-23921][SQL] Add array_sort function
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21021 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21030: typo rawPredicition changed to rawPrediction
GitHub user JBauerKogentix opened a pull request: https://github.com/apache/spark/pull/21030 typo rawPredicition changed to rawPrediction MultilayerPerceptronClassifier had 4 occurrences ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JBauerKogentix/spark patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21030.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21030 commit dd15d448798b6c6a905e23283d42962534caeea6 Author: JBauerKogentix <37910022+jbauerkogentix@...> Date: 2018-04-10T15:37:25Z typo rawPredicition changed to rawPrediction MultilayerPerceptronClassifier had 4 occurrences --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20858: [SPARK-23736][SQL] Extending the concat function to supp...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20858 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20560: [SPARK-23375][SQL] Eliminate unneeded Sort in Optimizer
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20560 **[Test build #89135 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89135/testReport)** for PR 20560 at commit [`e376c19`](https://github.com/apache/spark/commit/e376c193b44d5293cf9e7075b83149c93d1a9342). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20981: [SPARK-23873][SQL] Use accessors in interpreted L...
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/20981#discussion_r180468634
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala
---
@@ -65,11 +65,19 @@ trait ExpressionEvalHelper extends
GeneratorDrivenPropertyChecks {
checkEvaluationWithOptimization(expr, catalystValue, inputRow)
}
+
+ private def getActualDataType(dt: DataType): DataType = dt match {
--- End diff --
I added a similar method to the `UserDefinedType` companion. Which one
shall we add?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21029: [WIP][SPARK-23952] remove type parameter in DataR...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21029#discussion_r180468202
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -143,7 +144,7 @@ case class MemoryStream[A : Encoder](id: Int,
sqlContext: SQLContext)
logDebug(generateDebugString(newBlocks.flatten, startOrdinal,
endOrdinal))
newBlocks.map { block =>
-new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory[UnsafeRow]]
+new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory]
--- End diff --
cc @rdblue @jose-torres
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20986: [SPARK-23864][SQL] Add unsafe object writing to U...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20986 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2163/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21026: [SPARK-23951][SQL] Use actual java class instead of stri...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21026 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21029: [WIP][SPARK-23952] remove type parameter in DataR...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21029#discussion_r180468097
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala
---
@@ -143,7 +144,7 @@ case class MemoryStream[A : Encoder](id: Int,
sqlContext: SQLContext)
logDebug(generateDebugString(newBlocks.flatten, startOrdinal,
endOrdinal))
newBlocks.map { block =>
-new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory[UnsafeRow]]
+new
MemoryStreamDataReaderFactory(block).asInstanceOf[DataReaderFactory]
--- End diff --
I have seen this pattern many time, the java `List` is a little trouble
because it's invariance. Shall we change the interface to use array?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20938: [SPARK-23821][SQL] Collection function: flatten
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20938 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20986: [SPARK-23864][SQL] Add unsafe object writing to UnsafeWr...
Github user MarcSteven commented on the issue: https://github.com/apache/spark/pull/20986 cool --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21009: [SPARK-23905][SQL] Add UDF weekday
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21009#discussion_r180467960
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
---
@@ -426,15 +426,7 @@ case class DayOfMonth(child: Expression) extends
UnaryExpression with ImplicitCa
""",
since = "2.3.0")
// scalastyle:on line.size.limit
-case class DayOfWeek(child: Expression) extends UnaryExpression with
ImplicitCastInputTypes {
-
- override def inputTypes: Seq[AbstractDataType] = Seq(DateType)
-
- override def dataType: DataType = IntegerType
-
- @transient private lazy val c = {
-Calendar.getInstance(DateTimeUtils.getTimeZone("UTC"))
- }
+case class DayOfWeek(child: Expression) extends DayWeek {
--- End diff --
They are not duplicate.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21008: [SPARK-23902][SQL] Add roundOff flag to months_between
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21008 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21028: [SPARK-23922][SQL] Add arrays_overlap function
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21028 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21024: [SPARK-23917][SQL] Add array_max function
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21024 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21025: [SPARK-23918][SQL] Add array_min function
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21025 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21011: [SPARK-23916][SQL] Add array_join function
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21011 cc @ueshin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21029: [WIP][SPARK-23952] remove type parameter in DataR...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21029#discussion_r180467422
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExec.scala
---
@@ -95,21 +77,29 @@ case class DataSourceV2ScanExec(
sparkContext.getLocalProperty(ContinuousExecution.EPOCH_COORDINATOR_ID_KEY),
sparkContext.env)
.askSync[Unit](SetReaderPartitions(readerFactories.size))
- new ContinuousDataSourceRDD(sparkContext, sqlContext,
readerFactories)
-.asInstanceOf[RDD[InternalRow]]
-
-case r: SupportsScanColumnarBatch if r.enableBatchRead() =>
- new DataSourceRDD(sparkContext,
batchReaderFactories).asInstanceOf[RDD[InternalRow]]
-
+ if (readerFactories.exists(_.dataFormat() ==
DataFormat.COLUMNAR_BATCH)) {
+throw new IllegalArgumentException(
+ "continuous stream reader does not support columnar read yet.")
--- End diff --
cc @jose-torres do you know what's missing for this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
