[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551156 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551177 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551460 [Test build #40745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40745/consoleFull) for PR 8129 at commit [`9298fa0`](https://github.com/apache/spark/commit/9298fa0577dae7018c4d2aaa58301ce5e340251a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9877][Core] Fix StandaloneRestServer NP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8127#issuecomment-130551430 [Test build #40744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40744/consoleFull) for PR 8127 at commit [`fdb6158`](https://github.com/apache/spark/commit/fdb6158f6439dce80bdbd01ef2e483265ca5eb84). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9740] [SPARK-9592] [SPARK-9210] [SQL] C...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8113#issuecomment-130551263 [Test build #1542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1542/console) for PR 8113 at commit [`f828bdf`](https://github.com/apache/spark/commit/f828bdf1612a5fc9466b9a7e80700d0dd94faaf5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` * Set thresholds in multiclass (or binary) classification to adjust the probability of` * `case class First(child: Expression, ignoreNullsExpr: Expression) extends AlgebraicAggregate ` * `case class Last(child: Expression, ignoreNullsExpr: Expression) extends AlgebraicAggregate ` * `case class First(` * `case class FirstFunction(` * `case class Last(` * `case class LastFunction(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9885] [SQL] Also pass barrierPrefixes a...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8158#issuecomment-130560469 Verified under my local MySQL backed Hive 0.13.1 metastore and it works. Merging to master and branch-1.5. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130566232 @jkbradley I've addressed your comments. I'll have another pass at the generated docs, to see if there are other issues as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user MechCoder commented on a diff in the pull request:
https://github.com/apache/spark/pull/8126#discussion_r36947726
--- Diff:
mllib/src/test/java/org/apache/spark/mllib/stat/JavaStatisticsSuite.java ---
@@ -53,4 +54,12 @@ public void testCorr() {
// Check default method
assertEquals(corr1, corr2);
}
+
+ @Test
+ public void kolmogorovSmirnovTest() {
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130566054 [Test build #1539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1539/console) for PR 8111 at commit [`828144f`](https://github.com/apache/spark/commit/828144f96e4454824887c1d01fada20ce3510610). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8125] [SQL] Backports PR #7396 to branc...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/7664#issuecomment-130568125 @oliviertoupin Usually we only backports fixes of severe bugs to maintaining branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9757] [SQL] Fixes persistence of Parque...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8130#issuecomment-130570571 @yhuai @marmbrus Thanks for the review and helping fixing it! I'm merging this to master and branch-1.5. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36949497
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -270,6 +270,8 @@ private[sql] object PartitioningUtils {
private val upCastingOrder: Seq[DataType] =
Seq(NullType, IntegerType, LongType, FloatType, DoubleType, StringType)
+ val validPartitionColumnTypes: Set[DataType] = upCastingOrder.toSet
--- End diff --
BTW, I think you have to make `validPartitionColumnTypes` a method rather
than a `Set[DataType]` since `DecimalType` is not a singleton.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8161#issuecomment-130572392 [Test build #40748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40748/consoleFull) for PR 8161 at commit [`bdf1d5e`](https://github.com/apache/spark/commit/bdf1d5e357fe423d356405acd0844991c652f150). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user yjshen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36950167
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala
---
@@ -179,6 +179,13 @@ object ResolvedDataSource extends Logging {
val fs =
path.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
path.makeQualified(fs.getUri, fs.getWorkingDirectory)
}
+
+partitionColumnsSchema(data.schema, partitionColumns).foreach {
field =
+ if
(!PartitioningUtils.validPartitionColumnTypes.contains(field.dataType)) {
+throw new AnalysisException(sCannot use ${field.dataType} for
partition column)
+ }
+}
--- End diff --
ok, I'll make this a function in PartitioningUtils to throw an
analysisException.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8161#issuecomment-130575195 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8161#issuecomment-130575160 [Test build #40748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40748/console) for PR 8161 at commit [`bdf1d5e`](https://github.com/apache/spark/commit/bdf1d5e357fe423d356405acd0844991c652f150). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8101#issuecomment-130586425 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user yjshen commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130586521 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8101#issuecomment-130586441 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8813][SQL] Combine files when there're ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8125#discussion_r36946906
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/CombineSmallFile.scala ---
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.sources
+
+import org.apache.hadoop.fs.{FileStatus, FileSystem, Path}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.SQLContext
+
+object CombineSmallFile {
+ def combineWithFiles[T](rdd: RDD[T], sqlContext: SQLContext, inputFiles:
Array[FileStatus])
+ : RDD[T] = {
+if (sqlContext.conf.combineSmallFile) {
+ val totalLen = inputFiles.map { file =
+if (file.isDir) 0L else file.getLen
+ }.sum
+ val numPartitions = (totalLen / sqlContext.conf.splitSize + 1).toInt
+ rdd.coalesce(numPartitions)
--- End diff --
What if Hadoop block size is configured larger (as many users do)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130566469 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130566485 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130567185 [Test build #1543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1543/console) for PR 8111 at commit [`c4d44c9`](https://github.com/apache/spark/commit/c4d44c9139eff45048d849311c539dadda54004c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130567175 [Test build #40737 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40737/console) for PR 8111 at commit [`828144f`](https://github.com/apache/spark/commit/828144f96e4454824887c1d01fada20ce3510610). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130567224 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user MechCoder commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130570175 OK. I gave the diff another pass. That seems to be it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36949417
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
---
@@ -554,6 +556,21 @@ abstract class HadoopFsRelationTest extends QueryTest
with SQLTestUtils {
clonedConf.foreach(entry = configuration.set(entry.getKey,
entry.getValue))
}
}
+
+ test(SPARK-8887: Explicitly define which data types can be used as
dynamic partition columns) {
+val df = Seq(
+ (1, v1, Date.valueOf(2015-08-10)),
+ (2, v2, Date.valueOf(2015-08-11)),
+ (3, v3, Date.valueOf(2015-08-12))).toDF(a, b, c)
+withTempDir { file =
+ intercept[AnalysisException] {
+
df.write.format(dataSourceName).partitionBy(c).save(file.getCanonicalPath)
+ }
+}
+intercept[AnalysisException] {
+ df.write.format(dataSourceName).partitionBy(c).saveAsTable(t)
+}
--- End diff --
Please wrap this block with `withTable(t) { ... }` so that `t` gets
dropped.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8162#issuecomment-130574356 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8162#issuecomment-130574344 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130586461 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8531] [ML] Update ML user guide for Min...
Github user hhbyyh commented on the pull request: https://github.com/apache/spark/pull/7211#issuecomment-130551690 @jkbradley Thanks for the review. I'm not sure if the latex part looks good. And for python document, since python interface for MinMaxScaler is still under review, so I didn't add the example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551698 [Test build #40745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40745/console) for PR 8129 at commit [`9298fa0`](https://github.com/apache/spark/commit/9298fa0577dae7018c4d2aaa58301ce5e340251a). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ApplicationSubmission(val appInfo: ApplicationInfo, val submittedTime: Date)` * ` class Pool(val poolName: String, val priority: Int, val cores: Int) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130551699 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL]Filters are not passed throug...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/8049#discussion_r36947567
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
---
@@ -343,31 +343,57 @@ private[sql] object DataSourceStrategy extends
Strategy with Logging {
* and convert them.
*/
protected[sql] def selectFilters(filters: Seq[Expression]) = {
+import CatalystTypeConverters._
+
def translate(predicate: Expression): Option[Filter] = predicate match
{
case expressions.EqualTo(a: Attribute, Literal(v, _)) =
Some(sources.EqualTo(a.name, v))
case expressions.EqualTo(Literal(v, _), a: Attribute) =
Some(sources.EqualTo(a.name, v))
+ case expressions.EqualTo(Cast(a: Attribute, _), l: Literal) =
+Some(sources.EqualTo(a.name, convertToScala(Cast(l,
a.dataType).eval(), a.dataType)))
+ case expressions.EqualTo(l: Literal, Cast(a: Attribute, _)) =
+Some(sources.EqualTo(a.name, convertToScala(Cast(l,
a.dataType).eval(), a.dataType)))
--- End diff --
No, given the possibly trickiness here I think we should bump the fix to
1.6.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8813][SQL] Combine files when there're ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8125#issuecomment-130566869 @watermen The use case you mentioned totally makes sense. However, I think usually people choose to compact fine grained files into much larger and fewer files as time goes by. A more reasonable solution might be: 1. Saving the most recent hot data (say 1 hr) every 5 min in simple file formats like CSV or JSON. These files tend to be pretty small, and I'd assume that using complex columnar formats like ORC and Parquet generally don't give you much performance benefits on the read path, but you still suffer from their costs like larger memory footprints and lower speed on the write path (it's more related to the width of the table rather than the number of rows.) 2. Compacting outdated data periodically (say every a few hours) into much larger and fewer chunks of data files in analytics friendly formats like ORC and Parquet In this way you avoid reading a large number of small files and enjoy the performance benefits brought by columnar formats. 3. Exposing the whole dataset by making two (or more) DataFrames out of these two parts of data and union them Of course, the above comment is more like a design issue of the upper application. For this PR, the biggest problem I see is that, it makes a not recommended special use case as default case and introduces performance regression for other (more commonly seen) use cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9868] [SQL] [WIP] reproduce failure
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8160#issuecomment-130566969 [Test build #1548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1548/console) for PR 8160 at commit [`044d0c1`](https://github.com/apache/spark/commit/044d0c1ac723c77d84d349a57991ac86fb87de59). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36948745
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
---
@@ -270,6 +270,8 @@ private[sql] object PartitioningUtils {
private val upCastingOrder: Seq[DataType] =
Seq(NullType, IntegerType, LongType, FloatType, DoubleType, StringType)
+ val validPartitionColumnTypes: Set[DataType] = upCastingOrder.toSet
--- End diff --
I think all data types inherit from `AtomicType` should be valid here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36949251
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala
---
@@ -179,6 +179,13 @@ object ResolvedDataSource extends Logging {
val fs =
path.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
path.makeQualified(fs.getUri, fs.getWorkingDirectory)
}
+
+partitionColumnsSchema(data.schema, partitionColumns).foreach {
field =
+ if
(!PartitioningUtils.validPartitionColumnTypes.contains(field.dataType)) {
+throw new AnalysisException(sCannot use ${field.dataType} for
partition column)
+ }
+}
--- End diff --
Actually twice, the 3rd place is still quite similar though :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36949177
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ResolvedDataSource.scala
---
@@ -179,6 +179,13 @@ object ResolvedDataSource extends Logging {
val fs =
path.getFileSystem(sqlContext.sparkContext.hadoopConfiguration)
path.makeQualified(fs.getUri, fs.getWorkingDirectory)
}
+
+partitionColumnsSchema(data.schema, partitionColumns).foreach {
field =
+ if
(!PartitioningUtils.validPartitionColumnTypes.contains(field.dataType)) {
+throw new AnalysisException(sCannot use ${field.dataType} for
partition column)
+ }
+}
--- End diff --
Can we make this snippet a separate method? It's duplicated 3 times in this
PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user yjshen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36950083
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
---
@@ -554,6 +556,21 @@ abstract class HadoopFsRelationTest extends QueryTest
with SQLTestUtils {
clonedConf.foreach(entry = configuration.set(entry.getKey,
entry.getValue))
}
}
+
+ test(SPARK-8887: Explicitly define which data types can be used as
dynamic partition columns) {
+val df = Seq(
+ (1, v1, Date.valueOf(2015-08-10)),
+ (2, v2, Date.valueOf(2015-08-11)),
+ (3, v3, Date.valueOf(2015-08-12))).toDF(a, b, c)
+withTempDir { file =
+ intercept[AnalysisException] {
+
df.write.format(dataSourceName).partitionBy(c).save(file.getCanonicalPath)
+ }
+}
+intercept[AnalysisException] {
+ df.write.format(dataSourceName).partitionBy(c).saveAsTable(t)
+}
+ }
--- End diff --
OK. I get this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8161#issuecomment-130571993 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user yjshen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36950026
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
---
@@ -554,6 +556,21 @@ abstract class HadoopFsRelationTest extends QueryTest
with SQLTestUtils {
clonedConf.foreach(entry = configuration.set(entry.getKey,
entry.getValue))
}
}
+
+ test(SPARK-8887: Explicitly define which data types can be used as
dynamic partition columns) {
+val df = Seq(
+ (1, v1, Date.valueOf(2015-08-10)),
+ (2, v2, Date.valueOf(2015-08-11)),
+ (3, v3, Date.valueOf(2015-08-12))).toDF(a, b, c)
+withTempDir { file =
+ intercept[AnalysisException] {
+
df.write.format(dataSourceName).partitionBy(c).save(file.getCanonicalPath)
+ }
+}
+intercept[AnalysisException] {
+ df.write.format(dataSourceName).partitionBy(c).saveAsTable(t)
+}
--- End diff --
I didn't wrap this into withTable(t) because the saveAsTable will fail
here, and if wrapped in withTable,
`org.apache.spark.sql.catalyst.analysis.NoSuchTableException was thrown.`
```
def getTable(dbName: String, tableName: String): HiveTable = {
getTableOption(dbName, tableName).getOrElse(throw new
NoSuchTableException)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8161#issuecomment-130571975 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9757] [SQL] Fixes persistence of Parque...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8130 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130573888 **[Test build #1541 timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1541/console)** for PR 8111 at commit [`828144f`](https://github.com/apache/spark/commit/828144f96e4454824887c1d01fada20ce3510610) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Testing Jenkins do not merge.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8139#issuecomment-130574634 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8162#issuecomment-130574769 [Test build #40749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40749/consoleFull) for PR 8162 at commit [`4fb536d`](https://github.com/apache/spark/commit/4fb536dcd38b21d5844611c03e436edff996006f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130585654 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8101#issuecomment-130586781 [Test build #40751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40751/consoleFull) for PR 8101 at commit [`aed4162`](https://github.com/apache/spark/commit/aed41621a103ca83b134070c0a11b3c1ed5d6922). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8531] [ML] Update ML user guide for Min...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7211#issuecomment-130550285 [Test build #40741 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40741/console) for PR 7211 at commit [`b6ac0fc`](https://github.com/apache/spark/commit/b6ac0fc6eae4a03e3498891e9ee4ebfde418af8f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130550969 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9885] [SQL] Also pass barrierPrefixes a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8158#issuecomment-130553153 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130553847 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130553715 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9885] [SQL] Also pass barrierPrefixes a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8158#issuecomment-130552920 [Test build #40728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40728/console) for PR 8158 at commit [`2a134d5`](https://github.com/apache/spark/commit/2a134d548a64cdf3fc5299878262e98580d7eaa4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9757] [SQL] Fixes persistence of Parque...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8130#issuecomment-130558263 Verified under my local MySQL backed Hive 0.13.1 metastore and it works. Merging to master and branch-1.5. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9885] [SQL] Also pass barrierPrefixes a...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8158 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-8949 - Print warnings when using preferr...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/7874#discussion_r36947100 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -118,9 +118,11 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli * Can be generated using [[org.apache.spark.scheduler.InputFormatInfo.computePreferredLocations]] * from a list of input files or InputFormats for the application. */ + @Deprecated(Passing in preferred locations has no effect at all, see SPARK-8949) --- End diff -- this should be ```scala @deprecated((Passing in preferred locations has no effect at all, see SPARK-8949, 1.5.0) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36949342
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
---
@@ -554,6 +556,21 @@ abstract class HadoopFsRelationTest extends QueryTest
with SQLTestUtils {
clonedConf.foreach(entry = configuration.set(entry.getKey,
entry.getValue))
}
}
+
+ test(SPARK-8887: Explicitly define which data types can be used as
dynamic partition columns) {
+val df = Seq(
+ (1, v1, Date.valueOf(2015-08-10)),
+ (2, v2, Date.valueOf(2015-08-11)),
+ (3, v3, Date.valueOf(2015-08-12))).toDF(a, b, c)
+withTempDir { file =
+ intercept[AnalysisException] {
+
df.write.format(dataSourceName).partitionBy(c).save(file.getCanonicalPath)
+ }
+}
+intercept[AnalysisException] {
+ df.write.format(dataSourceName).partitionBy(c).saveAsTable(t)
+}
+ }
--- End diff --
We need to update this test case after putting `DateType` into the set of
valid types of partition columns.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9767] Remove ConnectionManager.
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/8161 [SPARK-9767] Remove ConnectionManager. We introduced the Netty network module for shuffle in Spark 1.2, and has turned it on by default for 3 releases. The old ConnectionManager is difficult to maintain. It's time to remove it for Spark 1.6. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-9767 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8161 commit bdf1d5e357fe423d356405acd0844991c652f150 Author: Reynold Xin r...@databricks.com Date: 2015-08-13T08:12:43Z [SPARK-9767] Remove ConnectionManager. We introduced the Netty network module for shuffle in Spark 1.2, and has turned it on by default for 3 releases. The old ConnectionManager is difficult to maintain. It's time to remove it for Spark 1.6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130571180 [Test build #1545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1545/console) for PR 8111 at commit [`c4d44c9`](https://github.com/apache/spark/commit/c4d44c9139eff45048d849311c539dadda54004c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `case class QRDecomposition[QType, RType](Q: QType, R: RType)` * `class VectorUDT extends UserDefinedType[Vector] ` * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9868] [SQL] [WIP] reproduce failure
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8160#issuecomment-130572627 [Test build #1549 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1549/console) for PR 8160 at commit [`044d0c1`](https://github.com/apache/spark/commit/044d0c1ac723c77d84d349a57991ac86fb87de59). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/8132#discussion_r36951210
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
---
@@ -554,6 +556,21 @@ abstract class HadoopFsRelationTest extends QueryTest
with SQLTestUtils {
clonedConf.foreach(entry = configuration.set(entry.getKey,
entry.getValue))
}
}
+
+ test(SPARK-8887: Explicitly define which data types can be used as
dynamic partition columns) {
+val df = Seq(
+ (1, v1, Date.valueOf(2015-08-10)),
+ (2, v2, Date.valueOf(2015-08-11)),
+ (3, v3, Date.valueOf(2015-08-12))).toDF(a, b, c)
+withTempDir { file =
+ intercept[AnalysisException] {
+
df.write.format(dataSourceName).partitionBy(c).save(file.getCanonicalPath)
+ }
+}
+intercept[AnalysisException] {
+ df.write.format(dataSourceName).partitionBy(c).saveAsTable(t)
+}
--- End diff --
Oh I see, makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Testing Jenkins do not merge.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8139#issuecomment-130574566 [Test build #40740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40740/console) for PR 8139 at commit [`e4e2254`](https://github.com/apache/spark/commit/e4e225457b8eac1640f5e06974047e3aabc83642). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130579622 **[Test build #40739 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40739/console)** for PR 8111 at commit [`c4d44c9`](https://github.com/apache/spark/commit/c4d44c9139eff45048d849311c539dadda54004c) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9720] [ML] Identifiable types need UID ...
Github user BertrandDechoux commented on the pull request: https://github.com/apache/spark/pull/8062#issuecomment-130579425 Like I said, I didn't run the test. There seems to be no clear easy way. I will have to invest time to find out. Is there a way to see the result of amplab jenkins for this pull request? https://amplab.cs.berkeley.edu/jenkins/ @mengxr I will take into account your 2 points. They are indeed both relevant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130579959 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130585700 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9818][SQL][WIP]Revert SPARK-6136 to ena...
Github user yjshen commented on the pull request: https://github.com/apache/spark/pull/8101#issuecomment-130586263 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8155#issuecomment-130587485 **[Test build #40743 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40743/console)** for PR 8155 at commit [`d09b603`](https://github.com/apache/spark/commit/d09b6031c42391dacc1d0d4bad44050b5885b1f5) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9877][Core] Fix StandaloneRestServer NP...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8127#issuecomment-130587302 [Test build #40744 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40744/console) for PR 8127 at commit [`fdb6158`](https://github.com/apache/spark/commit/fdb6158f6439dce80bdbd01ef2e483265ca5eb84). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8155#issuecomment-130587553 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9877][Core] Fix StandaloneRestServer NP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8127#issuecomment-130587346 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9868] [SQL] [WIP] reproduce failure
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8160#issuecomment-130550743 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8155#issuecomment-130550893 [Test build #40743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40743/consoleFull) for PR 8155 at commit [`d09b603`](https://github.com/apache/spark/commit/d09b6031c42391dacc1d0d4bad44050b5885b1f5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9877][Core] Fix StandaloneRestServer NP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8127#issuecomment-130550728 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9877][Core] Fix StandaloneRestServer NP...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8127#issuecomment-130550715 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9832] [SQL] add a thread-safe lookup fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8151#issuecomment-130550929 [Test build #1536 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1536/console) for PR 8151 at commit [`8daa2cd`](https://github.com/apache/spark/commit/8daa2cd1d8431fa6e10d6ec664e20e50da6f0139). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9868] [SQL] [WIP] reproduce failure
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8160#issuecomment-130552786 [Test build #1549 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1549/consoleFull) for PR 8160 at commit [`044d0c1`](https://github.com/apache/spark/commit/044d0c1ac723c77d84d349a57991ac86fb87de59). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9929][SQL] support metadata in withColu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8159#issuecomment-130555772 [Test build #40732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40732/console) for PR 8159 at commit [`4698d05`](https://github.com/apache/spark/commit/4698d05db5e874cc6cb7aa3dced022809bf3ba3d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130555864 [Test build #40746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40746/consoleFull) for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9929][SQL] support metadata in withColu...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8159#issuecomment-130555873 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9925] [SQL] [TESTS] Set SQLConf.SHUFFLE...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/8155#issuecomment-130562648 Here's another place where this configuration is changed https://github.com/apache/spark/blob/84a27916a62980c8fcb0977c3a7fdb73c0bd5812/sql/core/src/test/scala/org/apache/spark/sql/SQLConfSuite.scala#L77 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130565139 [Test build #1540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1540/console) for PR 8111 at commit [`828144f`](https://github.com/apache/spark/commit/828144f96e4454824887c1d01fada20ce3510610). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9868] [SQL] [WIP] reproduce failure
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8160#issuecomment-130566615 [Test build #1546 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1546/console) for PR 8160 at commit [`044d0c1`](https://github.com/apache/spark/commit/044d0c1ac723c77d84d349a57991ac86fb87de59). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9580] [SQL] Replace singletons in SQL t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8111#issuecomment-130566573 [Test build #1544 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1544/console) for PR 8111 at commit [`c4d44c9`](https://github.com/apache/spark/commit/c4d44c9139eff45048d849311c539dadda54004c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Identifiable ` * `class VectorUDT extends UserDefinedType[Vector] ` * ` implicit class StringToColumn(val sc: StringContext) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130566804 [Test build #40747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40747/consoleFull) for PR 8126 at commit [`5fa5f7a`](https://github.com/apache/spark/commit/5fa5f7aafbd547d50eaa17f7f657ba29751babfa). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130573459 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9934] Deprecate NIO ConnectionManager.
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/8162 [SPARK-9934] Deprecate NIO ConnectionManager. Deprecate NIO ConnectionManager in Spark 1.5.0, before removing it in Spark 1.6.0. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark SPARK-9934 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8162.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8162 commit 4fb536dcd38b21d5844611c03e436edff996006f Author: Reynold Xin r...@databricks.com Date: 2015-08-13T08:27:03Z [SPARK-9934] Deprecate NIO ConnectionManager. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9661] [MLLib] [ML] Java compatibility
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8126#issuecomment-130573340 [Test build #40747 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40747/console) for PR 8126 at commit [`5fa5f7a`](https://github.com/apache/spark/commit/5fa5f7aafbd547d50eaa17f7f657ba29751babfa). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130587234 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130587220 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130587199 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605258 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130605274 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130605400 [Test build #40756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40756/consoleFull) for PR 8132 at commit [`d926a61`](https://github.com/apache/spark/commit/d926a615bbbcb40aaeba2a977c9c8c4b1787ecd2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130591634 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9882][Core] Priority-based scheduling f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8129#issuecomment-130591580 **[Test build #40746 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40746/console)** for PR 8129 at commit [`867c941`](https://github.com/apache/spark/commit/867c9417d3119f53f055bb5064a8977b9ddd8304) after a configured wait of `175m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user yjshen commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130591739 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8887][SQL]Explicit define which data ty...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8132#issuecomment-130604506 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
