[GitHub] spark pull request #16308: [SPARK-18350][SQL][WIP] Support session local tim...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/16308#discussion_r92982685
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -111,7 +112,8 @@ case class CatalogTablePartition(
*/
   def toRow(partitionSchema: StructType): InternalRow = {
 InternalRow.fromSeq(partitionSchema.map { field =>
-  Cast(Literal(spec(field.name)), field.dataType).eval()
+  Cast(Literal(spec(field.name)), field.dataType,
+DateTimeUtils.defaultTimeZone().getID).eval()
--- End diff --

Hmm, now I think we should use timezone settings for partition values, 
because the values are also parts of data so they should be affected by the 
settings.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16291
  
**[Test build #70345 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70345/testReport)**
 for PR 16291 at commit 
[`66e4f12`](https://github.com/apache/spark/commit/66e4f12c726d7cc49620f9c5747ab6b1d70d5bf3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16290
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70335/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16290
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16290
  
**[Test build #70335 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70335/testReport)**
 for PR 16290 at commit 
[`f7b4772`](https://github.com/apache/spark/commit/f7b4772a4b72c28047afb7d614e3af3317af896d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/13909#discussion_r92979265
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
 ---
@@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) 
extends Expression {
   }
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-val arrayClass = classOf[GenericArrayData].getName
-val values = ctx.freshName("values")
-ctx.addMutableState("Object[]", values, s"this.$values = null;")
+val array = ctx.freshName("array")
 
-ev.copy(code = s"""
-  this.$values = new Object[${children.size}];""" +
+val et = dataType.elementType
+val evals = children.map(e => e.genCode(ctx))
+val isPrimitiveArray = ctx.isPrimitiveType(et)
+val primitiveTypeName = if (isPrimitiveArray) 
ctx.primitiveTypeName(et) else ""
+val (preprocess, arrayData, arrayWriter) =
+  GenArrayData.getCodeArrayData(ctx, et, children.size, 
isPrimitiveArray, array)
+
+ev.copy(code =
+  preprocess +
   ctx.splitExpressions(
 ctx.INPUT_ROW,
-children.zipWithIndex.map { case (e, i) =>
-  val eval = e.genCode(ctx)
-  eval.code + s"""
-if (${eval.isNull}) {
-  $values[$i] = null;
+evals.zipWithIndex.map { case (eval, i) =>
+  eval.code +
+(if (isPrimitiveArray) {
+  (if (!children(i).nullable) {
+s"\n$arrayWriter.write($i, ${eval.value});"
+  } else {
+s"""
+if (${eval.isNull}) {
+  $arrayWriter.setNull$primitiveTypeName($i);
+} else {
+  $arrayWriter.write($i, ${eval.value});
+}
+   """
+  })
 } else {
-  $values[$i] = ${eval.value};
-}
-   """
+  s"""
+  if (${eval.isNull}) {
+$array[$i] = null;
+  } else {
+$array[$i] = ${eval.value};
+  }
+ """
+})
 }) +
-  s"""
-final ArrayData ${ev.value} = new $arrayClass($values);
-this.$values = null;
-  """, isNull = "false")
+  s"\nfinal ArrayData ${ev.value} = $arrayData;\n",
+  isNull = "false")
   }
 
   override def prettyName: String = "array"
 }
 
+private [sql] object GenArrayData {
+  // This function returns Java code pieces based on DataType and 
isPrimitive
+  // for allocation of ArrayData class
+  def getCodeArrayData(
+  ctx: CodegenContext,
+  dt: DataType,
+  size: Int,
+  isPrimitive : Boolean,
+  array: String): (String, String, String) = {
+if (!isPrimitive) {
+  val arrayClass = classOf[GenericArrayData].getName
+  ctx.addMutableState("Object[]", array,
+s"this.$array = new Object[${size}];")
+  ("", s"new $arrayClass($array)", null)
+} else {
+  val holder = ctx.freshName("holder")
+  val arrayWriter = ctx.freshName("createArrayWriter")
+  val unsafeArrayClass = classOf[UnsafeArrayData].getName
+  val holderClass = classOf[BufferHolder].getName
+  val arrayWriterClass = classOf[UnsafeArrayWriter].getName
+  ctx.addMutableState(unsafeArrayClass, array, "")
+  ctx.addMutableState(holderClass, holder, "")
+  ctx.addMutableState(arrayWriterClass, arrayWriter, "")
+  val baseOffset = Platform.BYTE_ARRAY_OFFSET
+  val unsafeArraySizeInBytes =
+UnsafeArrayData.calculateHeaderPortionInBytes(size) +
+ByteArrayMethods.roundNumberOfBytesToNearestWord(dt.defaultSize * 
size)
+
+  (s"""
+$array = new $unsafeArrayClass();
+$holder = new $holderClass($unsafeArraySizeInBytes);
+$arrayWriter = new $arrayWriterClass();
--- End diff --

@viirya `org.apache.spark.sql.DataFrameSuite.Star Expansion - CreateStruct 
and CreateArray` failed at [this 
test](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69969/testReport/).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: 

[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/13909#discussion_r92979329
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
 ---
@@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) 
extends Expression {
   }
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-val arrayClass = classOf[GenericArrayData].getName
-val values = ctx.freshName("values")
-ctx.addMutableState("Object[]", values, s"this.$values = null;")
+val array = ctx.freshName("array")
 
-ev.copy(code = s"""
-  this.$values = new Object[${children.size}];""" +
+val et = dataType.elementType
+val evals = children.map(e => e.genCode(ctx))
+val isPrimitiveArray = ctx.isPrimitiveType(et)
+val primitiveTypeName = if (isPrimitiveArray) 
ctx.primitiveTypeName(et) else ""
+val (preprocess, arrayData, arrayWriter) =
+  GenArrayData.getCodeArrayData(ctx, et, children.size, 
isPrimitiveArray, array)
+
+ev.copy(code =
+  preprocess +
   ctx.splitExpressions(
 ctx.INPUT_ROW,
-children.zipWithIndex.map { case (e, i) =>
-  val eval = e.genCode(ctx)
-  eval.code + s"""
-if (${eval.isNull}) {
-  $values[$i] = null;
+evals.zipWithIndex.map { case (eval, i) =>
+  eval.code +
+(if (isPrimitiveArray) {
+  (if (!children(i).nullable) {
+s"\n$arrayWriter.write($i, ${eval.value});"
+  } else {
+s"""
+if (${eval.isNull}) {
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16335
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16335
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70334/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16335
  
**[Test build #70334 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70334/testReport)**
 for PR 16335 at commit 
[`c5d6938`](https://github.com/apache/spark/commit/c5d69388823aa7fc9d75826cabacfb666ee665d5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16321: [SPARK-18031][Tests]Fix flaky test ExecutorAllocationMan...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16321
  
**[Test build #70344 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70344/testReport)**
 for PR 16321 at commit 
[`471740b`](https://github.com/apache/spark/commit/471740b816014234359458fdcac73a6599e16a5b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16282
  
**[Test build #70343 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70343/testReport)**
 for PR 16282 at commit 
[`fd63513`](https://github.com/apache/spark/commit/fd63513d9e1ac3978f7ffe29382aff46ad86027b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16322: [SPARK-18908][SS] Creating StreamingQueryException shoul...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16322
  
**[Test build #70342 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70342/testReport)**
 for PR 16322 at commit 
[`2044bc0`](https://github.com/apache/spark/commit/2044bc0ebcb7eb760ba1b32a5a1d3cec6066a6dc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16335
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16335
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70333/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16335
  
**[Test build #70333 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70333/testReport)**
 for PR 16335 at commit 
[`e309285`](https://github.com/apache/spark/commit/e3092856fa207d1fcc118580d77a64bc653c62be).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16337
  
cc @hvanhovell @nsyca 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16337
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/16282
  
SPARK-18907


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16282: [DO_NOT_MERGE]Try to fix kafka

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/16282
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN subquery

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16337
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14365
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14365
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70336/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16336
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16336
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70340/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16336
  
**[Test build #70340 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70340/testReport)**
 for PR 16336 at commit 
[`2aa11d8`](https://github.com/apache/spark/commit/2aa11d831401969a1c6052ece749ec4d56beb117).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14365
  
**[Test build #70336 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)**
 for PR 14365 at commit 
[`141cb1d`](https://github.com/apache/spark/commit/141cb1d5232a14d72f6411afdb1212cf7ef4de0a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16337: [SPARK-18871][SQL] New test cases for IN/NOT IN s...

[kevinyu98]

GitHub user kevinyu98 opened a pull request:

https://github.com/apache/spark/pull/16337

[SPARK-18871][SQL] New test cases for IN/NOT IN subquery

## What changes were proposed in this pull request?
This PR extends the existing IN/NOT IN subquery test cases coverage, adding 
these cases:

- simple case

- GROUP BY

- ORDER BY

- HAVING

- JOIN

- UNION | UNION ALL | UNION DISTINCT | INTERSECT | EXCEPT 

- CTE


## How was this patch tested?

This patch is adding tests.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kevinyu98/spark spark-18871

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16337.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16337


commit 3b44c5978bd44db986621d3e8511e9165b66926b
Author: Kevin Yu 
Date:   2016-04-20T18:06:30Z

adding testcase

commit 18b4a31c687b264b50aa5f5a74455956911f738a
Author: Kevin Yu 
Date:   2016-04-22T21:48:00Z

Merge remote-tracking branch 'upstream/master'

commit 4f4d1c8f2801b1e662304ab2b33351173e71b427
Author: Kevin Yu 
Date:   2016-04-23T16:50:19Z

Merge remote-tracking branch 'upstream/master'
get latest code from upstream

commit f5f0cbed1eb5754c04c36933b374c3b3d2ae4f4e
Author: Kevin Yu 
Date:   2016-04-23T22:20:53Z

Merge remote-tracking branch 'upstream/master'
adding trim characters support

commit d8b2edbd13ee9a4f057bca7dcb0c0940e8e867b8
Author: Kevin Yu 
Date:   2016-04-25T20:24:33Z

Merge remote-tracking branch 'upstream/master'
get latest code for pr12646

commit 196b6c66b0d55232f427c860c0e7c6876c216a67
Author: Kevin Yu 
Date:   2016-04-25T23:45:57Z

Merge remote-tracking branch 'upstream/master'
merge latest code

commit f37a01e005f3e27ae2be056462d6eb6730933ba5
Author: Kevin Yu 
Date:   2016-04-27T14:15:06Z

Merge remote-tracking branch 'upstream/master'
merge upstream/master

commit bb5b01fd3abeea1b03315eccf26762fcc23f80c0
Author: Kevin Yu 
Date:   2016-04-30T23:49:31Z

Merge remote-tracking branch 'upstream/master'

commit bde5820a181cf84e0879038ad8c4cebac63c1e24
Author: Kevin Yu 
Date:   2016-05-04T03:52:31Z

Merge remote-tracking branch 'upstream/master'

commit 5f7cd96d495f065cd04e8e4cc58461843e45bc8d
Author: Kevin Yu 
Date:   2016-05-10T21:14:50Z

Merge remote-tracking branch 'upstream/master'

commit 893a49af0bfd153ccb59ba50b63a232660e0eada
Author: Kevin Yu 
Date:   2016-05-13T18:20:39Z

Merge remote-tracking branch 'upstream/master'

commit 4bbe1fd4a3ebd50338ccbe07dc5887fe289cd53d
Author: Kevin Yu 
Date:   2016-05-17T21:58:14Z

Merge remote-tracking branch 'upstream/master'

commit b2dd795e23c36cbbd022f07a10c0cf21c85eb421
Author: Kevin Yu 
Date:   2016-05-18T06:37:13Z

Merge remote-tracking branch 'upstream/master'

commit 8c3e5da458dbff397ed60fcb68f2a46d87ab7ba4
Author: Kevin Yu 
Date:   2016-05-18T16:18:16Z

Merge remote-tracking branch 'upstream/master'

commit a0eaa408e847fbdc3ac5b26348588ee0a1e276c7
Author: Kevin Yu 
Date:   2016-05-19T04:28:20Z

Merge remote-tracking branch 'upstream/master'

commit d03c940ed89795fa7fe1d1e9f511363b22cdf19d
Author: Kevin Yu 
Date:   2016-05-19T21:24:33Z

Merge remote-tracking branch 'upstream/master'

commit d728d5e002082e571ac47292226eb8b2614f479f
Author: Kevin Yu 
Date:   2016-05-24T20:32:57Z

Merge remote-tracking branch 'upstream/master'

commit ea104ddfbf7d180ed1bc53dd9a1005010264aa1f
Author: Kevin Yu 
Date:   2016-05-25T22:52:57Z

Merge remote-tracking branch 'upstream/master'

commit 6ab1215b781ad0cccf1752f3a625b4e4e371c38e
Author: Kevin Yu 
Date:   2016-05-27T17:18:46Z

Merge remote-tracking branch 'upstream/master'

commit 0c566533705331697eb1b287b30c8b16111f6fa2
Author: Kevin Yu 
Date:   2016-06-01T06:48:57Z

Merge remote-tracking branch 'upstream/master'

commit d7a187490b31185d0a803cbbdeda67cb26c40056
Author: Kevin Yu 
Date:   2016-06-01T22:55:17Z

Merge remote-tracking branch 'upstream/master'

commit 85d35002ce864d5ce6fd3be7215a868a8867caf9
Author: Kevin Yu 
Date:   2016-06-02T14:08:30Z

Merge remote-tracking branch 'upstream/master'

commit c056f91036ec75d1e2c93f6f47ad842eb28a3e0b
Author: Kevin Yu 
Date:   2016-06-03T06:06:51Z

Merge remote-tracking branch 'upstream/master'

commit 0b8189dd454897ae73bb3a5ffc245b2c65f6b226
Author: Kevin Yu 
Date:   

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16336
  
@dongjoon-hyun Just to make sure, could we actually build each 
individually, `cd ./python/docs && make html` for Python and `cd ./R && sh 
create-docs.sh` for R? (FWIW, I support this PR even if so because now these 
can be done in a single place).

BTW, I believe it would be nicer if the options are documented in 
https://github.com/apache/spark/blob/master/docs/README.md



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16290
  
After a research, for avoiding this flaky testcase, the simplest way is to 
remove the contents in `metastore_db` (whose location is specified through 
`javax.jdo.option.ConnectionURL`). 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16335
  
**[Test build #70341 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70341/testReport)**
 for PR 16335 at commit 
[`7ad462a`](https://github.com/apache/spark/commit/7ad462ade573688dcd3ad963d65395689d9b9bcc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16335
  

Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=6AC0B3E0-95BF-4B24-AF41-47DE7360FAB9=true)](https://ci.appveyor.com/project/spark-test/spark/branch/6AC0B3E0-95BF-4B24-AF41-47DE7360FAB9)
Build started: [TESTS] 
`org.apache.spark.sql.execution.datasources.json.JsonSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=5164A852-4B91-485F-ACF6-40CE627F2E83=true)](https://ci.appveyor.com/project/spark-test/spark/branch/5164A852-4B91-485F-ACF6-40CE627F2E83)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.state.StateStoreSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=2ED3B413-EF9F-4BF1-9CFE-BDC956D37E9A=true)](https://ci.appveyor.com/project/spark-test/spark/branch/2ED3B413-EF9F-4BF1-9CFE-BDC956D37E9A)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F9445F0A-49C1-45AF-9B14-C1328E5EC740=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F9445F0A-49C1-45AF-9B14-C1328E5EC740)
Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=98335C35-A44F-4C47-B3ED-4A8F73525878=true)](https://ci.appveyor.com/project/spark-test/spark/branch/98335C35-A44F-4C47-B3ED-4A8F73525878)
Diff: 
https://github.com/apache/spark/compare/master...spark-test:98335C35-A44F-4C47-B3ED-4A8F73525878


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92975296
  
--- Diff: 
core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala ---
@@ -35,6 +36,8 @@ class LauncherBackendSuite extends SparkFunSuite with 
Matchers {
 
   tests.foreach { case (name, master) =>
 test(s"$name: launcher handle") {
+  // The tests here are failed due to the cmd length limitation up to 
8K on Windows.
+  assume(!Utils.isWindows)
--- End diff --

This test is being failed due to cmd length limitation on Windows up to 8K. 
The actual command is roughly 16K (see 
https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16336: [SPARK-18923][DOC][BUILD] Support skipping R/Python API ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16336
  
**[Test build #70340 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70340/testReport)**
 for PR 16336 at commit 
[`2aa11d8`](https://github.com/apache/spark/commit/2aa11d831401969a1c6052ece749ec4d56beb117).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16336: [SPARK-API-DOC][DOC][BUILD] Support skipping R/Py...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/16336

[SPARK-API-DOC][DOC][BUILD] Support skipping R/Python API docs

## What changes were proposed in this pull request?

This PR aims to support `SKIP_PYTHONDOC` and `SKIP_RDOC` for documentation 
build. Currently, we can use `SKIP_SCALADOC` or `SKIP_API`. The reason 
providing additional options is that the Spark documentation build uses a 
number of tools to build HTML docs and API docs in Scala, Python and R. 
Especially,
- Python API docs requires `sphinx`.
- R API docs requires `R` installation and `knitr` (and more others 
libraries).

In other words, we cannot generate Python API docs without R installation. 
Also, we cannot generate R API docs without Python `sphinx` installation. If 
Spark provides `SKIP_PYTHONDOC` and `SKIP_RDOC` like `SKIP_SCALADOC`, it would 
be more convenient.

## How was this patch tested?

Manual.

**Skipping Scala/Java/Python API Doc Build**
```bash
$ cd docs
$ SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 jekyll build
$ ls api
DESCRIPTION R
```

**Skipping Scala/Java/R API Doc Build**
```bash
$ cd docs
$ SKIP_SCALADOC=1 SKIP_RDOC=1 jekyll build
$ ls api
python
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-18923

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16336


commit 2aa11d831401969a1c6052ece749ec4d56beb117
Author: Dongjoon Hyun 
Date:   2016-12-19T05:19:48Z

[SPARK-API-DOC][DOC][BUILD] Support skipping R/Python API docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16313
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16313
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70331/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16313
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16313
  
**[Test build #70331 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70331/testReport)**
 for PR 16313 at commit 
[`32857e6`](https://github.com/apache/spark/commit/32857e6c5fa89094b84d4ed78469217af8c515c7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16330
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70337/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16330
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16330
  
**[Test build #70337 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)**
 for PR 16330 at commit 
[`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16308: [SPARK-18350][SQL][WIP] Support session local tim...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/16308#discussion_r92973560
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -111,7 +112,8 @@ case class CatalogTablePartition(
*/
   def toRow(partitionSchema: StructType): InternalRow = {
 InternalRow.fromSeq(partitionSchema.map { field =>
-  Cast(Literal(spec(field.name)), field.dataType).eval()
+  Cast(Literal(spec(field.name)), field.dataType,
+DateTimeUtils.defaultTimeZone().getID).eval()
--- End diff --

Currently the behavior doesn't change by timezone setting, i.e. using 
system timezone.

This is a part that I was not sure which we should handle the partition 
values, use timezone settings or system timezone.
Should we use timezone settings?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16323: [SPARK-18911] [SQL] Define CatalogStatistics to interact...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16323
  
**[Test build #70339 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70339/testReport)**
 for PR 16323 at commit 
[`bd5eacc`](https://github.com/apache/spark/commit/bd5eaccabe6bdfcf3d7afbdcdb15676aaad78275).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16323: [SPARK-18911] [SQL] Define CatalogStatistics to interact...

[wzhfy]

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/16323
  
Ok, I think it's doable. But since it's not a small change, let's wait 
@rxin for his comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16334: estimate filter cardinality

[ron8hu]

Github user ron8hu commented on the issue:

https://github.com/apache/spark/pull/16334
  
Sorry.  This is a mistake.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16334: estimate filter cardinality

[ron8hu]

Github user ron8hu closed the pull request at:

https://github.com/apache/spark/pull/16334


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13909: [SPARK-16213][SQL] Reduce runtime overhead of a p...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/13909#discussion_r92972394
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala
 ---
@@ -56,33 +58,93 @@ case class CreateArray(children: Seq[Expression]) 
extends Expression {
   }
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
-val arrayClass = classOf[GenericArrayData].getName
-val values = ctx.freshName("values")
-ctx.addMutableState("Object[]", values, s"this.$values = null;")
+val array = ctx.freshName("array")
 
-ev.copy(code = s"""
-  this.$values = new Object[${children.size}];""" +
+val et = dataType.elementType
+val evals = children.map(e => e.genCode(ctx))
+val isPrimitiveArray = ctx.isPrimitiveType(et)
+val primitiveTypeName = if (isPrimitiveArray) 
ctx.primitiveTypeName(et) else ""
+val (preprocess, arrayData, arrayWriter) =
+  GenArrayData.getCodeArrayData(ctx, et, children.size, 
isPrimitiveArray, array)
+
+ev.copy(code =
+  preprocess +
   ctx.splitExpressions(
 ctx.INPUT_ROW,
-children.zipWithIndex.map { case (e, i) =>
-  val eval = e.genCode(ctx)
-  eval.code + s"""
-if (${eval.isNull}) {
-  $values[$i] = null;
+evals.zipWithIndex.map { case (eval, i) =>
+  eval.code +
+(if (isPrimitiveArray) {
+  (if (!children(i).nullable) {
+s"\n$arrayWriter.write($i, ${eval.value});"
+  } else {
+s"""
+if (${eval.isNull}) {
+  $arrayWriter.setNull$primitiveTypeName($i);
+} else {
+  $arrayWriter.write($i, ${eval.value});
+}
+   """
+  })
 } else {
-  $values[$i] = ${eval.value};
-}
-   """
+  s"""
+  if (${eval.isNull}) {
+$array[$i] = null;
+  } else {
+$array[$i] = ${eval.value};
+  }
+ """
+})
 }) +
-  s"""
-final ArrayData ${ev.value} = new $arrayClass($values);
-this.$values = null;
-  """, isNull = "false")
+  s"\nfinal ArrayData ${ev.value} = $arrayData;\n",
+  isNull = "false")
   }
 
   override def prettyName: String = "array"
 }
 
+private [sql] object GenArrayData {
+  // This function returns Java code pieces based on DataType and 
isPrimitive
+  // for allocation of ArrayData class
+  def getCodeArrayData(
+  ctx: CodegenContext,
+  dt: DataType,
+  size: Int,
+  isPrimitive : Boolean,
+  array: String): (String, String, String) = {
+if (!isPrimitive) {
+  val arrayClass = classOf[GenericArrayData].getName
+  ctx.addMutableState("Object[]", array,
+s"this.$array = new Object[${size}];")
+  ("", s"new $arrayClass($array)", null)
+} else {
+  val holder = ctx.freshName("holder")
+  val arrayWriter = ctx.freshName("createArrayWriter")
+  val unsafeArrayClass = classOf[UnsafeArrayData].getName
+  val holderClass = classOf[BufferHolder].getName
+  val arrayWriterClass = classOf[UnsafeArrayWriter].getName
+  ctx.addMutableState(unsafeArrayClass, array, "")
+  ctx.addMutableState(holderClass, holder, "")
+  ctx.addMutableState(arrayWriterClass, arrayWriter, "")
+  val baseOffset = Platform.BYTE_ARRAY_OFFSET
+  val unsafeArraySizeInBytes =
+UnsafeArrayData.calculateHeaderPortionInBytes(size) +
+ByteArrayMethods.roundNumberOfBytesToNearestWord(dt.defaultSize * 
size)
+
+  (s"""
+$array = new $unsafeArrayClass();
+$holder = new $holderClass($unsafeArraySizeInBytes);
+$arrayWriter = new $arrayWriterClass();
--- End diff --

@kiszk Do you remember which test failed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16308: [SPARK-18350][SQL][WIP] Support session local timezone.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16308
  
**[Test build #70338 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70338/testReport)**
 for PR 16308 at commit 
[`b860379`](https://github.com/apache/spark/commit/b8603790c3201c691415461a132b84ba882dc758).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16330
  
**[Test build #70337 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70337/testReport)**
 for PR 16330 at commit 
[`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehous...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/16290#discussion_r92971642
  
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -72,6 +72,20 @@ test_that("repeatedly starting and stopping 
SparkSession", {
   }
 })
 
+test_that("Default warehouse dir should be set to tempdir", {
+  sparkR.session.stop()
+  sparkR.session(enableHiveSupport = FALSE)
+
+  # Create a temporary table
+  sql("CREATE TABLE people_warehouse_test")
+  # spark-warehouse should be written only tempdir() and not current 
working directory
+  res <- list.files(path = ".", pattern = ".*spark-warehouse.*",
--- End diff --

That I agree completely


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16330
  
that's weird, I'm seeing a lot of seemingly unrelated flaky test failure 
lately?

```

org.apache.spark.util.collection.ExternalSorterSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(ExternalSorterSuite.scala:32)
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC 
overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16330
  
jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16290
  
I checked the most two recent failed test cases in Jenkins. They are not 
related to the changes in the PR. 

In the local environment, I can reproduce the error you mentioned above.  
```
[info]   2016-12-18 20:47:23.328 - stdout> path1: 
file:/Users/xiaoli/IdeaProjects/sparkDelivery/bin/spark-warehouse
[info]   2016-12-18 20:47:23.328 - stdout> path2: 
file:/Users/xiaoli/IdeaProjects/sparkDelivery/spark-warehouse/
```

The location of default database is still pointing to the original value of 
`hive.metastore.warehouse.dir` or `spark.sql.warehouse.dir` that was set in the 
previous test case or our previous local spark job. Ideally, our test suite 
should directly connect to Derby and drop the default database. Let me do more 
search. 

Also cc @yhuai 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14365
  
**[Test build #70336 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70336/testReport)**
 for PR 14365 at commit 
[`141cb1d`](https://github.com/apache/spark/commit/141cb1d5232a14d72f6411afdb1212cf7ef4de0a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16290
  
**[Test build #70335 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70335/testReport)**
 for PR 16290 at commit 
[`f7b4772`](https://github.com/apache/spark/commit/f7b4772a4b72c28047afb7d614e3af3317af896d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16290
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13909
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13909
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70327/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13909: [SPARK-16213][SQL] Reduce runtime overhead of a program ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13909
  
**[Test build #70327 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70327/testReport)**
 for PR 13909 at commit 
[`d24c7b1`](https://github.com/apache/spark/commit/d24c7b1e13c00b722cafb7230e1e59021f42eee4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16330
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16330
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70332/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16330
  
**[Test build #70332 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)**
 for PR 16330 at commit 
[`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehous...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/16290#discussion_r92968560
  
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -72,6 +72,20 @@ test_that("repeatedly starting and stopping 
SparkSession", {
   }
 })
 
+test_that("Default warehouse dir should be set to tempdir", {
+  sparkR.session.stop()
+  sparkR.session(enableHiveSupport = FALSE)
+
+  # Create a temporary table
+  sql("CREATE TABLE people_warehouse_test")
+  # spark-warehouse should be written only tempdir() and not current 
working directory
+  res <- list.files(path = ".", pattern = ".*spark-warehouse.*",
--- End diff --

I think my bigger concern for that is that usually tests are run all at 
time - i.e. core, sql, hive and then python, R. And there are no guarantees 
that other module tests won't create files inside `SPARK_HOME` afaik. So while 
we can check some basic things with our test, I dont think verifying a global 
property is always possible.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16232: [SPARK-18800][SQL] Fix UnsafeKVExternalSorter by correct...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/16232
  
ping @hvanhovell Do you think this fixing good now? Thanks.

cc @davies @JoshRosen


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16335
  
cc @srowen, do you mind if I ask to take a look please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16245: [SPARK-18824][SQL] Add optimizer rule to reorder Filter ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/16245
  
ping @hvanhovell @rxin Any thing I need to address more? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16335
  
Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=83107543-90AA-4CEB-B33C-C6EDEBF5D7A7=true)](https://ci.appveyor.com/project/spark-test/spark/branch/83107543-90AA-4CEB-B33C-C6EDEBF5D7A7)
Build started: [TESTS] 
`org.apache.spark.sql.execution.datasources.json.JsonSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=62FB21FD-889B-4A64-9BAF-D392C7997466=true)](https://ci.appveyor.com/project/spark-test/spark/branch/62FB21FD-889B-4A64-9BAF-D392C7997466)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.state.StateStoreSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=E0E0715A-4D36-430C-989F-7EBE7DBB00D1=true)](https://ci.appveyor.com/project/spark-test/spark/branch/E0E0715A-4D36-430C-989F-7EBE7DBB00D1)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F40EEC55-B520-4FD2-B49B-46E281C63887=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F40EEC55-B520-4FD2-B49B-46E281C63887)
Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=ECB4A22F-AC5F-46B4-AE80-BCB22C4F6C4C=true)](https://ci.appveyor.com/project/spark-test/spark/branch/ECB4A22F-AC5F-46B4-AE80-BCB22C4F6C4C)
Diff: 
https://github.com/apache/spark/compare/master...spark-test:F40EEC55-B520-4FD2-B49B-46E281C63887


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16263: [SPARK-18281][SQL][PySpark] Consumes the returned local ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/16263
  
ping @davies Can you take a look again? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92968297
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -209,14 +209,20 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
 }
 
 // Open and delete
-val f1 = fm.open(path)
-fm.delete(path)
-assert(!fm.exists(path))
-intercept[IOException] {
-  fm.open(path)
+if (Utils.isWindows) {
+  fm.open(path).close()
+  fm.delete(path)
+  fm.delete(path) // should not throw exception
--- End diff --

Windows holds an exclusive lock so it does not allow to remove a file when 
it is open. So, I re-wrote the test cases for WIndows.

cc @tdas, do you mind if I ask to see whether these tests on Windows are 
fine?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14365: [SPARK-16628][SQL] Translate file-based relation schema ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/14365
  
ping @cloud-fan @yhuai May you take a look? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92968267
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -209,14 +209,20 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
 }
 
 // Open and delete
-val f1 = fm.open(path)
-fm.delete(path)
-assert(!fm.exists(path))
-intercept[IOException] {
-  fm.open(path)
+if (Utils.isWindows) {
+  fm.open(path).close()
+  fm.delete(path)
+  fm.delete(path) // should not throw exception
+} else {
+  Utils.tryWithResource(fm.open(path)) { _ =>
--- End diff --

We should close when it emits an exception. Otherwise, it is possible to 
suppress the actual assertion error due to the exception in removing the path, 
that makes the debug harder.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16335
  
**[Test build #70334 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70334/testReport)**
 for PR 16335 at commit 
[`c5d6938`](https://github.com/apache/spark/commit/c5d69388823aa7fc9d75826cabacfb666ee665d5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16331
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70326/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16331
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16331: [SPARK-18920][HISTORYSERVER]Update outdated date formatt...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16331
  
**[Test build #70326 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70326/testReport)**
 for PR 16331 at commit 
[`1618d1d`](https://github.com/apache/spark/commit/1618d1d7855a6c2bdb01387330854b6e6159dd61).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16335
  
cc @srowen, do you mind if I ask to take a look please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92967906
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -209,14 +209,21 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
 }
 
 // Open and delete
-val f1 = fm.open(path)
-fm.delete(path)
-assert(!fm.exists(path))
-intercept[IOException] {
-  fm.open(path)
+// Open and delete
+if (Utils.isWindows) {
+  Utils.tryWithResource(fm.open(path))(_ => ())
+  fm.delete(path)
+  fm.delete(path) // should not throw exception
--- End diff --

cc @tdas, do you mind if I ask to see whether these tests on Windows are 
fine?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92967880
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -209,14 +209,21 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
 }
 
 // Open and delete
-val f1 = fm.open(path)
-fm.delete(path)
-assert(!fm.exists(path))
-intercept[IOException] {
-  fm.open(path)
+// Open and delete
+if (Utils.isWindows) {
+  Utils.tryWithResource(fm.open(path))(_ => ())
+  fm.delete(path)
+  fm.delete(path) // should not throw exception
+} else {
+  Utils.tryWithResource(fm.open(path)) { _ =>
--- End diff --

We should close when it emits an exception. Otherwise, it is possible to 
suppress the actual assertion error due to the exception in removing the path, 
that makes the debug harder.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92967755
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
 ---
@@ -88,14 +88,14 @@ class HDFSMetadataLogSuite extends SparkFunSuite with 
SharedSQLContext {
   s"fs.$scheme.impl",
   classOf[FakeFileSystem].getName)
 withTempDir { temp =>
-  val metadataLog = new HDFSMetadataLog[String](spark, 
s"$scheme://$temp")
+  val metadataLog = new HDFSMetadataLog[String](spark, 
s"$scheme://${temp.toURI.getPath}")
--- End diff --

This is using a fake scheme.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/16335#discussion_r92967723
  
--- Diff: 
core/src/test/scala/org/apache/spark/launcher/LauncherBackendSuite.scala ---
@@ -34,6 +35,8 @@ class LauncherBackendSuite extends SparkFunSuite with 
Matchers {
 "standalone/client" -> "local-cluster[1,1,1024]")
 
   tests.foreach { case (name, master) =>
+// The tests here are failed due to the cmd length limitation up to 8K 
on Windows.
+assume(!Utils.isWindows)
--- End diff --

This test is being failed due to cmd length limitation on Windows up to 8K. 
The actual command is roughly 16K (see 
https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16335
  
Build started: [TESTS] `org.apache.spark.sql.SQLQuerySuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=BFE6F12C-21BF-414F-B6D2-2A01025A963A=true)](https://ci.appveyor.com/project/spark-test/spark/branch/BFE6F12C-21BF-414F-B6D2-2A01025A963A)
Build started: [TESTS] 
`org.apache.spark.sql.execution.datasources.json.JsonSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=F76F11AC-3219-4AEA-800C-5D54666043B0=true)](https://ci.appveyor.com/project/spark-test/spark/branch/F76F11AC-3219-4AEA-800C-5D54666043B0)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.state.StateStoreSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=979C8BA8-CEB6-4056-B603-54168819FF35=true)](https://ci.appveyor.com/project/spark-test/spark/branch/979C8BA8-CEB6-4056-B603-54168819FF35)
Build started: [TESTS] 
`org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=58D6D032-EE90-4060-9DAB-C2F5DBAF978C=true)](https://ci.appveyor.com/project/spark-test/spark/branch/58D6D032-EE90-4060-9DAB-C2F5DBAF978C)
Build started: [TESTS] `org.apache.spark.launcher.LauncherBackendSuite` 
[![PR-16335](https://ci.appveyor.com/api/projects/status/github/spark-test/spark?branch=5726D75A-8D35-4C0C-B571-8413F5E8E1FC=true)](https://ci.appveyor.com/project/spark-test/spark/branch/5726D75A-8D35-4C0C-B571-8413F5E8E1FC)
Diff: 
https://github.com/apache/spark/compare/master...spark-test:F76F11AC-3219-4AEA-800C-5D54666043B0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16332
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70329/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16332
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16332: [SPARK-18921][SQL] check database existence with Hive.da...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16332
  
**[Test build #70329 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70329/testReport)**
 for PR 16332 at commit 
[`b1b152b`](https://github.com/apache/spark/commit/b1b152b0cb94c1f6b0a41ee3f13750d96c3420a0).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16335: [SPARK-18922][TESTS] Fix more resource-closing-related a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16335
  
**[Test build #70333 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70333/testReport)**
 for PR 16335 at commit 
[`e309285`](https://github.com/apache/spark/commit/e3092856fa207d1fcc118580d77a64bc653c62be).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16335: [SPARK-18922][TESTS] Fix more resource-closing-re...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/16335

[SPARK-18922][TESTS] Fix more resource-closing-related and path-related 
test failures in identified ones on Windows

## What changes were proposed in this pull request?

There are several tests failing due to resource-closing-related and 
path-related  problems on Windows as below.

- `SQLQuerySuite`:

```
- specifying database name for a temporary table is not allowed *** FAILED 
*** (125 milliseconds)
  org.apache.spark.sql.AnalysisException: Path does not exist: 
file:/C:projectsspark  arget mpspark-1f4471ab-aac0-4239-ae35-833d54b37e52;
  at 
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
  at 
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
```

- `JsonSuite`:

```
- Loading a JSON dataset from a text file with SQL *** FAILED *** (94 
milliseconds)
  org.apache.spark.sql.AnalysisException: Path does not exist: 
file:/C:projectsspark  arget mpspark-c918a8b7-fc09-433c-b9d0-36c0f78ae918;
  at 
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
  at 
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
```

- `StateStoreSuite`:

```
- SPARK-18342: commit fails when rename fails *** FAILED *** (16 
milliseconds)
  java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative 
path in absolute URI: 
StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0
  at org.apache.hadoop.fs.Path.initialize(Path.java:206)
  at org.apache.hadoop.fs.Path.(Path.java:116)
  at org.apache.hadoop.fs.Path.(Path.java:89)
  ...
  Cause: java.net.URISyntaxException: Relative path in absolute URI: 
StateStoreSuite29777261fs://C:%5Cprojects%5Cspark%5Ctarget%5Ctmp%5Cspark-ef349862-7281-4963-aaf3-add0d670a4ad%5C?-2218c2f8-2cf6-4f80-9cdf-96354e8246a77685899733421033312/0
  at java.net.URI.checkPath(URI.java:1823)
  at java.net.URI.(URI.java:745)
  at org.apache.hadoop.fs.Path.initialize(Path.java:203)
```

- `HDFSMetadataLogSuite`:

```
- FileManager: FileContextManager *** FAILED *** (94 milliseconds)
  java.io.IOException: Failed to delete: 
C:\projects\spark\target\tmp\spark-415bb0bd-396b-444d-be82-04599e025f21
  at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)
  at 
org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127)
  at 
org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38)

- FileManager: FileSystemManager *** FAILED *** (78 milliseconds)
  java.io.IOException: Failed to delete: 
C:\projects\spark\target\tmp\spark-ef8222cd-85aa-47c0-a396-bc7979e15088
  at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)
  at 
org.apache.spark.sql.test.SQLTestUtils$class.withTempDir(SQLTestUtils.scala:127)
  at 
org.apache.spark.sql.execution.streaming.HDFSMetadataLogSuite.withTempDir(HDFSMetadataLogSuite.scala:38)
```

And, there are some tests being failed due to the length limitation on cmd 
in Windows as below:

- `LauncherBackendSuite`:

```
- local: launcher handle *** FAILED *** (30 seconds, 120 milliseconds)
  The code passed to eventually never returned normally. Attempted 283 
times over 30.0960053 seconds. Last failure message: The reference was null. 
(LauncherBackendSuite.scala:56)
  org.scalatest.exceptions.TestFailedDueToTimeoutException:
  at 
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
  at 
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)

- standalone/client: launcher handle *** FAILED *** (30 seconds, 47 
milliseconds)
  The code passed to eventually never returned normally. Attempted 282 
times over 30.03798710002 seconds. Last failure message: The reference was 
null. (LauncherBackendSuite.scala:56)
  org.scalatest.exceptions.TestFailedDueToTimeoutException:
  at 
org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:420)
  at 
org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:438)
```

The executed command is, 
https://gist.github.com/HyukjinKwon/d3fdd2e694e5c022992838a618a516bd, which is 
16K length; however, the length limitation is 8K. So, it is being failed to 
launch.

This PR proposes to fix the test failures on Windows and skip the tests 
failed due to the length limitation

## How was this patch tested?

Manually tested via AppVeyor

**Before**

`SQLQuerySuite `: 

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16330
  
**[Test build #70332 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70332/testReport)**
 for PR 16330 at commit 
[`338b3c4`](https://github.com/apache/spark/commit/338b3c46ed2435daf2e780eab1ed04be3ed0b3ea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16330: [SPARK-18817][SPARKR][SQL] change derby log output and m...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16330
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16290: [SPARK-18817] [SPARKR] [SQL] Set default warehouse dir t...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16290
  
@shivaram with my PR #16330, `metastore_db` is in `tempdir` and is removed 
when the R process exits.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16313: [SPARK-18899][SPARK-18912][SPARK-18913][SQL] refactor th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16313
  
**[Test build #70331 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70331/testReport)**
 for PR 16313 at commit 
[`32857e6`](https://github.com/apache/spark/commit/32857e6c5fa89094b84d4ed78469217af8c515c7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16142: [SPARK-18716][CORE] Restrict the disk usage of sp...

[uncleGen]

Github user uncleGen closed the pull request at:

https://github.com/apache/spark/pull/16142


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16142: [SPARK-18716][CORE] Restrict the disk usage of spark eve...

[uncleGen]

Github user uncleGen commented on the issue:

https://github.com/apache/spark/pull/16142
  
@vanzin I will close this PR, but can't take your point completely.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16334: estimate filter cardinality

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16334
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16334: estimate filter cardinality

[ron8hu]

GitHub user ron8hu opened a pull request:

https://github.com/apache/spark/pull/16334

estimate filter cardinality

## What changes were proposed in this pull request?

This is a WIP PR. In this version, we set up the framework to traverse 
predicate and evaluate the equality (=) expression.

## How was this patch tested?

We just have a simple test case for now. More tests need to be added.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ron8hu/spark filterEstimate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16334.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16334


commit f0bbb43f7b27c59a3181ce428ed2bb0a7c1fc89d
Author: wangzhenhua 
Date:   2016-12-09T06:34:41Z

estimation for inner join

commit 64603b589bd1b971d2370d848f1dd19f11b52928
Author: wangzhenhua 
Date:   2016-12-09T07:30:33Z

wrong position of comments

commit 02bdf875babac7676a732a01fa41c8b8416a58d3
Author: Ron Hu 
Date:   2016-12-18T02:10:28Z

implemented first version of filter cardinality estimation

commit 6c9d06085f30dd19eb8b0a06f62742bda6ace736
Author: Ron Hu 
Date:   2016-12-18T02:15:57Z

maintained a mutable colStats in Filter class

commit 9e92b4b4c5033441f63e0e76544a1d609ac2a39e
Author: Ron Hu 
Date:   2016-12-19T00:57:00Z

Used mutableColStats to reflect changes for column stats

commit 8e77acc8630df3696dbb12453ee3f469c03b1193
Author: Ron Hu 
Date:   2016-12-19T02:15:41Z

use getOrElse to return filter stats

commit 62315985cc6ec20fb26c8ed109b33160e9f3f80b
Author: Ron Hu 
Date:   2016-12-19T03:27:40Z

added the test case




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16291
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70325/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16291
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16291: [SPARK-18838][CORE] Use separate executor service for ea...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16291
  
**[Test build #70325 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70325/testReport)**
 for PR 16291 at commit 
[`6763827`](https://github.com/apache/spark/commit/676382790b2e73c6ad3b4abe7c565ad266d9df1a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class StreamingQueryListenerBus(val sparkListenerBus: LiveListenerBus)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16325: [SPARK-18703] [SPARK-18675] [SQL] [BACKPORT-2.1] CTAS fo...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/16325
  
LGTM, merging to 2.1!

Let's send a new PR for 2.0 :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16333: Filter estimate

[ron8hu]

Github user ron8hu commented on the issue:

https://github.com/apache/spark/pull/16333
  
This is a mistake.  I point to the wrong repository.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org