[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22649 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97028/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22649 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22649 **[Test build #97028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97028/testReport)** for PR 22649 at commit [`5e0f6fc`](https://github.com/apache/spark/commit/5e0f6fc14cd468ae1d06ab40e53189fb292375c0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22641: [SPARK-25611][SPARK-25612][SQL][TESTS] Improve test run ...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22641 @mgaido91 Thanks for your input. I took another look at the testcase. Let me outline some of my understandings first. - The test validates the precedence rules in determining the resultant compression to be used in the presence of SessionLevel codecs and Table level codecs. - It verifies the correct compression is picked by reading the metadata information from parquet/orc file metadata. - The accepted configuration for parquet are : none, uncompressed, snappy, gzip, lzo, brotli, lz4, zstd - The accepted configuration for orc are : none, uncompressed, snappy, zlib, lzo - The testcase in question use only a SUBSET of allowable codecs for parquet : uncompressed, snappy, gzip - The test case in question use only a SUBSET of allowable codecs for orc : None, Snappy, Zlib One thing to note is that, the codecs being tested are not exhaustive and we pick a subset (perhaps the most popular ones). Other thing is that, we have a 3 way loop 1) isPartitioned 2) convertMetastore 3) useCTAS on top of the codec loop. So we will be calling the codec loop 6 times in a test for each unique combination of (isPartitioned, convertMetastore, useCTAS). And we have changed the codec loop to randomly pick one combination of table level and session level codecs. Given this, i feel we are getting a decent coverage and also i feel we should be able to catch regression as we will catch it in some jenkin run or the other. If you still feel uncomfortable, should we take 2 codecs as opposed to 1 ? It will generate a 24 (4 * 6) times loop as opposed to 54 (9 * 6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22060 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97023/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22060 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22060 **[Test build #97023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97023/testReport)** for PR 22060 at commit [`7fc1d11`](https://github.com/apache/spark/commit/7fc1d11388babe169cf45ce2376d898d89f299b7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22614: [SPARK-25561][SQL] HiveClient.getPartitionsByFilt...
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/22614#discussion_r223172392
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -746,34 +746,20 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
getAllPartitionsMethod.invoke(hive,
table).asInstanceOf[JSet[Partition]]
} else {
logDebug(s"Hive metastore filter is '$filter'.")
-val tryDirectSqlConfVar =
HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL
-// We should get this config value from the metaStore. otherwise
hit SPARK-18681.
-// To be compatible with hive-0.12 and hive-0.13, In the future we
can achieve this by:
-// val tryDirectSql =
hive.getMetaConf(tryDirectSqlConfVar.varname).toBoolean
-val tryDirectSql =
hive.getMSC.getConfigValue(tryDirectSqlConfVar.varname,
- tryDirectSqlConfVar.defaultBoolVal.toString).toBoolean
try {
// Hive may throw an exception when calling this method in some
circumstances, such as
- // when filtering on a non-string partition column when the hive
config key
- // hive.metastore.try.direct.sql is false
+ // when filtering on a non-string partition column.
getPartitionsByFilterMethod.invoke(hive, table, filter)
.asInstanceOf[JArrayList[Partition]]
} catch {
- case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] &&
- !tryDirectSql =>
+ case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] =>
--- End diff --
@gatorsmile : Sorry for late reply. We had seen issues with this in past
and resorted to do exponential backoff with retries. Fetching all the
partitions is going to be bad in a prod setting even if it makes it
through, the underlying problem if left un-noticed is bad for the system health.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22637 **[Test build #97031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97031/testReport)** for PR 22637 at commit [`db061b8`](https://github.com/apache/spark/commit/db061b855b0efa35f7b4ea5943d5396c2181bf83). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public abstract class RowBasedKeyValueBatch extends MemoryConsumer implements Closeable ` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22637 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97031/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22637 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22637 **[Test build #97031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97031/testReport)** for PR 22637 at commit [`db061b8`](https://github.com/apache/spark/commit/db061b855b0efa35f7b4ea5943d5396c2181bf83). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22648: [MINOR] Clean up the joinCriteria in SQL parser
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22648 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22637: [SPARK-25408] Move to more ideomatic Java8
Github user Fokko commented on the issue: https://github.com/apache/spark/pull/22637 Thanks @dongjoon-hyun. I've fixed the indentation issues. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22500: [SPARK-25488][TEST] Refactor MiscBenchmark to use main m...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22500 Hi, @wangyum . - Could you review and merge https://github.com/wangyum/spark/pull/15 ? - Could you add `[SQL]` to the PR title? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22047: [SPARK-19851] Add support for EVERY and ANY (SOME...
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22047#discussion_r223171963 --- Diff: python/pyspark/sql/functions.py --- @@ -403,6 +403,28 @@ def countDistinct(col, *cols): return Column(jc) +def every(col): --- End diff -- @gatorsmile OK. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI support h...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22650 Hi @srowen , Kindly review and merge. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI support h...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22650 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI support h...
Github user shahidki31 commented on the issue: https://github.com/apache/spark/pull/22650 Hi @srowen , Kindly review and merge. This PR will be dependent on the PR https://github.com/apache/spark/pull/22645 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI support h...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22650 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI support h...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22650 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22650: [SPARK-25575][FOLLOWUP]SQL tab in the spark UI su...
GitHub user shahidki31 opened a pull request:
https://github.com/apache/spark/pull/22650
[SPARK-25575][FOLLOWUP]SQL tab in the spark UI support hide tables
## What changes were proposed in this pull request?
After the PR, https://github.com/apache/spark/pull/22592, SQL tab supports
collapsing table.
However, after refreshing the page, it doesn't store it previous stage.
This was due to a typo in the argument list in the
collapseTablePageLoadCommand() function.
## How was this patch tested?
bin/spark-shell
```
sql("create table a (id int)")
for(i <- 1 to 100) sql(s"insert into a values ($i)")
```
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shahidki31/spark SPARK-25575-followUp
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22650.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22650
commit cd9ef14c4060d38a26dd31555b53a6bf9820fe17
Author: Shahid
Date: 2018-10-06T04:30:54Z
SPARK-25566
[Spark Job History] SQL UI Page does not support Pagination
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21732 **[Test build #97030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97030/testReport)** for PR 21732 at commit [`0f029b0`](https://github.com/apache/spark/commit/0f029b0a28700334dc6334f1ad89b3124f235a51). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3732/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21732 **[Test build #97029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97029/testReport)** for PR 21732 at commit [`23be39a`](https://github.com/apache/spark/commit/23be39a5414fe0f569a4ebd19fa65a91b3fbc808). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3731/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22638: [SPARK-25610][SQL][TEST] Improve execution time of Datas...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/22638 Thanks a lot @gatorsmile @mgaido91 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22634: [SPARK-25646][k8s] Fix docker-image-tool.sh on de...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22634 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22645: [SPARK-25566][SPARK-25567][WEBUI][SQL]Support pagination...
Github user shahidki31 commented on the issue:
https://github.com/apache/spark/pull/22645
Test step to reproduce OOM without the PR.
1) bin/spark-shell --conf spark.sql.ui.retainedExecutions=5
for (i <- 0 until 5) {
val df = Seq(
(1, 1),
(2, 2)
).toDF()
df.collect()
Without the PR:
After fix:
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22634: [SPARK-25646][k8s] Fix docker-image-tool.sh on dev build...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22634 Thank you. Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22649 **[Test build #97028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97028/testReport)** for PR 22649 at commit [`5e0f6fc`](https://github.com/apache/spark/commit/5e0f6fc14cd468ae1d06ab40e53189fb292375c0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3730/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22649 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3729/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22649 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build ...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22649 cc @zsxwing . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22649: [SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/22649
[SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build error due to
foreachBatch
## What changes were proposed in this pull request?
This PR fixes the Scala-2.12 build error due to ambiguity in `foreachBatch`
test cases.
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/428/console
```scala
[error]
/home/jenkins/workspace/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ForeachBatchSinkSuite.scala:102:
ambiguous reference to overloaded definition,
[error] both method foreachBatch in class DataStreamWriter of type
(function:
org.apache.spark.api.java.function.VoidFunction2[org.apache.spark.sql.Dataset[Int],Long])org.apache.spark.sql.streaming.DataStreamWriter[Int]
[error] and method foreachBatch in class DataStreamWriter of type
(function: (org.apache.spark.sql.Dataset[Int], Long) =>
Unit)org.apache.spark.sql.streaming.DataStreamWriter[Int]
[error] match argument types ((org.apache.spark.sql.Dataset[Int], Any) =>
Unit)
[error] ds.writeStream.foreachBatch((_, _) =>
{}).trigger(Trigger.Continuous("1 second")).start()
[error] ^
[error]
/home/jenkins/workspace/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/sources/ForeachBatchSinkSuite.scala:106:
ambiguous reference to overloaded definition,
[error] both method foreachBatch in class DataStreamWriter of type
(function:
org.apache.spark.api.java.function.VoidFunction2[org.apache.spark.sql.Dataset[Int],Long])org.apache.spark.sql.streaming.DataStreamWriter[Int]
[error] and method foreachBatch in class DataStreamWriter of type
(function: (org.apache.spark.sql.Dataset[Int], Long) =>
Unit)org.apache.spark.sql.streaming.DataStreamWriter[Int]
[error] match argument types ((org.apache.spark.sql.Dataset[Int], Any) =>
Unit)
[error] ds.writeStream.foreachBatch((_, _) =>
{}).partitionBy("value").start()
[error] ^
```
## How was this patch tested?
Manual.
Since this failure occurs in Scala-2.12 profile and test cases, Jenkins
will not test this. We need to build with Scala-2.12 and run the tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-SCALA212
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22649.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22649
commit 5e0f6fc14cd468ae1d06ab40e53189fb292375c0
Author: Dongjoon Hyun
Date: 2018-10-06T04:06:23Z
[SPARK-25644][SS][FOLLOWUP][BUILD] Fix Scala 2.12 build error due to
foreachBatch
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21732 **[Test build #97027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97027/testReport)** for PR 21732 at commit [`80e11d2`](https://github.com/apache/spark/commit/80e11d289d7775863cb9c28b2c1d4364292048a4). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22633: [SPARK-25644][SS]Fix java foreachBatch in DataStreamWrit...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22633 It turns out that we didn't check Scala 2.12 build. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7-ubuntu-scala-2.12/428/console I'll make a follow-up. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97021/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22623: [SPARK-25636][CORE] spark-submit cuts off the failure re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22623 **[Test build #97021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97021/testReport)** for PR 22623 at commit [`a82e75f`](https://github.com/apache/spark/commit/a82e75fb4019cf7c0e5ca8279a40e1ac8dbbf53e). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22648: [MINOR] Clean up the joinCriteria in SQL parser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22648 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97020/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22648: [MINOR] Clean up the joinCriteria in SQL parser
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22648 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22648: [MINOR] Clean up the joinCriteria in SQL parser
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22648 **[Test build #97020 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97020/testReport)** for PR 22648 at commit [`09b70cb`](https://github.com/apache/spark/commit/09b70cb421e330061e9a9b597f30f4b3a58f0d52). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22614: [SPARK-25561][SQL] HiveClient.getPartitionsByFilter shou...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22614 The PR description and title may need to change accordingly. Can you update it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22379 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97017/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22636: [SPARK-25629][TEST] Reduce ParquetFilterSuite: filter pu...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22636 Yea, it is not obvious and only few seconds - might not be so worth. But looks improvement because it fixes the test cases to test what the previous PR targeted. Wouldn't it be better just to go ahead rather then close this since the PR is already open? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22379: [SPARK-25393][SQL] Adding new function from_csv()
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22379 **[Test build #97017 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97017/testReport)** for PR 22379 at commit [`b318239`](https://github.com/apache/spark/commit/b318239f96c8b589ed493ec83e85ea40672647fd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22375: [SPARK-25388][Test][SQL] Detect incorrect nullabl...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22375#discussion_r223169695
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvalHelper.scala
---
@@ -221,6 +227,12 @@ trait ExpressionEvalHelper extends
GeneratorDrivenPropertyChecks with PlanTestBa
val unsafeRow = evaluateWithUnsafeProjection(expression, inputRow)
val input = if (inputRow == EmptyRow) "" else s", input: $inputRow"
+val dataType = expression.dataType
+if (!checkResult(unsafeRow.get(0, dataType), expected, dataType,
expression.nullable)) {
--- End diff --
We check different properties in these two `if` statements.
1. Line 231 checks consistency between value and `nullable` in `expected`
1. Line 245 checks bit-wise value between `expected` and `expression`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22375: [SPARK-25388][Test][SQL] Detect incorrect nullabl...
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22375#discussion_r223169637
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CodeGenerationSuite.scala
---
@@ -113,7 +113,7 @@ class CodeGenerationSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(actual.length == 1)
val expected = UTF8String.fromString("abc")
-if (!checkResult(actual.head, expected, expressions.head.dataType)) {
+if (!checkResult(actual.head, expected, expressions.head.dataType,
expressions.head.nullable)) {
--- End diff --
That is another option that I thought. On the other hand, to set default
has a risk to overlook a possible incosistency between value and `nullable` at
top level of `expected`.
Do we use the default value at the all of callers of `checkResult`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22646: [SPARK-25654][SQL] Support for nested JavaBean arrays, l...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/22646 The `createDataFrame` API for Java Beans doesn't have clear document about what JavaBeans are supportd. Can you also update it to explicitly document this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22646: [SPARK-25654][SQL] Support for nested JavaBean ar...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22646#discussion_r223169392
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1115,8 +1123,31 @@ object SQLContext {
})
}
}
-def createConverter(cls: Class[_], dataType: DataType): Any => Any =
dataType match {
- case struct: StructType => createStructConverter(cls,
struct.map(_.dataType))
+def createConverter(t: Type, dataType: DataType): Any => Any = (t,
dataType) match {
+ case (cls: Class[_], struct: StructType) =>
+createStructConverter(cls, struct.map(_.dataType))
+ case (arrayType: Class[_], array: ArrayType) =>
+val converter = createConverter(arrayType.getComponentType,
array.elementType)
+value => new GenericArrayData(
+ (0 until JavaArray.getLength(value)).map(i =>
+converter(JavaArray.get(value, i))).toArray)
+ case (_, array: ArrayType) =>
--- End diff --
Can you add few comments explaining why having two cases both for
`ArrayType`?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97015/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22612 **[Test build #97015 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97015/testReport)** for PR 22612 at commit [`a9f924c`](https://github.com/apache/spark/commit/a9f924c5943d6ed45e38a1c5aadd07045adbe138). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97018/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22612 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22612: [SPARK-24958] Add executors' process tree total memory i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22612 **[Test build #97018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97018/testReport)** for PR 22612 at commit [`a11e3a2`](https://github.com/apache/spark/commit/a11e3a267b78cf5a7e42190893f36e24e2aad2d4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97026/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #97026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97026/testReport)** for PR 20761 at commit [`f360e61`](https://github.com/apache/spark/commit/f360e61ad653107b8bbf1db4c055fab4b7eefdd2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22646: [SPARK-25654][SQL] Support for nested JavaBean ar...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22646#discussion_r223168936
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1098,12 +1099,19 @@ object SQLContext {
data: Iterator[_],
beanClass: Class[_],
attrs: Seq[AttributeReference]): Iterator[InternalRow] = {
+import scala.collection.JavaConverters._
+import java.lang.reflect.{Type, ParameterizedType, Array => JavaArray}
+def interfaceParameters(t: Type, interface: Class[_]): Array[Type] = t
match {
+ case parType: ParameterizedType if parType.getRawType == interface =>
+parType.getActualTypeArguments
+ case _ => throw new UnsupportedOperationException(s"$t is not an
$interface")
--- End diff --
This exception message looks a bit confusing. We can say the type is not
supported.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22646: [SPARK-25654][SQL] Support for nested JavaBean ar...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22646#discussion_r223168881
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -1098,12 +1099,19 @@ object SQLContext {
data: Iterator[_],
beanClass: Class[_],
attrs: Seq[AttributeReference]): Iterator[InternalRow] = {
+import scala.collection.JavaConverters._
+import java.lang.reflect.{Type, ParameterizedType, Array => JavaArray}
--- End diff --
Why add import here? Can we move it to top?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12922: [SPARK-15145][ML]:port binary classification evaluator t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12922 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12922: [SPARK-15145][ML]:port binary classification evaluator t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/12922 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97022/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12922: [SPARK-15145][ML]:port binary classification evaluator t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12922 **[Test build #97022 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97022/consoleFull)** for PR 12922 at commit [`3f91492`](https://github.com/apache/spark/commit/3f91492c6a46554313c6494bb1f31e21d2db4592). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #97026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97026/testReport)** for PR 20761 at commit [`f360e61`](https://github.com/apache/spark/commit/f360e61ad653107b8bbf1db4c055fab4b7eefdd2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22614: [SPARK-25561][SQL] HiveClient.getPartitionsByFilter shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22614: [SPARK-25561][SQL] HiveClient.getPartitionsByFilter shou...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22614 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97014/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20761 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97025/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #97025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97025/testReport)** for PR 20761 at commit [`707eb18`](https://github.com/apache/spark/commit/707eb18e1325974d0e95b5634793e539673628ad). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22614: [SPARK-25561][SQL] HiveClient.getPartitionsByFilter shou...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22614 **[Test build #97014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97014/testReport)** for PR 22614 at commit [`f42bbec`](https://github.com/apache/spark/commit/f42bbec8d7ba23cca77f2bf83230ad2e2ceafeb9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-25642] : Adding two new metrics to record the num...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22498 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-25642] : Adding two new metrics to record the num...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97013/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22498: [SPARK-25642] : Adding two new metrics to record the num...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22498 **[Test build #97013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97013/testReport)** for PR 22498 at commit [`70472a2`](https://github.com/apache/spark/commit/70472a255e5da3ea4522959e26f5c403641e1ce6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20761: [SPARK-20327][CORE][YARN] Add CLI support for YARN custo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20761 **[Test build #97025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97025/testReport)** for PR 20761 at commit [`707eb18`](https://github.com/apache/spark/commit/707eb18e1325974d0e95b5634793e539673628ad). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22647 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22647 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3728/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22647 **[Test build #97024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97024/testReport)** for PR 22647 at commit [`adb63e4`](https://github.com/apache/spark/commit/adb63e4b0a04a8bb2c1f3054646fb2c9bdac49f1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22647 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22060 **[Test build #97023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97023/testReport)** for PR 22060 at commit [`7fc1d11`](https://github.com/apache/spark/commit/7fc1d11388babe169cf45ce2376d898d89f299b7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22060 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3727/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22060: [DO NOT MERGE][TEST ONLY] Add once-policy rule check
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22060 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22632: [SPARK-25606][TEST] Reduce DateExpressionsSuite test tim...
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/22632
@gatorsmile I have some confusion. Is this
https://github.com/apache/spark/blob/58c55cb4a6d72d72df908e37aa63f617b3cc5587/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala#L118-L122
should be
```scala
(0 to 24).foreach { h =>
c.add(Calendar.HOUR_OF_DAY, h)
checkEvaluation(Quarter(Literal(new Date(c.getTimeInMillis))),
c.get(Calendar.MONTH) / 3 + 1)
}
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22647 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97011/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22647 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22647: [SPARK-25655] [BUILD] Add -Pspark-ganglia-lgpl to the sc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22647 **[Test build #97011 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97011/testReport)** for PR 22647 at commit [`adb63e4`](https://github.com/apache/spark/commit/adb63e4b0a04a8bb2c1f3054646fb2c9bdac49f1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22636: [SPARK-25629][TEST] Reduce ParquetFilterSuite: filter pu...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/22636 The time reduction is not obvious. Let us keep this unchanged? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22295: [SPARK-25255][PYTHON]Add getActiveSession to Spar...
Github user huaxingao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22295#discussion_r223165392
--- Diff: python/pyspark/sql/session.py ---
@@ -252,6 +255,20 @@ def newSession(self):
"""
return self.__class__(self._sc, self._jsparkSession.newSession())
+@classmethod
+@since(2.5)
+def getActiveSession(cls):
+"""
+Returns the active SparkSession for the current thread, returned
by the builder.
+>>> s = SparkSession.getActiveSession()
+>>> l = [('Alice', 1)]
+>>> rdd = s.sparkContext.parallelize(l)
+>>> df = s.createDataFrame(rdd, ['name', 'age'])
+>>> df.select("age").collect()
+[Row(age=1)]
+"""
+return cls._activeSession
--- End diff --
@HyukjinKwon I am not sure if I follow your suggestion correctly. Does the
following look right to you?
session.py
```
@classmethod
@since(3.0)
def getActiveSession(cls):
from pyspark.sql import functions
return functions.getActiveSession()
```
functions.py
```
@since(3.0)
def getActiveSession():
from pyspark.sql import SparkSession
sc = SparkContext._active_spark_context
if sc is None:
sc = SparkContext()
if sc._jvm.SparkSession.getActiveSession().isDefined():
SparkSession(sc, sc._jvm.SparkSession.getActiveSession().get())
return SparkSession._activeSession
else:
return None
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22638: [SPARK-25610][SQL][TEST] Improve execution time o...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22638 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22047: [SPARK-19851] Add support for EVERY and ANY (SOME...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22047#discussion_r223164281 --- Diff: python/pyspark/sql/functions.py --- @@ -403,6 +403,28 @@ def countDistinct(col, *cols): return Column(jc) +def every(col): --- End diff -- Please keep the SQL functions and remove the function APIs. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20832: [SPARK-20536][SQL] Extend ColumnName to create StructFie...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20832 @efimpoberezkin Could you please close this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20832: [SPARK-20536][SQL] Extend ColumnName to create St...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20832#discussion_r223164144
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Column.scala ---
@@ -1208,85 +1208,172 @@ class ColumnName(name: String) extends
Column(name) {
*/
def boolean: StructField = StructField(name, BooleanType)
+ /**
+ * Creates a new `StructField` of type boolean.
+ * @since 2.4.0
+ */
+ def boolean(nullable: Boolean): StructField = StructField(name,
BooleanType, nullable)
--- End diff --
The NULL hints are not enforced. Thus, it is kind of risky to expose this
to end users since it could generate a wrong result. We plan to ignore the
user-specified NULL hints in the upcoming release.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #12922: [SPARK-15145][ML]:port binary classification evaluator t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/12922 **[Test build #97022 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97022/consoleFull)** for PR 12922 at commit [`3f91492`](https://github.com/apache/spark/commit/3f91492c6a46554313c6494bb1f31e21d2db4592). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22622: [SPARK-25635][SQL][BUILD] Support selective direct encod...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22622 Thank you, @gatorsmile, @HyukjinKwon , @viirya , @dilipbiswal ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
