[GitHub] spark issue #21632: [SPARK-19591][ML][MLlib] Add sample weights to decision ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21632 **[Test build #94165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94165/testReport)** for PR 21632 at commit [`6326bdf`](https://github.com/apache/spark/commit/6326bdf8856a93e193b320c6579009d75db9409a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client version...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21955 I see that `zookeeper.connect` is set to `[zkhost]:[zkport]` after ZK is started, so looks like it's trying to do so. That much hasn't changed at least. I can increase the timeout; although I don't know why it takes a while, it could explain why it appears flaky. It's also consistent with KAFKA-7193 although I think we're on ZK 3.4.5 or so in Spark? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21909 **[Test build #94179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94179/testReport)** for PR 21909 at commit [`6248c01`](https://github.com/apache/spark/commit/6248c015955930a2f2abe38e407e1190c0e297e0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21909 jenkins, retest this, please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94158/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21992 **[Test build #4233 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4233/testReport)** for PR 21992 at commit [`d15ab4b`](https://github.com/apache/spark/commit/d15ab4b4833487d1c3f87f4461fafc2f9a13cd49). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1777/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21898 **[Test build #94158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94158/testReport)** for PR 21898 at commit [`16ee90e`](https://github.com/apache/spark/commit/16ee90e5bbecbb94b54d61703e7d7d17a58b3bcb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21403 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21403: [SPARK-24341][SQL] Support only IN subqueries with the s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21403 **[Test build #94178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94178/testReport)** for PR 21403 at commit [`a6114a6`](https://github.com/apache/spark/commit/a6114a655305f318230bf1bbd25394e952793a94). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user dongido001 commented on the issue: https://github.com/apache/spark/pull/21992 I can only spot this for now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client version...
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21955 Looks similar to https://issues.apache.org/jira/browse/KAFKA-7193, are we using the same ZK ip in the server and client? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21994: [SPARK-24529][Build][test-maven][follow-up] Add spotbugs...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21994 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21994: [SPARK-24529][Build][test-maven][follow-up] Add spotbugs...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21994 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1776/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21991: [SPARK-25018] [Infra] Use `Co-Authored-By` git trailer i...
Github user dbtsai commented on the issue: https://github.com/apache/spark/pull/21991 There are couple other common used git trailers used in linux community, https://git.wiki.kernel.org/index.php/CommitMessageConventions We may consider to add `Signed-off-by:` and `Reviewed-by:` which can be helpful to track the code development. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21994: [SPARK-24529][Build][test-maven][follow-up] Add spotbugs...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21994 **[Test build #94177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94177/testReport)** for PR 21994 at commit [`3501209`](https://github.com/apache/spark/commit/3501209badca1c4d38e4ca7d0ddc45d2977e34e0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21994: [SPARK-24529][Build][test-maven][follow-up] Add s...
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/21994 [SPARK-24529][Build][test-maven][follow-up] Add spotbugs 3.1.6 into maven build process ## What changes were proposed in this pull request? This PR tries to add spotbugs after [upgrading mvn](https://issues.apache.org/jira/browse/SPARK-24956). This is because we met [a problem](https://issues.apache.org/jira/browse/SPARK-24895) with old mvn. ## How was this patch tested? existing UTs You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-24529-follow Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21994.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21994 commit 3501209badca1c4d38e4ca7d0ddc45d2977e34e0 Author: Kazuaki Ishizaki Date: 2018-08-03T21:24:21Z add spotbugs 3.1.6 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21909 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94154/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21909 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21909 **[Test build #94154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94154/testReport)** for PR 21909 at commit [`05c8dbb`](https://github.com/apache/spark/commit/05c8dbb3f15e5a210758b66e0c6d47a519a01b65). * This patch **fails from timeout after a configured wait of \`300m\`**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #94176 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94176/testReport)** for PR 20838 at commit [`accc7d0`](https://github.com/apache/spark/commit/accc7d065abcf6cbe6493f280290946edaaeb83e). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20838 Ok only failure is in Kafka, very likely unrelated. Jenkins retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user dongido001 commented on the issue: https://github.com/apache/spark/pull/21992 Sure thing @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client version...
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/21955 I don't see an obvious issue. Looks like zookeeper.connection.timeout.ms isn't being set, so it's defaulting to 6 seconds... could try tweaking it upwards. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1775/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21898 **[Test build #94175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94175/testReport)** for PR 21898 at commit [`16ee90e`](https://github.com/apache/spark/commit/16ee90e5bbecbb94b54d61703e7d7d17a58b3bcb). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user mengxr commented on the issue: https://github.com/apache/spark/pull/21898 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94159/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94159/testReport)** for PR 21911 at commit [`c950b6e`](https://github.com/apache/spark/commit/c950b6e5e5f943155b296542238c08a412a35472). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94155/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21909 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21909: [SPARK-24959][SQL] Speed up count() for JSON and CSV
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21909 **[Test build #94155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94155/testReport)** for PR 21909 at commit [`6248c01`](https://github.com/apache/spark/commit/6248c015955930a2f2abe38e407e1190c0e297e0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client ...
Github user koeninger commented on a diff in the pull request:
https://github.com/apache/spark/pull/21955#discussion_r207664852
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
---
@@ -109,7 +109,7 @@ private[kafka010] class KafkaTestUtils extends Logging {
brokerConf = new KafkaConfig(brokerConfiguration, doLog = false)
server = new KafkaServer(brokerConf)
server.startup()
- brokerPort = server.boundPort()
+ brokerPort = server.boundPort(brokerConf.interBrokerListenerName)
--- End diff --
Isn't the test hanging on the line right before that change though?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94157/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #94157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94157/testReport)** for PR 20838 at commit [`accc7d0`](https://github.com/apache/spark/commit/accc7d065abcf6cbe6493f280290946edaaeb83e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21993: [SPARK-24983][Catalyst] Add configuration for maximum nu...
Github user dvogelbacher commented on the issue: https://github.com/apache/spark/pull/21993 @HyukjinKwon can you help with finding reviewers for this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21955#discussion_r207663151
--- Diff:
external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
---
@@ -109,7 +109,7 @@ private[kafka010] class KafkaTestUtils extends Logging {
brokerConf = new KafkaConfig(brokerConfiguration, doLog = false)
server = new KafkaServer(brokerConf)
server.startup()
- brokerPort = server.boundPort()
+ brokerPort = server.boundPort(brokerConf.interBrokerListenerName)
--- End diff --
In particular @zsxwing , regarding the failure below, I wonder if my change
on this line really wasn't quite equivalent. Is it possible it needs to be
"PLAINTEXT" like in yours?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21955: [SPARK-18057][FOLLOW-UP][SS] Update Kafka client version...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21955 Hm, I might need your eyes on this again @koeninger @zsxwing or even @ijuma . I see this failure in the test, which looks like it's no longer happy starting ZK in some cases. Does that ring any bells? not sure it would. ``` [info] org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite *** ABORTED *** (8 seconds, 741 milliseconds) [info] kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING [info] at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply$mcV$sp(ZooKeeperClient.scala:230) [info] at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226) [info] at kafka.zookeeper.ZooKeeperClient$$anonfun$kafka$zookeeper$ZooKeeperClient$$waitUntilConnected$1.apply(ZooKeeperClient.scala:226) [info] at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251) [info] at kafka.zookeeper.ZooKeeperClient.kafka$zookeeper$ZooKeeperClient$$waitUntilConnected(ZooKeeperClient.scala:226) [info] at kafka.zookeeper.ZooKeeperClient.(ZooKeeperClient.scala:95) [info] at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1580) [info] at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:348) [info] at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:372) [info] at kafka.server.KafkaServer.startup(KafkaServer.scala:202) [info] at org.apache.spark.streaming.kafka010.KafkaTestUtils$$anonfun$setupEmbeddedKafkaServer$2.apply(KafkaTestUtils.scala:111) ``` See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94150/ for instance. However I note that this test has also passed as well, at least in one of the Maven-based test jobs on Jenkins. Could be a little build-specific but nevertheless a problem. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21993: [SPARK-24983][Catalyst] Add configuration for maximum nu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21993 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21978 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94163/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21992 That's fine. Can you look for other typos while you're at it? we try to swat several at once. Even a grep for "to [word] to" could be a good use of time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21993: [SPARK-24983][Catalyst] Add configuration for maximum nu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21993 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21919: [SPARK-24933][SS] Report numOutputRows in SinkPro...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/21919#discussion_r207662087
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/streaming/StreamWriterCommitProgress.java
---
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.sources.v2.writer.streaming;
+
+import org.apache.spark.annotation.InterfaceStability;
+import org.apache.spark.sql.sources.v2.writer.WriterCommitMessage;
+
+import java.io.Serializable;
+
+/**
+ * Sink progress information collected from {@link WriterCommitMessage}.
+ */
+@InterfaceStability.Evolving
+public interface StreamWriterCommitProgress extends Serializable {
--- End diff --
Why this is a public API? This is just created and consumed inside Spark.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21993: [SPARK-24983][Catalyst] Add configuration for maximum nu...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21993 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21978: SPARK-25006: Add CatalogTableIdentifier.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21978 **[Test build #94163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94163/testReport)** for PR 21978 at commit [`6fe2d07`](https://github.com/apache/spark/commit/6fe2d07de57feff8903f9c91eb0841eaa4646505). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class UnresolvedRelation(table: CatalogTableIdentifier) extends LeafNode ` * `sealed trait IdentifierWithOptionalDatabaseAndCatalog ` * `case class CatalogTableIdentifier(table: String, database: Option[String], catalog: Option[String])` * `class TableIdentifier(name: String, db: Option[String])` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21993: [SPARK-24983][SQL] Add configuration for maximum ...
GitHub user dvogelbacher opened a pull request: https://github.com/apache/spark/pull/21993 [SPARK-24983][SQL] Add configuration for maximum number of leaf expressions in collapsed project nodes ## What changes were proposed in this pull request? Add a configuration option for the maximum number of leaf expressions in collapsed project nodes. If a collapsed project node (result of the `CollapseProject` optimizer rule) would have more leaf expressions than the configured maximum number we don't collapse. This is to protect against an exponentially exploding number of leaf expressions when collapsing many (binary) expression that refer to the same columns (see https://issues.apache.org/jira/browse/SPARK-24983). ## How was this patch tested? Add a new unit test. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dvogelbacher/spark dv/limitProjectCollapse Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21993.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21993 commit 096340b2d9b65d32cd033f4f9c1d2668336d1fbf Author: David Vogelbacher Date: 2018-08-03T19:40:25Z implement config setting limiting number of leaf expressions after collapsing commit b7ced5433a55d19e333806c84a3bdcb19e15 Author: David Vogelbacher Date: 2018-08-03T20:26:11Z refactor a bit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21987: [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21987 **[Test build #4232 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4232/testReport)** for PR 21987 at commit [`654b918`](https://github.com/apache/spark/commit/654b918fc536de67c88e1bc4d618d27aa4fb76f1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 > Are there any other blockers to enabling this by default now that @mallman fixed the currently known broken queries? The functionality exercised by the ignored test in `ParquetSchemaPruningSuite.scala` is still broken. That's something we're hoping to fix in a follow on PR. This PR has to be merged first. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21102 **[Test build #94174 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94174/testReport)** for PR 21102 at commit [`ce1bfb0`](https://github.com/apache/spark/commit/ce1bfb04e774b3c18b31a33c13ce1a0cc0632419). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1774/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21102: [SPARK-23913][SQL] Add array_intersect function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21102 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user ajacques commented on the issue: https://github.com/apache/spark/pull/21889 Are there any other blockers to enabling this by default now that @mallman fixed the currently known broken queries? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21992 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21992 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21992: Little typo
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21992 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21992: Little typo
GitHub user dongido001 opened a pull request: https://github.com/apache/spark/pull/21992 Little typo ## What changes were proposed in this pull request? Fixed little typo for a comment ## How was this patch tested? Manual test Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dongido001/spark patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21992.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21992 commit df8c492db7b4ce2d91cde5f9e2a3ef15adff2d41 Author: Onwuka Gideon Date: 2018-08-03T20:16:05Z Little typo --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21911: [SPARK-24940][SQL] Coalesce and Repartition Hint for SQL...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21911 **[Test build #94173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94173/testReport)** for PR 21911 at commit [`acda5e2`](https://github.com/apache/spark/commit/acda5e281f1e3e8fd87faf4c8582d39ea857d215). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21987: [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21987 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...
Github user skambha commented on the issue: https://github.com/apache/spark/pull/17185 The last push has changes that adds the map lookup for the 3 part name. It implements a solution to address the lookup.. although I think there are more ways we can go about it and could possibly improve on it in the future. @cloud-fan, please take a look. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21987: [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21987 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94150/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21987: [SPARK-25015][BUILD] Update Hadoop 2.7 to 2.7.7
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21987 **[Test build #94150 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94150/testReport)** for PR 21987 at commit [`654b918`](https://github.com/apache/spark/commit/654b918fc536de67c88e1bc4d618d27aa4fb76f1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...
Github user skambha commented on the issue: https://github.com/apache/spark/pull/17185 I rebased and ran the catalyst unit test suite only as a sanity test. fwiw, I had run the sql and hive and catalyst suites earlier prior to this last rebase. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21986 **[Test build #94170 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94170/testReport)** for PR 21986 at commit [`3f88e2a`](https://github.com/apache/spark/commit/3f88e2a927c22f4fc509b8ca96027ef381f7fe84). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17185: [SPARK-19602][SQL] Support column resolution of fully qu...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17185 **[Test build #94172 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94172/testReport)** for PR 17185 at commit [`90cd6d3`](https://github.com/apache/spark/commit/90cd6d33f59fdae16a3a386ed14cefb3f28d35a8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21895 **[Test build #94171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94171/testReport)** for PR 21895 at commit [`14ae790`](https://github.com/apache/spark/commit/14ae790350b88c9ed64d63bd67cf21a22f8ffdd9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1773/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1772/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21986 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user mallman commented on the issue: https://github.com/apache/spark/pull/21889 Success! Now where do we stand, @gatorsmile @HyukjinKwon ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21986: [SPARK-23937][SQL] Add map_filter SQL function
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21986 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/21895 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21982 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21982 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1771/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21982 **[Test build #94169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94169/testReport)** for PR 21982 at commit [`26bf379`](https://github.com/apache/spark/commit/26bf37960a1534da8e2119181dce5794e3b48172). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21982: [SPARK-23911][SQL] Add aggregate function.
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21982 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21919 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21919 **[Test build #94168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94168/testReport)** for PR 21919 at commit [`d3a00d4`](https://github.com/apache/spark/commit/d3a00d432db35d2401dacec65110ad75cfe03349). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DataWritingSparkTaskResult(numRows: Long, writerCommitMessage: WriterCommitMessage)` * `trait StreamWriterProgressCollector ` * `class MicroBatchWriter(batchId: Long, writer: StreamWriter) extends DataSourceWriter` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21919 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94168/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21889 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94142/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21889 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21889: [SPARK-4502][SQL] Parquet nested column pruning - founda...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21889 **[Test build #94142 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94142/testReport)** for PR 21889 at commit [`da50429`](https://github.com/apache/spark/commit/da50429112c4ce7f387f3fec586edcfba0b51151). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress
Github user vackosar commented on the issue: https://github.com/apache/spark/pull/21919 @jose-torres I removed use of commit to report the row count. Would you have a look? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21919: [SPARK-24933][SS] Report numOutputRows in SinkProgress v...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21919 **[Test build #94168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94168/testReport)** for PR 21919 at commit [`d3a00d4`](https://github.com/apache/spark/commit/d3a00d432db35d2401dacec65110ad75cfe03349). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21721 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94147/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94139/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21898 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21721: [SPARK-24748][SS] Support for reporting custom metrics v...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21721 **[Test build #94147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94147/testReport)** for PR 21721 at commit [`1775c2a`](https://github.com/apache/spark/commit/1775c2a1db2bf790ddf1cad0113c7ead2409ba65). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21898: [SPARK-24817][Core] Implement BarrierTaskContext.barrier...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21898 **[Test build #94139 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94139/testReport)** for PR 21898 at commit [`33a8926`](https://github.com/apache/spark/commit/33a89269906e94780e1b992e9400c9d8f98698f5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21895: [SPARK-24948][SHS] Delegate check access permissi...
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/21895#discussion_r207641341 --- Diff: core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala --- @@ -779,6 +808,8 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock) listing.delete(classOf[LogInfo], log.logPath) } } +// Clean the blacklist from the expired entries. +clearBlacklist(CLEAN_INTERVAL_S) --- End diff -- I misread it as MAX_LOG_AGE_S ... CLEAN_INTERVAL_S should be fine here, you are right. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21953: [SPARK-24992][Core] spark should randomize yarn local di...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21953 test this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17185: [SPARK-19602][SQL] Support column resolution of f...
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207640501
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -262,17 +262,47 @@ abstract class Star extends LeafExpression with
NamedExpression {
*/
case class UnresolvedStar(target: Option[Seq[String]]) extends Star with
Unevaluable {
- override def expand(input: LogicalPlan, resolver: Resolver):
Seq[NamedExpression] = {
+ /**
+ * Returns true if the nameParts match the qualifier of the attribute
+ *
+ * There are two checks: i) Check if the nameParts match the qualifier
fully.
+ * E.g. SELECT db.t1.* FROM db1.t1 In this case, the nameParts is
Seq("db1", "t1") and
+ * qualifier of the attribute is Seq("db1","t1")
+ * ii) If (i) is not true, then check if nameParts is only a single
element and it
+ * matches the table portion of the qualifier
+ *
+ * E.g. SELECT t1.* FROM db1.t1 In this case nameParts is Seq("t1") and
+ * qualifier is Seq("db1","t1")
+ * SELECT a.* FROM db1.t1 AS a
+ * In this case nameParts is Seq("a") and qualifier for
+ * attribute is Seq("a")
+ */
+ private def matchedQualifier(
+ attribute: Attribute,
+ nameParts: Seq[String],
+ resolver: Resolver): Boolean = {
+val qualifierList = attribute.qualifier.getOrElse(Seq.empty)
+
+// match the qualifiers and nameParts
+val matched = nameParts.corresponds(qualifierList)(resolver) match {
--- End diff --
done. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21608: [SPARK-24626] [SQL] Improve location size calculation in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21608 **[Test build #94167 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94167/testReport)** for PR 21608 at commit [`253af70`](https://github.com/apache/spark/commit/253af70cd4cd525db4951db67fce01a5ca1f0014). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21895: [SPARK-24948][SHS] Delegate check access permissions to ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21895 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94140/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
