[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Ack I missed something here: there's an override of kafka.version for Scala 2.12, from when it had to be bumped _up_ to work with 2.12. That no longer works when compiling with 2.12. I'll submit a follow up. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 @wangyum, can you please file a Kafka JIRA with details of what the test is doing (even if the failure is transient)? From the stacktrace, it looks like a potential broker issue (assuming there are no real disk issues where these tests were executed). If there is indeed a new issue (we have to verify since the test seems to be transient), it would likely only affect tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Hm, looking at the dashboard, I don't see this failure consistently in the master tests: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ Let's keep a close eye on it to see if it repeats. It might have been something transient. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 I used the following command and the test passed: mvn test -Phadoop-2.6 -Pyarn -Phive -Dtest=KafkaMicroBatchSourceSuite -rf external/kafka-0-10-sql Please take a look at the 'Disk error' message and see if it was related to test failure. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/21488 It seems this commit cause `KafkaSourceStressForDontFailOnDataLossSuite` failed: ```java ... [info] KafkaSourceStressSuite: [info] - stress test with multiple topics and partitions (19 seconds, 255 milliseconds) [info] KafkaSourceStressForDontFailOnDataLossSuite: [info] - stress test for failOnDataLoss=false *** FAILED *** (45 seconds, 648 milliseconds) [info] java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.KafkaStorageException: Disk error when trying to access log file on the disk. [info] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) [info] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:77) [info] at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) [info] at org.apache.spark.sql.kafka010.KafkaTestUtils$$anonfun$2.apply(KafkaTestUtils.scala:254) [info] at org.apache.spark.sql.kafka010.KafkaTestUtils$$anonfun$2.apply(KafkaTestUtils.scala:248) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) [info] at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) [info] at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) [info] at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) [info] at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) [info] at org.apache.spark.sql.kafka010.KafkaTestUtils.sendMessages(KafkaTestUtils.scala:248) [info] at org.apache.spark.sql.kafka010.KafkaTestUtils.sendMessages(KafkaTestUtils.scala:238) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite$$anonfun$48$$anonfun$apply$26$$anonfun$apply$27.apply(KafkaMicroBatchSourceSuite.scala:1268) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite$$anonfun$48$$anonfun$apply$26$$anonfun$apply$27.apply(KafkaMicroBatchSourceSuite.scala:1267) [info] at scala.collection.immutable.Range.foreach(Range.scala:160) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite$$anonfun$48$$anonfun$apply$26.apply(KafkaMicroBatchSourceSuite.scala:1267) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite$$anonfun$48$$anonfun$apply$26.apply(KafkaMicroBatchSourceSuite.scala:1265) [info] at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) [info] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite$$anonfun$48.apply(KafkaMicroBatchSourceSuite.scala:1265) [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) [info] at org.scalatest.Transformer.apply(Transformer.scala:22) [info] at org.scalatest.Transformer.apply(Transformer.scala:20) [info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186) [info] at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:103) [info] at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183) [info] at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196) [info] at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196) [info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289) [info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(KafkaMicroBatchSourceSuite.scala:1155) [info] at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221) [info] at org.apache.spark.sql.kafka010.KafkaSourceStressForDontFailOnDataLossSuite.runTest(KafkaMicroBatchSourceSuite.scala:1155) [info] at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229) [info] at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229) [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396) [info] at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384) [info] at scala.collection.immutable.List.foreach(List.scala:392) [info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384) [info] at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 Thanks! Merging to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes no objections. There's no concrete problem, and there is upside to making the change. I think you're aware of the potential issues, and have thought through them in the context of deeper knowledge of Kafka's APIs than I have. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @srowen May I read your comment as "no objections"? The current PR looks good to me. If you don't have objections, I will go ahead and merge it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes, good argument. Having been burned in the past (actually, by Kafka and ZK changes, though that's in the past), I'm aware that even compiling against version B instead of A, with no library code changes, may still mean the library no longer runs against version A. Changing exception signatures is also sometimes a problem. It may not be the case here, unlikely, but do want to think this through fully. Yes differences like https://github.com/apache/kafka/commit/a4c292173618e22b3f99ab8615c6b585c7688a40 are what I am worried about. Still I have no concrete example of this type of problem in the wild. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Anyway, overall I think you should definitely make this change. Spark users are currently penalised heavily when running on clusters with the message format introduced in 0.11.0, which has important resiliency improvements (KIP-101). And, as @zsxwing said, people can choose an older version if necessary (I can't think of a reason why, the reason why have focused on making client jars compatible is so that people can just use the latest independently of broker versions). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @ijuma I see. I was looking at 0.10 jar. Thanks for correcting me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 @zsxwing `org.apache.kafka.clients.admin.AdminClient` is in the clients jar. It's not relevant for this PR, I am just mentioning it so if Spark decides to migrate to that eventually. I see code using `ZkUtils`, which is an internal class that will be removed eventually. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Yeah, the Java client libraries have been evolved in a compatible manner for the most part since 0.10.0. The set of broker versions supported by 0.10.0 and 2.0.0 is exactly the same. The consumer/producer API has been enriched (transactions, idempotent producer, offsetsForTimes), but the existing methods have been kept. A small number of deprecated, but rarely used methods have been removed (not in KafkaProducer or KafkaConsumer though): https://github.com/apache/kafka/commit/a4c292173618e22b3f99ab8615c6b585c7688a40 In 0.10.1, a heartbeat thread was added to the Java consumer: https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread This is helpful to users who could not call `poll` often enough, but the default configs should be fine for most cases. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @srowen just to be clear, `AdminClient` is not in `kafka-clients` jar. The user has to add `kafka` jar as a dependency to use `AdminClient`. In addition, even if this upgrade has some unexpected issues, the user can still switch back to an old `kafka-clients` version by pining the version in their pom.xml. This Kafka connector in Spark uses only 0.10 APIs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 Yes, and that then impacts programs that depend on this module to use Kafka 0.10+ from Spark. They'd have to update `kafka-clients` too, or would be automatically updated really. I don't see a reason a caller would use AdminClient directly in a Spark program though. Any other API changes you can see possibly causing a problem? my concern is breaking compatibility in a minor release. Yes Spark requires Java 8. You're saying although the (two) major version bumps look scary there was little change in the client library itself? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 @srowen I meant that the library is compatible, so if you just change the version in the pom, it's fine. If you changed the code to use some methods in AdminClient, then you'd have to be more careful. Does that make sense? The major version bump is because we now require Java 8 (which I believe you require too) and the Scala clients were removed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 @ijuma this changes the version of `kakfa-clients` in compile scope, too -- right? looks like it in the POM. That's not just a test change. It's probably OK if the client is backwards-compatible with 0.10.x brokers, which is what this integration currently supports. I guess I'm always nervous about making a major-version change to a user-visible dependency; this is a big one too. But you all may know better whether there is any potential user impact. Would this change break any user code that was previously compiling against the 0.10.x client? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 Yes, Kafka clients 0.10.2 and higher support brokers from 0.10.0 and higher if the protocols being used are available in that version. Since we only changed test code in this PR, upgrading to 2.0.0 is safe. If Spark were to use the AdminClient create topics, then the brokers would have to be 0.10.1 or higher. Other admin protocols like config management were added in more recent versions and so on. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 > I might have missed this in the shuffle here -- is this fully compatible with 0.10.x brokers too? Yep. @ijuma could you confirm it? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21488 I might have missed this in the shuffle here -- is this fully compatible with 0.10.x brokers too? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 @zsxwing Is there anything I should do for this PR ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21488 https://github.com/apache/kafka/releases KAFKA 2.0.0 release is publicly available. We can finish it before the code freeze? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93745/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93745/testReport)** for PR 21488 at commit [`aa69915`](https://github.com/apache/spark/commit/aa69915165d9aaca2bcb5d22fb2fc9467bf16826). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93745/testReport)** for PR 21488 at commit [`aa69915`](https://github.com/apache/spark/commit/aa69915165d9aaca2bcb5d22fb2fc9467bf16826). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1457/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 ``` 22:36:05.028 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 16314.0 (TID 39181, localhost, executor driver): java.io.FileNotFoundException: File file:/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-0bbc239c-37c5-4df2-b86d-e9c7628ceb28/f1=1/f2=1/part-0-390ac6da-50dc-4d32-ba08-462da1e8a0c2.c000.snappy.parquet does not exist It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved. at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:131) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:182) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:109) ``` Doesn't seem to be related to PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93736/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93736 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93736/testReport)** for PR 21488 at commit [`aa69915`](https://github.com/apache/spark/commit/aa69915165d9aaca2bcb5d22fb2fc9467bf16826). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1451/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93736 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93736/testReport)** for PR 21488 at commit [`aa69915`](https://github.com/apache/spark/commit/aa69915165d9aaca2bcb5d22fb2fc9467bf16826). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Ryan: Thanks for the close follow-up. Once Kafka 2.0.0 is released, I will incorporate the above. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @tedyu fyi, here is the last fix for 2.0.0: https://github.com/zsxwing/spark/commit/7ccc520b540106ca4c2dac2e42834984332d6804 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/21488 @tedyu I just realized topic deletion might be used by some users. This Kafka version will introduce a regression for them. Since 2.0 fixes this issue and is going out soon, how about just wait for 2.0? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 @zsxwing Is there anything that needs to be done from my side ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93374/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)** for PR 21488 at commit [`1738642`](https://github.com/apache/spark/commit/17386429150d26d838f6895ec9698b7176765ffc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Test failure was in Hive test, not related to this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93372/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93372 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)** for PR 21488 at commit [`241878c`](https://github.com/apache/spark/commit/241878c886f206dabc44fd5d55d3fe6908a35a3b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user guozhangwang commented on the issue: https://github.com/apache/spark/pull/21488 > 1.1.1 has been released, maybe we can upgrade to that. +1 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1194/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93374/testReport)** for PR 21488 at commit [`1738642`](https://github.com/apache/spark/commit/17386429150d26d838f6895ec9698b7176765ffc). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Thanks for the reminder, @ijuma Updated pom.xml and title accordingly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user ijuma commented on the issue: https://github.com/apache/spark/pull/21488 1.1.1 has been released, maybe we can upgrade to that. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21488 **[Test build #93372 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93372/testReport)** for PR 21488 at commit [`241878c`](https://github.com/apache/spark/commit/241878c886f206dabc44fd5d55d3fe6908a35a3b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1192/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21488 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21488: [SPARK-18057][SS] Update Kafka client version from 0.10....
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/21488 Ryan: Thanks for the reminder. I have disabled that test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
