[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user salex89 commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-75203789 OK, sorry for necroposting then. Should I open an issue on JIRA with the snippet? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-75357849 Yeah, please do it and describe it in detail so we can easily reproduce it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-75181614 Hi @salex89 , seems your exception is a little different to this PR, maybe I need a careful look at this exception, thanks for your reporting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user salex89 commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-74517113 I'm not sure if this is the exactly same issue, but I am experience this with Spark 1.2.1 and Scala 2.10.4 also. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50852462 OK, this is very confusing, sequence of two results is very confusing. Let me run the tests again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50852470 Jenkins, test this again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50870159 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50870349 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17670/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50870356 A bunch of tests have been failing spuriously with java.net.BindException: Address already in use. It's not the PR. I wonder what recent change could have made this happen? Is something being stricter about assigning a fixed port? did a config change to let multiple tests run on the same virtual machine? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50870434 Also, a number of them have been failing for spurious python mllib issues. At least was the case yesterday. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50873840 QA results for PR 1508:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17670/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50874514 Heyyyaa, it finally passed. I am merging this. Thanks @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50730899 Again, this is an unrelated failure. Running it again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50730915 Jenkins, test this again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50786130 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50786599 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17584/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50793631 QA results for PR 1508:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17584/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50793921 Can you add these to the Mima excludes. ProblemFilters.exclude[IncompatibleMethTypeProblem](org.apache.spark.streaming.kafka.KafkaUtils.createStream) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50841708 QA tests have started for PR 1508. This patch DID NOT merge cleanly! brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17627/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50841873 @jerryshao Oops, not sure how that is possible given what this patch touches. Can you merge with master nonetheless. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50842067 Ok, I will update the code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50843421 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17629/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50845649 QA results for PR 1508:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17629/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50846353 QA results for PR 1508:br- This patch PASSES unit tests.brbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17627/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50678591 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50678921 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17484/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50679139 LGTM. I am okay with the binary compatibility change, since it is really wrong that we were using manifest instead of classtags. But I am not sure I really understand what is the reason behind the failure of the `val keyDecoder = manifest[U].runtimeClass.getConstructor(classOf[VerifiableProperties])` Why could the manifest not get the runtimeClass correctly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50679779 Oh, please add the appropriate exclude in the Mima exclusions https://github.com/apache/spark/blob/master/project/MimaExcludes.scala See the error in the jenkins console output for the exact exception string that you need to add. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50684839 QA results for PR 1508:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17484/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50685458 This failure is unrelated to this patch. Its a current known issue in Jenkins, will run the test again when this error gets cleared. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50702692 I get it now. Let me try to run the Jenkins once again, to figure out what needs to be added to Mima --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50702782 Jenkins, test this again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50703601 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-50703951 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17533/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-49581911 QA results for PR 1508:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16900/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-49590837 It's the MIMA test that fails, since the method signature is changed. It's possible to keep and deprecate the existing method of course. Should we just do that, or OK to remove the method on the grounds that the API doesn't quite work? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-49687085 Hi @srowen , thanks for your advice. I've also noticed that seems MIMA cannot allow me to change the method signature. But I think it is meaningful and will not change the user's code, only need to recompile. @tdas, would you mind giving me some suggestions, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/1508 [SPARK-2103][Streaming] Change to ClassTag for KafkaInputDStream and fix reflection issue This PR updates previous Manifest for KafkaInputDStream's Decoder to ClassTag, also fix the problem addressed in [SPARK-2103](https://issues.apache.org/jira/browse/SPARK-2103). Previous Java interface cannot actually get the type of Decoder, so when using this Manifest to reconstruct the decode object will meet reflection exception. Also for other two Java interfaces, ClassTag[String] is useless because calling Scala API will get the right implicit ClassTag. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-2103 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1508.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1508 commit 58f2a1ab4281290c3f214a672fa6d58d33925748 Author: jerryshao saisai.s...@intel.com Date: 2014-07-21T06:15:41Z Change Manifest to ClassTag for KafkaInputDStream's Decoder and fix Decoder construct issue when using Java API --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-49575634 QA tests have started for PR 1508. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16900/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2103][Streaming] Change to ClassTag for...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1508#issuecomment-49577894 Nice one Jerry! this actually enables using Kafka with non-String data in Java. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
