Re: [VOTE] Apache Bahir 2.2.2 (RC1)
+1 Ran unit test suite which passed. On Thu, Nov 8, 2018 at 1:34 PM Luciano Resende wrote: > Dear community member, > > Please vote to approve the release of Apache Bahir 2.2.2 (RC1) based on > Apache Spark 2.2.2. > > Tag: v2.2.2-rc1 (821a8c67c21f4f4ab4a7caa8e2f85a2c396683d4) > > https://github.com/apache/bahir/tree/v2.2.2-rc1 > > Release files: > > https://repository.apache.org/content/repositories/orgapachebahir-1024 > > Source distribution: > > https://dist.apache.org/repos/dist/dev/bahir/bahir-spark/2.2.2-rc1/ > > > The vote is open for at least 72 hours and passes if a majority of at least > 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Bahir 2.2.2 > [ ] -1 Do not release this package because ... > > > Thanks for your vote! > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
[jira] [Commented] (BAHIR-166) Migrate akka sql streaming source to datasrouce v2 API
[ https://issues.apache.org/jira/browse/BAHIR-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679268#comment-16679268 ] ASF subversion and git services commented on BAHIR-166: --- Commit be1effaaf7cfde28d19e032e038694e01fbf169b in bahir's branch refs/heads/master from shimamoto [ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=be1effa ] [BAHIR-166] Migrate akka sql streaming source to DataSource v2 API Migrate akka sql streaming source to DataSource v2 API. Closes #67 > Migrate akka sql streaming source to datasrouce v2 API > -- > > Key: BAHIR-166 > URL: https://issues.apache.org/jira/browse/BAHIR-166 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.3.0 >Reporter: Prashant Sharma >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-179) fail silently when tests need docker image to be running
[ https://issues.apache.org/jira/browse/BAHIR-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679533#comment-16679533 ] ASF GitHub Bot commented on BAHIR-179: -- Github user eskabetxe commented on the issue: https://github.com/apache/bahir-flink/pull/35 hi @lresende, the changes are because i change: - the tests with docker images, this affect the projects of flume and kudu - upgrade junit to version 5.3.1 - the test disabled, is not working on master, because its ignored because actual parent pom have this **/Test*.java **/*Test.java **/*TestCase.java **/*Suite.java and the test class is named ActiveMQConnectorITCase and its not working, if you launch the test on local you will see that fail.. > fail silently when tests need docker image to be running > > > Key: BAHIR-179 > URL: https://issues.apache.org/jira/browse/BAHIR-179 > Project: Bahir > Issue Type: Test > Components: Flink Streaming Connectors >Reporter: Joao Boto >Assignee: Joao Boto >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-164) Spark Streaming with MQTT fails with Spark 2.3.0
[ https://issues.apache.org/jira/browse/BAHIR-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679250#comment-16679250 ] ASF subversion and git services commented on BAHIR-164: --- Commit b3902bac67edc2134bcc2c755fadc5c60c8ae01c in bahir's branch refs/heads/master from [~prashant.x90] [ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=b3902ba ] [BAHIR-164][BAHIR-165] Port Mqtt sql source to datasource v2 API Migrating Mqtt spark structured streaming connector to DatasourceV2 API. Closes #65 > Spark Streaming with MQTT fails with Spark 2.3.0 > > > Key: BAHIR-164 > URL: https://issues.apache.org/jira/browse/BAHIR-164 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.2.0, Spark-2.3 >Reporter: Jussi-Pekka Partanen >Priority: Blocker > > Currently I’m not able to use latest Spark 2.3.0 release together with Bahir, > but getting the following error: > > _java.lang.AssertionError: assertion failed: DataFrame returned by getBatch > from org.apache.bahir.sql.streaming.mqtt.MQTTTextStreamSource@66847d45 did > not have isStreaming=true_ > > org/apache/bahir/sql/streaming/mqtt/MQTTStreamSource.scala line 164 seems to > be causing the issue, i.e. it’s returning a normal dataframe instead of > streaming dataframe. > > There seems to be discussion about the same topic, but on stream source for > Kafka: > https://lists.apache.org/thread.html/015f2fbe9fbeafb86214a969a2d883a4898621189a86f21a4c6fbb75@%3Cdev.spark.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-181) username and password should be available for pyspark when using mqtt streaming
[ https://issues.apache.org/jira/browse/BAHIR-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679305#comment-16679305 ] ASF GitHub Bot commented on BAHIR-181: -- Github user ZisZ commented on the issue: https://github.com/apache/bahir/pull/69 > Please let me know your jira username so I can assign BAHIR-181 to yourself. CarreyZhan > username and password should be available for pyspark when using mqtt > streaming > --- > > Key: BAHIR-181 > URL: https://issues.apache.org/jira/browse/BAHIR-181 > Project: Bahir > Issue Type: Improvement > Components: Spark Streaming Connectors >Affects Versions: Spark-2.3.0 > Environment: spark 2.3.1, python3.6 >Reporter: Carrey Zhan >Priority: Minor > > When using spark-streaming-mqtt with pyspark to access rabbitmq, I found > there are no username and password provied for python api; > These two params are important and necessary for rabbitmq especially when > using rabbitmq virtual hosts, so I added a group of functions here; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BAHIR-166) Migrate akka sql streaming source to datasrouce v2 API
[ https://issues.apache.org/jira/browse/BAHIR-166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved BAHIR-166. --- Resolution: Fixed Assignee: Takako Shimamoto Fix Version/s: Spark-2.3.0 > Migrate akka sql streaming source to datasrouce v2 API > -- > > Key: BAHIR-166 > URL: https://issues.apache.org/jira/browse/BAHIR-166 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.3.0 >Reporter: Prashant Sharma >Assignee: Takako Shimamoto >Priority: Major > Fix For: Spark-2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Release votes and direction moving forward...
I finally got a chance to work on catching up with Spark releases again. I have sent a vote for 2.1.3 RC1 and 2.2.2 RC1 which should make us current with releases before 2.3 (which introduces the Data Source 2 API) I have also created a branch 2.x-before-datasource-v2 that we can use to work on any future releases below 2.3 which will enable us to move forward with Data Source 2.0 work in the master branch. -- Luciano Resende http://twitter.com/lresende1975 http://lresende.blogspot.com/
[jira] [Commented] (BAHIR-181) username and password should be available for pyspark when using mqtt streaming
[ https://issues.apache.org/jira/browse/BAHIR-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679282#comment-16679282 ] ASF GitHub Bot commented on BAHIR-181: -- Github user asfgit closed the pull request at: https://github.com/apache/bahir/pull/69 > username and password should be available for pyspark when using mqtt > streaming > --- > > Key: BAHIR-181 > URL: https://issues.apache.org/jira/browse/BAHIR-181 > Project: Bahir > Issue Type: Improvement > Components: Spark Streaming Connectors >Affects Versions: Spark-2.3.0 > Environment: spark 2.3.1, python3.6 >Reporter: Carrey Zhan >Priority: Minor > > When using spark-streaming-mqtt with pyspark to access rabbitmq, I found > there are no username and password provied for python api; > These two params are important and necessary for rabbitmq especially when > using rabbitmq virtual hosts, so I added a group of functions here; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-166) Migrate akka sql streaming source to datasrouce v2 API
[ https://issues.apache.org/jira/browse/BAHIR-166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679269#comment-16679269 ] ASF GitHub Bot commented on BAHIR-166: -- Github user asfgit closed the pull request at: https://github.com/apache/bahir/pull/67 > Migrate akka sql streaming source to datasrouce v2 API > -- > > Key: BAHIR-166 > URL: https://issues.apache.org/jira/browse/BAHIR-166 > Project: Bahir > Issue Type: Sub-task > Components: Spark Structured Streaming Connectors >Affects Versions: Spark-2.3.0 >Reporter: Prashant Sharma >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BAHIR-165) The avro messages to streaming-mqtt gives negative value.
[ https://issues.apache.org/jira/browse/BAHIR-165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679251#comment-16679251 ] ASF subversion and git services commented on BAHIR-165: --- Commit b3902bac67edc2134bcc2c755fadc5c60c8ae01c in bahir's branch refs/heads/master from [~prashant.x90] [ https://git-wip-us.apache.org/repos/asf?p=bahir.git;h=b3902ba ] [BAHIR-164][BAHIR-165] Port Mqtt sql source to datasource v2 API Migrating Mqtt spark structured streaming connector to DatasourceV2 API. Closes #65 > The avro messages to streaming-mqtt gives negative value. > -- > > Key: BAHIR-165 > URL: https://issues.apache.org/jira/browse/BAHIR-165 > Project: Bahir > Issue Type: Bug > Components: Spark Streaming Connectors >Affects Versions: Spark-2.1.1 > Environment: OS : Linux > MQTT publisher : Java application using Eclipse Paho >Reporter: Mohammed Hassan Farish >Priority: Major > Labels: issue, newbie, parsing > Original Estimate: 48h > Remaining Estimate: 48h > > Sending an Avro message with fields id and name to streaming-mqtt. The > messages gets parsed but the value of id is returned as some random negative > number. The result is negative only when the id input is greater than 63. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BAHIR-181) username and password should be available for pyspark when using mqtt streaming
[ https://issues.apache.org/jira/browse/BAHIR-181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved BAHIR-181. --- Resolution: Fixed Assignee: Carrey Zhan Fix Version/s: Spark-2.3.0 > username and password should be available for pyspark when using mqtt > streaming > --- > > Key: BAHIR-181 > URL: https://issues.apache.org/jira/browse/BAHIR-181 > Project: Bahir > Issue Type: Improvement > Components: Spark Streaming Connectors >Affects Versions: Spark-2.3.0 > Environment: spark 2.3.1, python3.6 >Reporter: Carrey Zhan >Assignee: Carrey Zhan >Priority: Minor > Fix For: Spark-2.3.0 > > > When using spark-streaming-mqtt with pyspark to access rabbitmq, I found > there are no username and password provied for python api; > These two params are important and necessary for rabbitmq especially when > using rabbitmq virtual hosts, so I added a group of functions here; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
