[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094057#comment-16094057 ] ASF GitHub Bot commented on BAHIR-100: -- Github user lresende commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r128406934 --- Diff: streaming-mqtt/README.md --- @@ -52,12 +52,14 @@ this actor can be configured to handle failures, etc. val lines = MQTTUtils.createStream(ssc, brokerUrl, topic) val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic) +val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic) --- End diff -- Fixed the typo. Validated there was no other StreamStream > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Fix For: Spark-2.2.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094009#comment-16094009 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ckadner commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r128401599 --- Diff: streaming-mqtt/README.md --- @@ -52,12 +52,14 @@ this actor can be configured to handle failures, etc. val lines = MQTTUtils.createStream(ssc, brokerUrl, topic) val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic) +val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic) --- End diff -- @davidrosenstark -- I assume the `StreamStream` word duplication is a copy-paste error? ~`val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic)`~ `val lines = MQTTUtils.createPairedByteArrayStream(ssc, brokerUrl, topic)` > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Fix For: Spark-2.2.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093975#comment-16093975 ] ASF GitHub Bot commented on BAHIR-100: -- Github user asfgit closed the pull request at: https://github.com/apache/bahir/pull/47 > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093494#comment-16093494 ] ASF GitHub Bot commented on BAHIR-100: -- Github user lresende commented on the issue: https://github.com/apache/bahir/pull/47 LGTM, merging if there are no more comments. > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081749#comment-16081749 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/47 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/72/ > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081738#comment-16081738 ] ASF GitHub Bot commented on BAHIR-100: -- Github user davidrosenstark commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r126603788 --- Diff: streaming-mqtt/src/main/scala/org/apache/spark/streaming/mqtt/MQTTPairedByteArrayInputDStream.scala --- @@ -0,0 +1,143 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.streaming.mqtt + +import org.eclipse.paho.client.mqttv3._ +import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence + +import org.apache.spark.storage.StorageLevel +import org.apache.spark.streaming.StreamingContext +import org.apache.spark.streaming.dstream._ +import org.apache.spark.streaming.receiver.Receiver + +/** + * Input stream that subscribe messages from a Mqtt Broker. + * Uses eclipse paho as MqttClient http://www.eclipse.org/paho/ + * @param brokerUrl Url of remote mqtt publisher --- End diff -- this was missing in all three classes. adding to all 3 > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16081553#comment-16081553 ] ASF GitHub Bot commented on BAHIR-100: -- Github user tedyu commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r126583567 --- Diff: streaming-mqtt/README.md --- @@ -52,12 +52,14 @@ this actor can be configured to handle failures, etc. val lines = MQTTUtils.createStream(ssc, brokerUrl, topic) val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic) +val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic) --- End diff -- Where is createPairedByteArrayStreamStream defined ? > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077686#comment-16077686 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/47 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/71/ > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077685#comment-16077685 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/47 Build successful > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077124#comment-16077124 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ApacheBahir commented on the issue: https://github.com/apache/bahir/pull/47 Refer to this link for build results (access rights to CI server needed): http://169.45.79.58:8080/job/bahir_spark_pr_builder/69/ > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009477#comment-16009477 ] Anntinu Josy commented on BAHIR-100: Sorry for the delay, was busy with some personal things. I will create the PR in a week > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004023#comment-16004023 ] Christian Kadner commented on BAHIR-100: [~anntinutj] were you able to make any progress on this? > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947787#comment-15947787 ] Christian Kadner commented on BAHIR-100: [~anntinutj] I was not aware of the issue you describe, feel free to create a PR, but please elaborate on a concrete example scenario that produces the corrupt result you described and make sure to add a test case that fails before your change and succeeds after your change. Thank you! > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
