[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094057#comment-16094057 ] ASF GitHub Bot commented on BAHIR-100: -- Github user lresende commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r128406934 --- Diff: streaming-mqtt/README.md --- @@ -52,12 +52,14 @@ this actor can be configured to handle failures, etc. val lines = MQTTUtils.createStream(ssc, brokerUrl, topic) val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic) +val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic) --- End diff -- Fixed the typo. Validated there was no other StreamStream > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Fix For: Spark-2.2.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094009#comment-16094009 ] ASF GitHub Bot commented on BAHIR-100: -- Github user ckadner commented on a diff in the pull request: https://github.com/apache/bahir/pull/47#discussion_r128401599 --- Diff: streaming-mqtt/README.md --- @@ -52,12 +52,14 @@ this actor can be configured to handle failures, etc. val lines = MQTTUtils.createStream(ssc, brokerUrl, topic) val lines = MQTTUtils.createPairedStream(ssc, brokerUrl, topic) +val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic) --- End diff -- @davidrosenstark -- I assume the `StreamStream` word duplication is a copy-paste error? ~`val lines = MQTTUtils.createPairedByteArrayStreamStream(ssc, brokerUrl, topic)`~ `val lines = MQTTUtils.createPairedByteArrayStream(ssc, brokerUrl, topic)` > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Fix For: Spark-2.2.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luciano Resende resolved BAHIR-100. --- Resolution: Fixed Fix Version/s: Spark-2.2.0 > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Fix For: Spark-2.2.0 > > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093975#comment-16093975 ] ASF GitHub Bot commented on BAHIR-100: -- Github user asfgit closed the pull request at: https://github.com/apache/bahir/pull/47 > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-100) Providing MQTT Spark Streaming to return encoded Byte[] message without corruption
[ https://issues.apache.org/jira/browse/BAHIR-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093494#comment-16093494 ] ASF GitHub Bot commented on BAHIR-100: -- Github user lresende commented on the issue: https://github.com/apache/bahir/pull/47 LGTM, merging if there are no more comments. > Providing MQTT Spark Streaming to return encoded Byte[] message without > corruption > -- > > Key: BAHIR-100 > URL: https://issues.apache.org/jira/browse/BAHIR-100 > Project: Bahir > Issue Type: New Feature > Components: Spark Streaming Connectors >Reporter: Anntinu Josy >Assignee: Anntinu Josy > Labels: mqtt, spark, streaming > Original Estimate: 168h > Remaining Estimate: 168h > > Now a days Network bandwidth is becoming a serious resource that need to be > conserver in IoT ecosystem, For this puropse we are using different byte[] > based encoding such as Protocol Buffer and flat Buffer, Once this encoded > message is converted into string the data becomes corrupted, So same byte[] > format need to be preserved when forwarded. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BAHIR-110) Replace use of _all_docs API with _changes API in all receivers
[ https://issues.apache.org/jira/browse/BAHIR-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093136#comment-16093136 ] ASF GitHub Bot commented on BAHIR-110: -- Github user emlaver commented on the issue: https://github.com/apache/bahir/pull/45 Stability tests update: I did three successful loads into Spark yesterday with a 15GB (time: 8 1/2 mins), 20GB (17 mins), and 46GB (25 mins) database. @mayya-sharipova @yanglei99 @ricellis > Replace use of _all_docs API with _changes API in all receivers > --- > > Key: BAHIR-110 > URL: https://issues.apache.org/jira/browse/BAHIR-110 > Project: Bahir > Issue Type: Improvement >Reporter: Esteban Laver > Original Estimate: 216h > Remaining Estimate: 216h > > Today we use the _changes API for Spark streaming receiver and _all_docs API > for non-streaming receiver. _all_docs API supports parallel reads (using > offset and range) but performance of _changes API is still better in most > cases (even with single threaded support). > With this ticket we want to: > a) re-implement all receivers using _changes API > b) compare performance between the two implementations based on _changes and > _all_docs > Based on the results in b) we could decide to either > - replace _all_docs implementation with _changes based implementation OR > - allow customers to pick one (with a solid documentation about pros and > cons) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
