[GitHub] storm issue #1724: STORM-2131: Add blob command to worker-launcher, make sto...

[knusbaum]

Github user knusbaum commented on the issue:

https://github.com/apache/storm/pull/1724
  
@kishorvpatil @revans2 @jerrypeng 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1724: STORM-2131: Add blob command to worker-launcher, make sto...

[kishorvpatil]

Github user kishorvpatil commented on the issue:

https://github.com/apache/storm/pull/1724
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1714: STORM-2125 Use Calcite's implementation of Rex Compiler

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1714
  
Added full of scalar function tests which Calcite provides. Calcite 1.9.0 
has some bugs so not all functions are working well, but it can be fixed from 
Calcite side.
(I left comments to test codes about bugs I found.)
Once we track and document the bugs to our doc. page, I think it would be 
OK.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Are storm metrics reported through JMX too?

[S G]

Hi,

We want to use Telegraf (
https://github.com/influxdata/telegraf/tree/master/plugins) for getting
storm's metrics.

But we do not want to add a HttpForwardingMetricsServer just to get the
metrics and send them to telegraf.

Other option is to use Jolokia (https://jolokia.org/) that can read JMX and
write into telegraf.

Does storm report all its metrics (including those of custom spouts/bolts)
into JMX?
Or spawning a HttpForwardingMetricsServer is the only option?

Thanks
SG

[GitHub] storm pull request #1725: STORM-2135 document covered features of SQL refere...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1725

STORM-2135 document covered features of SQL reference in Calcite

* NOTE: This should be merged after STORM-2125 is merged to master

* add a new page "storm-sql-reference" based on Calcite SQL reference

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2135

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1725


commit 355594f7b5423c1085698ca63de9a6523b265ec1
Author: Jungtaek Lim 
Date:   2016-10-07T00:18:36Z

STORM-2135 document covered features of SQL reference in Calcite

* add a new page "storm-sql-reference" based on Calcite SQL reference
* NOTE: This should be merged after STORM-2125 is merged to master




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1714: STORM-2125 Use Calcite's implementation of Rex Compiler

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1714
  
Addressed documenting covered feature from other PR: #1725


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Plan for merge SQE and Storm SQL

[Jungtaek Lim]

For preventing this discussion to be staled, I'd like to put a note how
Storm SQL is going now.
(including the change after my tech. analysis)

1. There're 6 pull requests opened regarding Storm SQL.
https://github.com/apache/storm/pulls/HeartSaVioR

2. When STORM-2125  is merged to
master, Storm SQL can handle most of things which Calcite publishes to SQL
reference page. I wrote Storm SQL's own reference page

and submitted it to another pull request (one of six pull requests)

STORM-2125 is having +1 but waiting for Calcite 1.10.0 to be released in
order to apply bugfix. (Yes, someone can state that it's a bit tightly
coupled.) I'm expecting they can cut 1.10.0 RC1 in next week. If it doesn't
happen in next week, we can just stick with Calcite 1.9.0 and immediately
upgrade 1.10.0 when it's available.

3. During adding tests I found some of bugs on Calcite side (left notes for
each test and also reference doc), which we can contribute back to Calcite
community. I think this would be a positive feedback loop between Storm and
Calcite project.

There're still many rooms available to work on Storm SQL.

1. I haven't had much time to do now, but would like to learn Calcite and
try to optimize Storm SQL via STORM-1446
. If someone who
understands Calcite well takes over and contributes this area I'd be much
appreciated.

2. I think Trident seems not good backend API for Storm SQL for a long
term.
Trident doesn't support typed operation, join and aggregation is limited to
micro-batch so result is non deterministic. I'm waiting for higher-level
API (STORM-1961  is the
start) and plan to move the backend API.

3. Storm SQL needs to have more connectors as data sources. It isn't hard
to work on, but a bit time-consuming.

4. Storm SQL needs to have time to stabilize by experiment loop:
experimenting from early adopters, reporting, and fixing bugs.

5. Long term work: we can expand supporting features on Storm SQL. Join
between streaming data source and normal table should help to enrich or
filter data with other external data source. There's also continuous effort
from Calcite community: 'Streaming SQL' led by Julian. etc.

Looking forward to continue discussion with JW Player folks.

Thanks,
Jungtaek Lim (HeartSaVioR)

ps.
I'd really like to make revamped Storm SQL available to the community.
Do we think merge process can pause this effort? I hope not, but I can
follow the community's decision.

2016년 9월 30일 (금) 오전 3:58, P. Taylor Goetz 님이 작성:

FYI, I’ve merged the SQE code and documentation into the sqe_import branch:

https://github.com/apache/storm/tree/sqe_import

Note the build will fail on the last component (storm-sqe) due to the
compilation issues mentioned earlier.

-Taylor


On Sep 29, 2016, at 11:13 AM, Bobby Evans 
wrote:

Agreed, or if we can find a way to not break compatibility (like perhaps
having a common base class for most of the logic and one subclass that uses
a string while another that uses the byte array).   - Bobby

   On Thursday, September 29, 2016 10:05 AM, Jungtaek Lim 
wrote:


The change on storm-redis seems to require breaking backward compatibility,
so I would love to see another pull request to integrate it via general
review process.
If it can be integrated to SQE without changing storm-redis, that would be
nice.

Does it make sense?

- Jungtaek Lim (HeartSaVioR)

2016년 9월 29일 (목) 오전 6:08, P. Taylor Goetz 님이 작성:

The changes are available in GitHub, I just overlooked it. And they’re
actually neatly contained in two commits:

The changes to storm-kafka can be found here:


https://github.com/jwplayer/storm/commit/2069c76695a225e4bb8f402c89e572836104755a


The changes to storm-redis are here:


https://github.com/jwplayer/storm/commit/30d000d3ff673efa8b927d23e554a705fb2928b8
<
https://github.com/jwplayer/storm/commit/2069c76695a225e4bb8f402c89e572836104755a
>


The changes to storm-kafka are straightforward and implemented in such a
way that they would be useful for use cases outside of SQE. As the commit
message states, it adds a new kafka deserialization scheme (FullScheme)
that includes the key, value, topic, partition and offset when reading from
kafka, which is a feature I can see as being valuable for some use cases. I
would be +1 for merging that code.

The changes to storm-redis are a little different, as Morrigan pointed
out, because it only addresses the Trident API, but IMHO it looks like a
good direction.

@HeartSavior — Would you have some time to take a look at the storm-redis
changes and provide your opinion, since you’re one of the original authors
of that code?

-Taylor


On Sep 26, 2016, at 6:28 PM, Jungtaek Lim  wrote:

Great!

For storm-redis, we might need to modify key/value mapper to use byte[]
rather than St