[jira] [Comment Edited] (STORM-1846) Need a smoke test for the generated Python code

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288181#comment-15288181
 ] 

Jungtaek Lim edited comment on STORM-1846 at 5/18/16 5:37 AM:
--

I thought it's easy to handle since we just set the pythonpath to include 
storm-core/src/py and run "echo 'from storm import ttypes' | python", and see 
the return code.
(Maybe we want to set PYTHONPATH and running echo at once.)

But it has preconditions which in fact makes this not easy to handle without 
forcing developer to do something.

# thrift should be installed to not getting error from 'import storm.ttypes'.
## we could even create another python script which import thrift first, and 
skip actual import if it fails to import thrift. Test may always pass on 
developer machine, but can fail on Travis CI since we can modify .travis.yml to 
install 'thrift' before testing.
# path should include python (or python.exe in Windows).

Since we got rid of multilang tests so AFAIK there's no requirement to have 
python, ruby, nodejs. 
It will require having python again, which seems not good even though Linux 
distributions / OSX normally have python installed by default.


was (Author: kabhwan):
I thought it's easy to handle since we just set the pythonpath to include 
storm-core/src/py and run "echo 'from storm import ttypes' | python", and see 
the return code.
(Maybe we want to set PYTHONPATH and running echo at once.)

But it has preconditions which in fact makes this not easy to handle without 
forcing developer to do something.

# thrift should be installed to not getting error from 'import storm.ttypes'.
## we could even create another python script which import thrift first, and 
skip actual import if it fails to import thrift. Test may always pass on 
developer machine, but can fail on Travis CI since we can modify .travis.yml to 
install 'thrift' before testing.
# path should include python (or python.exe in Windows).

Since we got rid of multilang tests so AFAIK there's no requirement to have 
python, ruby, nodejs. 
It requires to have python again, which seems not good even though Linux 
distributions / OSX normally have python installed by default.

> Need a smoke test for the generated Python code
> ---
>
> Key: STORM-1846
> URL: https://issues.apache.org/jira/browse/STORM-1846
> Project: Apache Storm
>  Issue Type: Test
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>
> This was inspired by STORM-1842. For that issue, simply attempting to import 
> ttypes.py would've detected the error. I'm guessing there is no such test. It 
> would be good to add one in order to help avoid problems like that in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (STORM-1846) Need a smoke test for the generated Python code

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288181#comment-15288181
 ] 

Jungtaek Lim edited comment on STORM-1846 at 5/18/16 4:22 AM:
--

I thought it's easy to handle since we just set the pythonpath to include 
storm-core/src/py and run "echo 'from storm import ttypes' | python", and see 
the return code.
(Maybe we want to set PYTHONPATH and running echo at once.)

But it has preconditions which in fact makes this not easy to handle without 
forcing developer to do something.

# thrift should be installed to not getting error from 'import storm.ttypes'.
## we could even create another python script which import thrift first, and 
skip actual import if it fails to import thrift. Test may always pass on 
developer machine, but can fail on Travis CI since we can modify .travis.yml to 
install 'thrift' before testing.
# path should include python (or python.exe in Windows).

Since we got rid of multilang tests so AFAIK there's no requirement to have 
python, ruby, nodejs. 
It requires to have python again, which seems not good even though Linux 
distributions / OSX normally have python installed by default.


was (Author: kabhwan):
I thought it's easy to handle since we just set the pythonpath to include 
storm-core/src/py and run "echo 'from storm import ttypes' | python", and see 
the return code.
(Maybe we want to set PYTHONPATH and running echo at once.)

But it has preconditions which in fact makes this not easy to handle without 
forcing developer to do something.

# thrift should be installed to not getting error from 'import storm.ttypes'.
## we could even create another python script which import thrift first, and 
skip actual import if it fails to import thrift.
# path should include python (or python.exe in Windows).

Since we got rid of multilang tests so AFAIK there's no requirement to have 
python, ruby, nodejs. 
It requires to have python again, which seems not good even though Linux 
distributions / OSX normally have python installed by default.

> Need a smoke test for the generated Python code
> ---
>
> Key: STORM-1846
> URL: https://issues.apache.org/jira/browse/STORM-1846
> Project: Apache Storm
>  Issue Type: Test
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>
> This was inspired by STORM-1842. For that issue, simply attempting to import 
> ttypes.py would've detected the error. I'm guessing there is no such test. It 
> would be good to add one in order to help avoid problems like that in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1846) Need a smoke test for the generated Python code

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288181#comment-15288181
 ] 

Jungtaek Lim commented on STORM-1846:
-

I thought it's easy to handle since we just set the pythonpath to include 
storm-core/src/py and run "echo 'from storm import ttypes' | python", and see 
the return code.
(Maybe we want to set PYTHONPATH and running echo at once.)

But it has preconditions which in fact makes this not easy to handle without 
forcing developer to do something.

# thrift should be installed to not getting error from 'import storm.ttypes'.
## we could even create another python script which import thrift first, and 
skip actual import if it fails to import thrift.
# path should include python (or python.exe in Windows).

Since we got rid of multilang tests so AFAIK there's no requirement to have 
python, ruby, nodejs. 
It requires to have python again, which seems not good even though Linux 
distributions / OSX normally have python installed by default.

> Need a smoke test for the generated Python code
> ---
>
> Key: STORM-1846
> URL: https://issues.apache.org/jira/browse/STORM-1846
> Project: Apache Storm
>  Issue Type: Test
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>
> This was inspired by STORM-1842. For that issue, simply attempting to import 
> ttypes.py would've detected the error. I'm guessing there is no such test. It 
> would be good to add one in order to help avoid problems like that in the 
> future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Regarding types of DataPoint value for Metrics Consumer

[Jungtaek Lim]

Hi devs,

Since IMetric#getValueAndReset doesn't restrict return type, it gives us
flexibility but metrics consumer should parse the value without context
(having just some assumptions).

I've look into some open source metrics consumers, and many of them support
Number, Map, and one of them supports Nested Map.
For the case of Map its key is appended to metric key and value is
converted to 'double'. I think it would be enough, but I'm not sure we can
rely on all metrics consumers to handle properly, too.

I feel if we can recommend proper types of DataPoint value for storing
metrics to time-series DB via metrics consumer it would be great. It can be
used to protocol between IMetric users and metrics consumer developers.

What do you think?

Thanks,
Jungtaek Lim (HeartSaVioR)

ps. I'm not heavy user of time-series DBs (I researched some, but they
don't document type/size of value clearly) so if someone could give the
information of supporting type/size of value of time-series DBs it should
be great for me. Or we can just assume number as 'double' as above and go
forward.

[jira] [Commented] (STORM-1838) [storm-kafka-client] the resumed OffsetEntry goes different

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288104#comment-15288104
 ] 

ASF GitHub Bot commented on STORM-1838:
---

Github user flisky commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219908452
  
By the way, it seems that we didn't handle the situation `offset out of 
range` when I test the `UNCOMMITTED_LATEST` strategy.


> [storm-kafka-client] the resumed OffsetEntry goes different
> ---
>
> Key: STORM-1838
> URL: https://issues.apache.org/jira/browse/STORM-1838
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Jifeng Yin
> Fix For: 1.0.1
>
>
> There are no more new messages after consumer rejoins the group. And it turns 
> out the old OffsetEntry instance kept commitOffsetsForAckedTuples failing.
> the comment "leave the acked offsets as they were to resume where it left 
> off" doesn't work for me, but the ackedMsgs goes different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1838 update OffsetEntry to support EARLI...

[flisky]

Github user flisky commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219908452
  
By the way, it seems that we didn't handle the situation `offset out of 
range` when I test the `UNCOMMITTED_LATEST` strategy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1838) [storm-kafka-client] the resumed OffsetEntry goes different

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288100#comment-15288100
 ] 

ASF GitHub Bot commented on STORM-1838:
---

Github user flisky commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219907956
  
Sorry, I didn't say it clearly, but this patch still applies.

Ｗhen a consumer re-balance occurs and 
**FirstPollOffsetStrategy#EARLIEST/LATEST**,  the retained partition wouldn't 
use the committed offset to resume but use the -2 / -1 offset, as I observed, 
so this is reason why the comment (`leave the acked offsets as they were to 
resume where it left off`) doesn't apply.


> [storm-kafka-client] the resumed OffsetEntry goes different
> ---
>
> Key: STORM-1838
> URL: https://issues.apache.org/jira/browse/STORM-1838
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Jifeng Yin
> Fix For: 1.0.1
>
>
> There are no more new messages after consumer rejoins the group. And it turns 
> out the old OffsetEntry instance kept commitOffsetsForAckedTuples failing.
> the comment "leave the acked offsets as they were to resume where it left 
> off" doesn't work for me, but the ackedMsgs goes different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1838 update OffsetEntry to support EARLI...

[flisky]

Github user flisky commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219907956
  
Sorry, I didn't say it clearly, but this patch still applies.

ï¼·hen a consumer re-balance occurs and 
**FirstPollOffsetStrategy#EARLIEST/LATEST**,  the retained partition wouldn't 
use the committed offset to resume but use the -2 / -1 offset, as I observed, 
so this is reason why the comment (`leave the acked offsets as they were to 
resume where it left off`) doesn't apply.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1765) KafkaBolt with new producer api should be part of storm-kaka-client

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288094#comment-15288094
 ] 

Sriharsha Chintalapani commented on STORM-1765:
---

[~supermonk] You can use use existing storm-kafka KafkaBolt which uses new 
producer API . You should configure SSL and pass the right configs to enable 
SSL in kafka producer Follow the doc here 
http://kafka.apache.org/documentation.html#security_ssl . 

> KafkaBolt with new producer api should be part of storm-kaka-client
> ---
>
> Key: STORM-1765
> URL: https://issues.apache.org/jira/browse/STORM-1765
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Hugo Louro
>
> During the discussion of storm-kaka-client we agreed on following
> "We talked about this some more and it seems to me it would make more sense 
> to leave this new spout in storm-kafka-client (or whatever you want to call 
> it) and move the KafkaBolt which uses the new producer api over here also. 
> That way this component only needs to depend on the new kafka-clients java 
> api and not on the entire scala kafka core. We can make the old storm-kafka 
> depend on this component so it still picks up the bolt so if anyone is using 
> that its still works. We can deprecate the old KafkaSpout but keep it around 
> for people using older versions of Kafka - tgravescs"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1772) Create topologies for measuring performance

[Manu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288091#comment-15288091
 ] 

Manu Zhang commented on STORM-1772:
---

Thanks, very good suggestions. Maybe not a proper place to ask, but do you have 
plans to move the yahoo streaming benchmarks forward ? It has not been updated 
for months, and it seems no one is looking at opened pull requests.

> Create topologies for measuring performance
> ---
>
> Key: STORM-1772
> URL: https://issues.apache.org/jira/browse/STORM-1772
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Roshan Naik
>Assignee: Roshan Naik
>
> Would be very useful to have some simple reference topologies included with 
> Storm that can be used to measure performance both by devs during development 
> (to start with) and perhaps also on a real storm cluster (subsequently). 
> To start with, the goal is to put the focus on the performance 
> characteristics of individual building blocks such as specifics bolts, 
> spouts,  grouping options, queues, etc. So, initially biased towards 
> micro-benchmarking but subsequently we could add higher level ones too.
> Although there is a storm benchmarking tool (originally written by Intel?) 
> that can be used, and i have personally used it, its better for this to be 
> integrated into Storm proper and also maintained by devs as storm evolves. 
> On a side note, in some instances I have noticed (to my surprise) that the 
> perf numbers change when the topologies written for Intel benchmark when 
> rewritten without the required wrappers so that they runs directly under 
> Storm.
> Have a few topologies in mind for measuring each of these:
> # *Queuing and Spout Emit Performance:* A topology with a Generator Spout but 
> no bolts.
> # *Queuing & Grouping performance*:   Generator Spout -> A grouping method -> 
> DevNull Bolt
> # *Hdfs Bolt:*Generator Spout ->  Hdfs Bolt
> # *Hdfs Spout:*   Hdfs Spout ->  DevNull Botl
> # *Kafka Spout:*   Kafka Spout ->  DevNull Bolt 
> # *Simple Data Movement*: Kafka Spout -> Hdfs Bolt
> Shall add these for Storm core first. Then we can have the same for Trident 
> also.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1765) KafkaBolt with new producer api should be part of storm-kaka-client

[Narendra Bidari (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288088#comment-15288088
 ] 

Narendra Bidari commented on STORM-1765:


[~hmclouro] ,[~sriharsha]: I have a quick question around Storm-kafka bolt, I 
thought it might be related to this Jira so decided to ask here. Is it possible 
to use KafkaBolt of pervious version to write messages to Kafka (0.9) enabled 
with SSL + ACL? or should STORM-1765 should address it?

> KafkaBolt with new producer api should be part of storm-kaka-client
> ---
>
> Key: STORM-1765
> URL: https://issues.apache.org/jira/browse/STORM-1765
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Sriharsha Chintalapani
>Assignee: Hugo Louro
>
> During the discussion of storm-kaka-client we agreed on following
> "We talked about this some more and it seems to me it would make more sense 
> to leave this new spout in storm-kafka-client (or whatever you want to call 
> it) and move the KafkaBolt which uses the new producer api over here also. 
> That way this component only needs to depend on the new kafka-clients java 
> api and not on the entire scala kafka core. We can make the old storm-kafka 
> depend on this component so it still picks up the bolt so if anyone is using 
> that its still works. We can deprecate the old KafkaSpout but keep it around 
> for people using older versions of Kafka - tgravescs"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287679#comment-15287679
 ] 

ASF GitHub Bot commented on STORM-1842:
---

Github user barrywhart commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219865469
  
I tested with the updated storm.thrift from this pull request. It addresses 
the Python Thrift issues I was encountering.

From my perspective, this change is ready for merge.


> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 Forward references in storm.thrift ...

[barrywhart]

Github user barrywhart commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219865469
  
I tested with the updated storm.thrift from this pull request. It addresses 
the Python Thrift issues I was encountering.

From my perspective, this change is ready for merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Barry Hart (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287676#comment-15287676
 ] 

Barry Hart commented on STORM-1842:
---

I tested with the updated storm.thrift from your pull request. This addresses 
my Python thrift issues.

I'm now working on some unrelated Java compilation and runtime issues.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-946) We should remove Closed Client form cached-node+port->socket in worker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287618#comment-15287618
 ] 

ASF GitHub Bot commented on STORM-946:
--

Github user kevinconaway commented on the pull request:

https://github.com/apache/storm/pull/639#issuecomment-219860409
  
Actually it looks like this issue

>There is another more serious issue that lead to huge problems in one of 
out topologies whenever a worker crashed due to some exception.
If worker A sucessfully connects to worker B for the first time during 
startup but worker B closes the connection for some reason before the 
:worker-active-flag is set to true (here 
https://github.com/apache/storm/blob/v0.9.6/storm-core/src/clj/backtype/storm/daemon/worker.clj#L356),
 there will be no further reconnect attempts, since no messages will be 
processed and neither send() nor flushMessages() will ever be called.

may be fixed by STORM-1609 with the addition of the client keepalive 
TimerTask


> We should remove Closed Client form cached-node+port->socket in worker
> --
>
> Key: STORM-946
> URL: https://issues.apache.org/jira/browse/STORM-946
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0, 1.0.0
>Reporter: xiajun
>
> The client may be Closed status after reconnect failed, and we will remove 
> closed client from Context to escape memory leak.
> But there is also reference for the closed Client in cached-node+port->socket 
> in worker, for this reason we should also remove closed Client from 
> cached-node+port->socket.  
> Meanwhile there is another reason for us to do so. Think about this 
> situation: worker A connect to worker B1 B2, but for some reason worker B1 B2 
> died at the same, then nimbus reschedule worker B1 B1. And new B1 B2 may 
> partly rescheduled at the some host:port as old B1 B2, that is (old B1: 
> host1+port1, old B2: host2+port2, new B1: host2+port2, new B2: host3+port3). 
> Worker A realized worker B1 B2 died and start reconnect to worker B1 B2, but 
> before new worker B1 and old B2 have the same host+port, and by the current 
> logic, we will remove old B1 Client and and create new Client for new worker 
> B2, and do nothing to old B2 and new B1 because they have the same host+port. 
> This will result the topology stop processing tuples. Once we remove closed 
> Client from cached-node+port->socket before refresh-connections, this  will 
> not happen again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-946: We should remove Closed Client form...

[kevinconaway]

Github user kevinconaway commented on the pull request:

https://github.com/apache/storm/pull/639#issuecomment-219860409
  
Actually it looks like this issue

>There is another more serious issue that lead to huge problems in one of 
out topologies whenever a worker crashed due to some exception.
If worker A sucessfully connects to worker B for the first time during 
startup but worker B closes the connection for some reason before the 
:worker-active-flag is set to true (here 
https://github.com/apache/storm/blob/v0.9.6/storm-core/src/clj/backtype/storm/daemon/worker.clj#L356),
 there will be no further reconnect attempts, since no messages will be 
processed and neither send() nor flushMessages() will ever be called.

may be fixed by STORM-1609 with the addition of the client keepalive 
TimerTask


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1394) Netty Client never continue reconnection when worker started a moment ago.

[Kevin Conaway (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287614#comment-15287614
 ] 

Kevin Conaway commented on STORM-1394:
--

This may be fixed by STORM-1609

> Netty Client never continue reconnection when worker started a moment ago.
> --
>
> Key: STORM-1394
> URL: https://issues.apache.org/jira/browse/STORM-1394
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Jingsong Lee
>
> Worker will wait until all connections are ok.
> This is a situation lead to worker hang forever.
> 1.worker A and B started.
> 2.worker A wait all connections are ok.
> 3.worker A connect to B.
> 4.B dead when A is not active(other connections are not ok).
> 5.B is launched by B' supervisor again (Assignment is not changed).
> 6.A hang forever because there is nobody to reconnect the client of A to B.
> We can fix this problem by 2 method.
> 1.Add closeChannelAndReconnect in Client' status
> Or
> 2.Add closeChannelAndReconnect in StormClientHandler' exceptionCaught



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1707) Improve supervisor latency by removing 2-min wait

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287562#comment-15287562
 ] 

ASF GitHub Bot commented on STORM-1707:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1370#issuecomment-219854927
  
Pending #1427 (1.x) to be verified.


> Improve supervisor latency by removing 2-min wait
> -
>
> Key: STORM-1707
> URL: https://issues.apache.org/jira/browse/STORM-1707
> Project: Apache Storm
>  Issue Type: Improvement
>Reporter: Paul Poulosky
>Assignee: Paul Poulosky
>
> After launching workers, the supervisor waits up to 2 minutes synchronously 
> for the workers to be "launched".
> We should remove this, and instead keep track of launch time, making the 
> "killer" function smart enough to determine the difference between a worker 
> that's still launching, one that's timed out, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1707] Remove two minute timeout after w...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1370#issuecomment-219854927
  
Pending #1427 (1.x) to be verified.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1701) Add simple JSON mapping to storm-hbase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287556#comment-15287556
 ] 

ASF GitHub Bot commented on STORM-1701:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1427#issuecomment-219854465
  
@ppoulosk I didn't take a look carefully yet but seems like builds are 
failing consistently. I've also seen build failure for local machine. Could you 
take a look?


> Add simple JSON mapping to storm-hbase
> --
>
> Key: STORM-1701
> URL: https://issues.apache.org/jira/browse/STORM-1701
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Kristopher Kane
>Priority: Trivial
> Fix For: 2.0.0
>
>
> storm-hbase includes a way to map Storm fields to HBase CQs.  This adds a 
> similar ability where a single field contains a flat JSON and the keys map to 
> CQs.  The flat JSON must contain the row key.  
> No intention of reverse mapping from HBaseLookUp as the existing HBaseMapper 
> works well for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1701] Port PR 1370 to 1.x-branch (Git r...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1427#issuecomment-219854465
  
@ppoulosk I didn't take a look carefully yet but seems like builds are 
failing consistently. I've also seen build failure for local machine. Could you 
take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286594#comment-15286594
 ] 

Jungtaek Lim edited comment on STORM-1842 at 5/17/16 8:54 PM:
--

[~barrywhart] 
Here's custom build of Storm 1.0.2 SNAPSHOT + STORM-1842 patch.
http://people.apache.org/~kabhwan/storm-core-1.0.2-SNAPSHOT.jar

If it incurs other issues please let me know so that I just apply STORM-1842 
with Storm 1.0.1 and rebuild.


was (Author: kabhwan):
[~barrywhart] 
Here's custom build of Storm 1.0.2 SNAPSHOT + STORM-1842 patch.
http://people.apache.org/~kabhwan/storm-core-1.0.2-SNAPSHOT.jar

If it incurs other issues please let me know that I just apply STORM-1842 with 
Storm 1.0.1 and rebuild.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1701) Add simple JSON mapping to storm-hbase

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287504#comment-15287504
 ] 

ASF GitHub Bot commented on STORM-1701:
---

GitHub user ppoulosk opened a pull request:

https://github.com/apache/storm/pull/1427

[STORM-1701] Port PR 1370 to 1.x-branch  (Git rid of hardcoded wait after 
worker launch)

Ported this to 1.x branch from master.  @HeartSaVioR 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppoulosk/storm PortTo1xBranch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1427


commit a9d51c60a4ffa27da9503a42d329906196912f2c
Author: Paul Poulosky 
Date:   2016-05-17T20:30:03Z

Port PR 1370 to 1.x-branch




> Add simple JSON mapping to storm-hbase
> --
>
> Key: STORM-1701
> URL: https://issues.apache.org/jira/browse/STORM-1701
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-hbase
>Reporter: Kristopher Kane
>Priority: Trivial
> Fix For: 2.0.0
>
>
> storm-hbase includes a way to map Storm fields to HBase CQs.  This adds a 
> similar ability where a single field contains a flat JSON and the keys map to 
> CQs.  The flat JSON must contain the row key.  
> No intention of reverse mapping from HBaseLookUp as the existing HBaseMapper 
> works well for this. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: [STORM-1701] Port PR 1370 to 1.x-branch (Git r...

[ppoulosk]

GitHub user ppoulosk opened a pull request:

https://github.com/apache/storm/pull/1427

[STORM-1701] Port PR 1370 to 1.x-branch  (Git rid of hardcoded wait after 
worker launch)

Ported this to 1.x branch from master.  @HeartSaVioR 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ppoulosk/storm PortTo1xBranch

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1427.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1427


commit a9d51c60a4ffa27da9503a42d329906196912f2c
Author: Paul Poulosky 
Date:   2016-05-17T20:30:03Z

Port PR 1370 to 1.x-branch




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1682) Kafka spout can lose partitions

[JIRA]

 [ 
https://issues.apache.org/jira/browse/STORM-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stig Rohde Døssing updated STORM-1682:
--
Description: 
The KafkaSpout can lose partitions for a period, or hang because getBrokersInfo 
(https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L77)
 may get a NoNodeException if there is no broker info in Zookeeper 
corresponding to the leader id in Zookeeper. When this error occurs, the spout 
ignores the partition until the next time getBrokersInfo is called, which isn't 
until the next time the spout gets an exception on fetch. If the timing is 
really bad, it might ignore all the partitions and never restart.

As far as I'm aware, Kafka doesn't update leader and brokerinfo atomically, so 
it's possible to get unlucky and hit the NoNodeException when a broker has just 
died.

I have a few suggestions for dealing with this. 

getBrokerInfo could simply retry the inner loop over partitions if it gets the 
NoNodeException (probably with a limit and a short sleep between attempts). If 
it fails repeatedly, the spout should be crashed.

Alternatively the DynamicBrokersReader could instead lookup all brokers in 
Zookeeper, create a consumer and send a TopicMetadataRequest on it. The 
response contains the leader for each partition and host/port for the relevant 
brokers.

Edit: I noticed that the spout periodically refreshes the brokers info, so the 
issue isn't as bad as I thought. I still think this change has value, since it 
avoids the spout temporarily dropping a partition.

  was:
The KafkaSpout can lose partitions for a period, or hang because getBrokersInfo 
(https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L77)
 may get a NoNodeException if there is no broker info in Zookeeper 
corresponding to the leader id in Zookeeper. When this error occurs, the spout 
ignores the partition until the next time getBrokersInfo is called, which isn't 
until the next time the spout gets an exception on fetch. If the timing is 
really bad, it might ignore all the partitions and never restart.

As far as I'm aware, Kafka doesn't update leader and brokerinfo atomically, so 
it's possible to get unlucky and hit the NoNodeException when a broker has just 
died.

I have a few suggestions for dealing with this. 

getBrokerInfo could simply retry the inner loop over partitions if it gets the 
NoNodeException (probably with a limit and a short sleep between attempts). If 
it fails repeatedly, the spout should be crashed.

Alternatively the DynamicBrokersReader could instead lookup all brokers in 
Zookeeper, create a consumer and send a TopicMetadataRequest on it. The 
response contains the leader for each partition and host/port for the relevant 
brokers.


> Kafka spout can lose partitions
> ---
>
> Key: STORM-1682
> URL: https://issues.apache.org/jira/browse/STORM-1682
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> The KafkaSpout can lose partitions for a period, or hang because 
> getBrokersInfo 
> (https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L77)
>  may get a NoNodeException if there is no broker info in Zookeeper 
> corresponding to the leader id in Zookeeper. When this error occurs, the 
> spout ignores the partition until the next time getBrokersInfo is called, 
> which isn't until the next time the spout gets an exception on fetch. If the 
> timing is really bad, it might ignore all the partitions and never restart.
> As far as I'm aware, Kafka doesn't update leader and brokerinfo atomically, 
> so it's possible to get unlucky and hit the NoNodeException when a broker has 
> just died.
> I have a few suggestions for dealing with this. 
> getBrokerInfo could simply retry the inner loop over partitions if it gets 
> the NoNodeException (probably with a limit and a short sleep between 
> attempts). If it fails repeatedly, the spout should be crashed.
> Alternatively the DynamicBrokersReader could instead lookup all brokers in 
> Zookeeper, create a consumer and send a TopicMetadataRequest on it. The 
> response contains the leader for each partition and host/port for the 
> relevant brokers.
> Edit: I noticed that the spout periodically refreshes the brokers info, so 
> the issue isn't as bad as I thought. I still think this change has value, 
> since it avoids the spout temporarily dropping a partition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1394) Netty Client never continue reconnection when worker started a moment ago.

[Kevin Conaway (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287468#comment-15287468
 ] 

Kevin Conaway commented on STORM-1394:
--

There is some related discussion to this on STORM-946

> Netty Client never continue reconnection when worker started a moment ago.
> --
>
> Key: STORM-1394
> URL: https://issues.apache.org/jira/browse/STORM-1394
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Reporter: Jingsong Lee
>
> Worker will wait until all connections are ok.
> This is a situation lead to worker hang forever.
> 1.worker A and B started.
> 2.worker A wait all connections are ok.
> 3.worker A connect to B.
> 4.B dead when A is not active(other connections are not ok).
> 5.B is launched by B' supervisor again (Assignment is not changed).
> 6.A hang forever because there is nobody to reconnect the client of A to B.
> We can fix this problem by 2 method.
> 1.Add closeChannelAndReconnect in Client' status
> Or
> 2.Add closeChannelAndReconnect in StormClientHandler' exceptionCaught



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1682) Kafka spout can lose partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287429#comment-15287429
 ] 

ASF GitHub Bot commented on STORM-1682:
---

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1340

STORM-1682: Refactor DynamicBrokersReader to lookup leader informatio…

…n via Kafka metadata API. Put Zookeeper metadata lookup into new class.

See https://issues.apache.org/jira/browse/STORM-1682.

This solution splits DynamicBrokersReader into two. Instead of looking up 
leaders and brokers in Zookeeper, ZkBrokersReader will use the brokers in 
Zookeeper to construct one or more SimpleConsumers, which DynamicBrokersReader 
can then use to lookup partition leaders.

DynamicBrokersReader should now cause the spout to restart in the following 
scenarios:
If it can't find any brokers in Zookeeper
If it can't contact any of the brokers it found in Zookeeper
If wildcard topics are listed in Zookeeper, and no broker has metadata for 
all of them

In order to test behavior when brokers shut down, it was necessary to 
separate the Kafka test broker from Zookeeper so multiple brokers can run in a 
cluster, which is the reason for modifications in many tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1682

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1340


commit 78a3d0683d1bfbfec7bbc7425927347b4b47154d
Author: Stig Rohde Døssing 
Date:   2016-04-12T18:24:49Z

STORM-1682: Refactor DynamicBrokersReader to lookup leader information via 
Kafka metadata API. Put Zookeeper metadata lookup into new class.

commit af72ce6229ec1332d7569846121e3b07b3b42b33
Author: Stig Rohde Døssing 
Date:   2016-04-14T14:37:47Z

STORM-1682: Add check of metadata request error code in DynamicBrokersReader

commit 430473be42f68d65aed471af9a808f5c7c599f39
Author: Stig Rohde Døssing 
Date:   2016-04-14T15:38:09Z

STORM-1682: Replace TopicCommand with AdminUtils, replace 
KafkaServerStartable with KafkaServer. The two former contain System.exit calls

commit 3d323392c1f809d6e829b641d0ecb1719865bde8
Author: Stig Rohde Døssing 
Date:   2016-04-21T09:50:20Z

Merge branch 'master' of https://github.com/apache/storm into STORM-1682

commit e9f3f860b5aecccd9c9fb9cc086fac1bb619f5bd
Author: Stig Rohde Døssing 
Date:   2016-04-21T09:45:39Z

STORM-1682: Fix NPE in DynamicBrokersReader when leader is missing/down

commit dc3c0fb73cb1074fd7d38d1ada9ef0c021957eb5
Author: Stig Rohde Døssing 
Date:   2016-05-16T11:38:54Z

STORM-1682: Adjust for Kafka 0.8.2.1




> Kafka spout can lose partitions
> ---
>
> Key: STORM-1682
> URL: https://issues.apache.org/jira/browse/STORM-1682
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> The KafkaSpout can lose partitions for a period, or hang because 
> getBrokersInfo 
> (https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L77)
>  may get a NoNodeException if there is no broker info in Zookeeper 
> corresponding to the leader id in Zookeeper. When this error occurs, the 
> spout ignores the partition until the next time getBrokersInfo is called, 
> which isn't until the next time the spout gets an exception on fetch. If the 
> timing is really bad, it might ignore all the partitions and never restart.
> As far as I'm aware, Kafka doesn't update leader and brokerinfo atomically, 
> so it's possible to get unlucky and hit the NoNodeException when a broker has 
> just died.
> I have a few suggestions for dealing with this. 
> getBrokerInfo could simply retry the inner loop over partitions if it gets 
> the NoNodeException (probably with a limit and a short sleep between 
> attempts). If it fails repeatedly, the spout should be crashed.
> Alternatively the DynamicBrokersReader could instead lookup all brokers in 
> Zookeeper, create a consumer and send a TopicMetadataRequest on it. The 
> response contains the leader for each partition and host/port for the 
> relevant brokers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1682) Kafka spout can lose partitions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287428#comment-15287428
 ] 

ASF GitHub Bot commented on STORM-1682:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1340#issuecomment-219834590
  
The tests pass on 0.8.2.1, which is the version targeted by 
https://github.com/apache/storm/pull/1386. Reopening.


> Kafka spout can lose partitions
> ---
>
> Key: STORM-1682
> URL: https://issues.apache.org/jira/browse/STORM-1682
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>
> The KafkaSpout can lose partitions for a period, or hang because 
> getBrokersInfo 
> (https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/DynamicBrokersReader.java#L77)
>  may get a NoNodeException if there is no broker info in Zookeeper 
> corresponding to the leader id in Zookeeper. When this error occurs, the 
> spout ignores the partition until the next time getBrokersInfo is called, 
> which isn't until the next time the spout gets an exception on fetch. If the 
> timing is really bad, it might ignore all the partitions and never restart.
> As far as I'm aware, Kafka doesn't update leader and brokerinfo atomically, 
> so it's possible to get unlucky and hit the NoNodeException when a broker has 
> just died.
> I have a few suggestions for dealing with this. 
> getBrokerInfo could simply retry the inner loop over partitions if it gets 
> the NoNodeException (probably with a limit and a short sleep between 
> attempts). If it fails repeatedly, the spout should be crashed.
> Alternatively the DynamicBrokersReader could instead lookup all brokers in 
> Zookeeper, create a consumer and send a TopicMetadataRequest on it. The 
> response contains the leader for each partition and host/port for the 
> relevant brokers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1340#issuecomment-219834590
  
The tests pass on 0.8.2.1, which is the version targeted by 
https://github.com/apache/storm/pull/1386. Reopening.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1682: Refactor DynamicBrokersReader to l...

[srdo]

GitHub user srdo reopened a pull request:

https://github.com/apache/storm/pull/1340

STORM-1682: Refactor DynamicBrokersReader to lookup leader informatio…

…n via Kafka metadata API. Put Zookeeper metadata lookup into new class.

See https://issues.apache.org/jira/browse/STORM-1682.

This solution splits DynamicBrokersReader into two. Instead of looking up 
leaders and brokers in Zookeeper, ZkBrokersReader will use the brokers in 
Zookeeper to construct one or more SimpleConsumers, which DynamicBrokersReader 
can then use to lookup partition leaders.

DynamicBrokersReader should now cause the spout to restart in the following 
scenarios:
If it can't find any brokers in Zookeeper
If it can't contact any of the brokers it found in Zookeeper
If wildcard topics are listed in Zookeeper, and no broker has metadata for 
all of them

In order to test behavior when brokers shut down, it was necessary to 
separate the Kafka test broker from Zookeeper so multiple brokers can run in a 
cluster, which is the reason for modifications in many tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1682

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1340.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1340


commit 78a3d0683d1bfbfec7bbc7425927347b4b47154d
Author: Stig Rohde Døssing 
Date:   2016-04-12T18:24:49Z

STORM-1682: Refactor DynamicBrokersReader to lookup leader information via 
Kafka metadata API. Put Zookeeper metadata lookup into new class.

commit af72ce6229ec1332d7569846121e3b07b3b42b33
Author: Stig Rohde Døssing 
Date:   2016-04-14T14:37:47Z

STORM-1682: Add check of metadata request error code in DynamicBrokersReader

commit 430473be42f68d65aed471af9a808f5c7c599f39
Author: Stig Rohde Døssing 
Date:   2016-04-14T15:38:09Z

STORM-1682: Replace TopicCommand with AdminUtils, replace 
KafkaServerStartable with KafkaServer. The two former contain System.exit calls

commit 3d323392c1f809d6e829b641d0ecb1719865bde8
Author: Stig Rohde Døssing 
Date:   2016-04-21T09:50:20Z

Merge branch 'master' of https://github.com/apache/storm into STORM-1682

commit e9f3f860b5aecccd9c9fb9cc086fac1bb619f5bd
Author: Stig Rohde Døssing 
Date:   2016-04-21T09:45:39Z

STORM-1682: Fix NPE in DynamicBrokersReader when leader is missing/down

commit dc3c0fb73cb1074fd7d38d1ada9ef0c021957eb5
Author: Stig Rohde Døssing 
Date:   2016-05-16T11:38:54Z

STORM-1682: Adjust for Kafka 0.8.2.1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-946: We should remove Closed Client form...

[kevinconaway]

Github user kevinconaway commented on the pull request:

https://github.com/apache/storm/pull/639#issuecomment-219828158
  
@revans2 was there ever an issue filed about this? We are seeing the same 
behavior in Storm 0.10.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-946) We should remove Closed Client form cached-node+port->socket in worker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287366#comment-15287366
 ] 

ASF GitHub Bot commented on STORM-946:
--

Github user kevinconaway commented on the pull request:

https://github.com/apache/storm/pull/639#issuecomment-219828158
  
@revans2 was there ever an issue filed about this? We are seeing the same 
behavior in Storm 0.10.0


> We should remove Closed Client form cached-node+port->socket in worker
> --
>
> Key: STORM-946
> URL: https://issues.apache.org/jira/browse/STORM-946
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 0.10.0, 1.0.0
>Reporter: xiajun
>
> The client may be Closed status after reconnect failed, and we will remove 
> closed client from Context to escape memory leak.
> But there is also reference for the closed Client in cached-node+port->socket 
> in worker, for this reason we should also remove closed Client from 
> cached-node+port->socket.  
> Meanwhile there is another reason for us to do so. Think about this 
> situation: worker A connect to worker B1 B2, but for some reason worker B1 B2 
> died at the same, then nimbus reschedule worker B1 B1. And new B1 B2 may 
> partly rescheduled at the some host:port as old B1 B2, that is (old B1: 
> host1+port1, old B2: host2+port2, new B1: host2+port2, new B2: host3+port3). 
> Worker A realized worker B1 B2 died and start reconnect to worker B1 B2, but 
> before new worker B1 and old B2 have the same host+port, and by the current 
> logic, we will remove old B1 Client and and create new Client for new worker 
> B2, and do nothing to old B2 and new B1 because they have the same host+port. 
> This will result the topology stop processing tuples. Once we remove closed 
> Client from cached-node+port->socket before refresh-connections, this  will 
> not happen again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287241#comment-15287241
 ] 

Robert Joseph Evans commented on STORM-1757:


OK so that was not clear from the javadocs in state. I should have read through 
more of the documentation that you sent out.  Thanks for putting up with some 
of my crazy questions.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Arun Mahadevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287098#comment-15287098
 ] 

Arun Mahadevan commented on STORM-1757:
---

> So if I ask all bolts to checkpoint and most succeed but one does not, I can 
> not longer restore everyone to a consistent place, so I can start replaying 
> again. This is what I am concerned about.

The checkpointing handle this via a prepare and a commit phase. 
- First a "prepare" message is send. If the prepare fails (most of the bolts 
succeeded but one did not), the checkpoint is restored to the last successful 
point (rollback) and tuples are replayed.
- If prepare succeeds, a "commit" message is send.
- If commit fails, the commit is re-attempted by sending the "commit" message 
again. The bolts that had already committed ignores this and the bolt that had 
previously not committed would now commit. They can do so because the prepared 
data is persisted in the state during prepare phase.
- Once the commit succeeds the txn is marked as complete. 

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1838) [storm-kafka-client] the resumed OffsetEntry goes different

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287033#comment-15287033
 ] 

ASF GitHub Bot commented on STORM-1838:
---

Github user hmcl commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219788238
  
@flisky so what does your last comment mean? That this patch no longer 
applies?


> [storm-kafka-client] the resumed OffsetEntry goes different
> ---
>
> Key: STORM-1838
> URL: https://issues.apache.org/jira/browse/STORM-1838
> Project: Apache Storm
>  Issue Type: Bug
>Reporter: Jifeng Yin
> Fix For: 1.0.1
>
>
> There are no more new messages after consumer rejoins the group. And it turns 
> out the old OffsetEntry instance kept commitOffsetsForAckedTuples failing.
> the comment "leave the acked offsets as they were to resume where it left 
> off" doesn't work for me, but the ackedMsgs goes different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1838 update OffsetEntry to support EARLI...

[hmcl]

Github user hmcl commented on the pull request:

https://github.com/apache/storm/pull/1419#issuecomment-219788238
  
@flisky so what does your last comment mean? That this patch no longer 
applies?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286933#comment-15286933
 ] 

Robert Joseph Evans commented on STORM-1757:


[~arunmahadevan],

BEAM does exactly once, by doing at least once, but making sure that the 
checkpoints happen in an idempotent and coordinated way.  The issue is not at 
least once vs exactly once, but is around failure cases and consistency.  From 
what I have seen with the state API + windowing offers some coordination and 
restore, reprocess, re-checkpoint should be idempotent, but it can only role 
back to the last successful checkpoint.  So if I ask all bolts to checkpoint 
and most succeed but one does not, I can not longer restore everyone to a 
consistent place, so I can start replaying again.  This is what I am concerned 
about.  I would like to be able to fully support the BEAM model, and then if we 
can offer better performance by reducing guarantees I am OK with that.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1599) Don't mark dependencies as provided unless they are in lib

[Robert Joseph Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated STORM-1599:
---
Description: 
When we mark a dependency as provided it indicates the shade and assembly 
plugins to not include this particular dependency in the uber topology jar 
because it will be {{provided}} on the class path by the system.

We have been doing this for all of our kafka dependencies incorrectly, 
storm-cassandra does this for cassandra-driver-core, and storm-starter is doing 
it for storm-clojure as well.

This means that storm-starter does not have any version of kafka or 
storm-clojure packaged it the resulting jar and any example that uses kafka, 
TridentKafkaWordCount, will fail with missing class errors. 

storm-starter/pom.xml has should change its dependency on storm-kafka to be 
compile, and it should delete dependencies on kafka and kafka-clients as those 
should come from storm-kafka as transitive dependencies.

the main pom.xml should not have kafka-clients marked as provided in the 
dependency management section.

storm-kafka should remove its provided tag on kafka, and flux examples + 
storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
storm-kafka should not me marked as provided. 

the flux and sql code I am not as familiar with, but looking at them, and 
running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like

  was:
When we mark a dependency as provided it indicates the shade and assembly 
plugins to not include this particular dependency in the uber topology jar 
because it will be {provided} on the class path by the system.

We have been doing this for all of our kafka dependencies incorrectly.  This 
means that storm-starter does not have any version of kafka packaged it the 
resulting jar and any example that uses kafka, TridentKafkaWordCount, will fail 
with missing class errors. 

storm-starter/pom.xml has should change its dependency on storm-kafka to be 
compile, and it should delete dependencies on kafka and kafka-clients as those 
should come from storm-kafka as transitive dependencies.

the main pom.xml should not have kafka-clients marked as provided in the 
dependency management section.

storm-kafka should remove its provided tag on kafka, and flux examples + 
storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
storm-kafka should not me marked as provided. 

the flux and sql code I am not as familiar with, but looking at them, and 
running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like


> Don't mark dependencies as provided unless they are in lib
> --
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
>  Issue Type: Bug
>  Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly 
> plugins to not include this particular dependency in the uber topology jar 
> because it will be {{provided}} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly, 
> storm-cassandra does this for cassandra-driver-core, and storm-starter is 
> doing it for storm-clojure as well.
> This means that storm-starter does not have any version of kafka or 
> storm-clojure packaged it the resulting jar and any example that uses kafka, 
> TridentKafkaWordCount, will fail with missing class errors. 
> storm-starter/pom.xml has should change its dependency on storm-kafka to be 
> compile, and it should delete dependencies on kafka and kafka-clients as 
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the 
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples + 
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
> storm-kafka should not me marked as provided. 
> the flux and sql code I am not as familiar with, but looking at them, and 
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1599) Kafka dependencies all marked as provided (so storm-starter does not run)

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286905#comment-15286905
 ] 

Robert Joseph Evans commented on STORM-1599:


I think that marking them as provided is the outlier.  The majority of storm-* 
don't mark their dependencies as provided.  In general though we should work 
towards doing a much better job of managing the system classpath, dependencies, 
and making it consistent everywhere.  Hopefully we can do that as part of this 
JIRA.

Right now the only things that storm "provides" on the system classpath are 
storm-core and it's non-shaded dependencies, and storm-rename-hack.  Nothing 
else is truly provided by storm.

I ran {{find . -iname pom.xml | xargs grep -B 3 provided | grep artifactId | 
egrep -v 'storm-core|storm-rename-hack'}} to find everywhere that we list 
something as provided that really isn't

{code}
./examples/storm-starter/pom.xml-storm-clojure
./examples/storm-starter/pom.xml-  storm-kafka
./examples/storm-starter/pom.xml-  
${kafka.artifact.id}
./external/sql/storm-sql-core/pom.xml-
storm-sql-runtime
./external/sql/storm-sql-kafka/pom.xml-
storm-sql-runtime
./external/sql/storm-sql-kafka/pom.xml-
storm-kafka
./external/sql/storm-sql-kafka/pom.xml-
${kafka.artifact.id}
./external/storm-cassandra/pom.xml-
cassandra-driver-core
./external/storm-kafka/pom.xml-
${kafka.artifact.id}
./storm-buildtools/maven-shade-clojure-transformer/pom.xml-
maven-shade-plugin
./storm-buildtools/storm-maven-plugins/pom.xml-  
maven-plugin-annotations
{code}

storm-clojure is new in 2.0 and should not be provided.  It is not in lib.
storm-kafka and kafka.artifact.id is what this jira is about.  storm-sql-* had 
some issues where it was on the classpath even though it should not have been, 
so they need to be updated accordingly too.

Looks like storm-cassandra should have the cassandra-driver-core fixed as well. 
 Then all that is left is the build-tools, which are not part of a topology jar 
and can be ignored.

As such I will update the title of this JIRA to indicate that we should fix 
storm-cassandra and strom-starter's dependency on storm-clojure.

> Kafka dependencies all marked as provided (so storm-starter does not run)
> -
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
>  Issue Type: Bug
>  Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly 
> plugins to not include this particular dependency in the uber topology jar 
> because it will be {provided} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly.  This 
> means that storm-starter does not have any version of kafka packaged it the 
> resulting jar and any example that uses kafka, TridentKafkaWordCount, will 
> fail with missing class errors. 
> storm-starter/pom.xml has should change its dependency on storm-kafka to be 
> compile, and it should delete dependencies on kafka and kafka-clients as 
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the 
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples + 
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
> storm-kafka should not me marked as provided. 
> the flux and sql code I am not as familiar with, but looking at them, and 
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (STORM-1599) Don't mark dependencies as provided unless they are in lib

[Robert Joseph Evans (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated STORM-1599:
---
Summary: Don't mark dependencies as provided unless they are in lib  (was: 
Kafka dependencies all marked as provided (so storm-starter does not run))

> Don't mark dependencies as provided unless they are in lib
> --
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
>  Issue Type: Bug
>  Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly 
> plugins to not include this particular dependency in the uber topology jar 
> because it will be {provided} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly.  This 
> means that storm-starter does not have any version of kafka packaged it the 
> resulting jar and any example that uses kafka, TridentKafkaWordCount, will 
> fail with missing class errors. 
> storm-starter/pom.xml has should change its dependency on storm-kafka to be 
> compile, and it should delete dependencies on kafka and kafka-clients as 
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the 
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples + 
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
> storm-kafka should not me marked as provided. 
> the flux and sql code I am not as familiar with, but looking at them, and 
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Arun Mahadevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286896#comment-15286896
 ] 

Arun Mahadevan commented on STORM-1757:
---

The state transition doc tries to explain some of this.

https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/spout/CheckPointState.java#L25


> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Arun Mahadevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286881#comment-15286881
 ] 

Arun Mahadevan commented on STORM-1757:
---

[~revans2] the current checkpointing supports at-least once and may be we could 
use it as is for the initial prototype if we are ok with the guarantee. The 
current implementation does not checkpoint the state of the spout, but I think 
it can be extended to do so either by having the checkpoint spout co-ordinating 
with user spouts via zookeeper or by having  the checkpoint spout act like a 
co-ordinator with user spouts running as bolts (similar to trident). I will put 
more thought around this and try to come up with a prototype. 

I am not sure I understand correctly the requirement for having a commit id in 
rollback. The current restore happens by discarding any prepared (but 
un-commited) changes or rolling forward any commits that were in progress.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1599) Kafka dependencies all marked as provided (so storm-starter does not run)

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286777#comment-15286777
 ] 

Sriharsha Chintalapani commented on STORM-1599:
---

[~revans2] sure thats works. But we are inconsistent with this approach with 
other connectors and also we are packaging external dependencies into 
connectors as well. If the issue only with running examples than we should 
include it in the module rather than the other way.

> Kafka dependencies all marked as provided (so storm-starter does not run)
> -
>
> Key: STORM-1599
> URL: https://issues.apache.org/jira/browse/STORM-1599
> Project: Apache Storm
>  Issue Type: Bug
>  Components: examples, Flux, storm-kafka
>Affects Versions: 0.10.0, 1.0.0, 2.0.0
>Reporter: Robert Joseph Evans
>Assignee: Hugo Louro
>
> When we mark a dependency as provided it indicates the shade and assembly 
> plugins to not include this particular dependency in the uber topology jar 
> because it will be {provided} on the class path by the system.
> We have been doing this for all of our kafka dependencies incorrectly.  This 
> means that storm-starter does not have any version of kafka packaged it the 
> resulting jar and any example that uses kafka, TridentKafkaWordCount, will 
> fail with missing class errors. 
> storm-starter/pom.xml has should change its dependency on storm-kafka to be 
> compile, and it should delete dependencies on kafka and kafka-clients as 
> those should come from storm-kafka as transitive dependencies.
> the main pom.xml should not have kafka-clients marked as provided in the 
> dependency management section.
> storm-kafka should remove its provided tag on kafka, and flux examples + 
> storm-sql-kafka should remove dependencies on kafka and kafka-clients, and 
> storm-kafka should not me marked as provided. 
> the flux and sql code I am not as familiar with, but looking at them, and 
> running `mvn dependecy:tree` and `mvn dependency:analyze` it looks like



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1844) Some tests are flaky due to low timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286760#comment-15286760
 ] 

ASF GitHub Bot commented on STORM-1844:
---

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1421#issuecomment-219741847
  
This should help us avoid more of those unwanted build failures.


> Some tests are flaky due to low timeout
> ---
>
> Key: STORM-1844
> URL: https://issues.apache.org/jira/browse/STORM-1844
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Minor
>
> We saw a few test failures likely due to low timeouts 
> https://github.com/apache/storm/pull/1417#issuecomment-219545865. The 
> timeouts for these tests should be increased, and the test mentioned by 
> [~kishorvpatil] should reflect STORM_TEST_TIMEOUT_MS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1844) Some tests are flaky due to low timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286758#comment-15286758
 ] 

ASF GitHub Bot commented on STORM-1844:
---

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1421#issuecomment-219741678
  
+1


> Some tests are flaky due to low timeout
> ---
>
> Key: STORM-1844
> URL: https://issues.apache.org/jira/browse/STORM-1844
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Minor
>
> We saw a few test failures likely due to low timeouts 
> https://github.com/apache/storm/pull/1417#issuecomment-219545865. The 
> timeouts for these tests should be increased, and the test mentioned by 
> [~kishorvpatil] should reflect STORM_TEST_TIMEOUT_MS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1837) Running local clusters without simulating time breaks Testing.completeTopology, and may cause message loss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286756#comment-15286756
 ] 

ASF GitHub Bot commented on STORM-1837:
---

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1417#issuecomment-219741531
  
@HeartSaVioR, you are right, I did not notice the `DisruptorQueueTest` 
failure.
@srdo, Thank you for putting up #1421. It's ironic to see `LocalNimbusTest` 
fail for that PR. :) I am bumping this on up.
+1


> Running local clusters without simulating time breaks 
> Testing.completeTopology, and may cause message loss
> --
>
> Key: STORM-1837
> URL: https://issues.apache.org/jira/browse/STORM-1837
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.0.1
>Reporter: Stig Rohde Døssing
>Priority: Minor
>
> Since https://github.com/apache/storm/pull/810 it is no longer possible to 
> call Testing.completeTopology when time is not simulating, because a call to 
> advance-cluster-time is made from the function, which calls Time/advanceTime. 
> advance-cluster-time should only be called if time is simulating.
> Since https://github.com/apache/storm/pull/830 a local cluster run without 
> time simulation may lose messages. When a worker emits messages for a worker 
> that hasn't started yet, the message is lost. This can happen because spouts 
> may start emitting before all workers have started, when time simulation is 
> disabled. Local clusters usually run without message timeouts, so this will 
> make tests relying on Testing.withLocalCluster flaky.
> The problem is that there are no longer any queues to store messages for 
> workers that haven't started yet. See 
> https://github.com/apache/storm/pull/830/files#diff-c6ff4208ef84c7a5a1a6b8b6bd1f7d19R104.
>  A queue should be added for messages for workers that haven't registered a 
> receive callback yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

[kishorvpatil]

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1417#issuecomment-219741531
  
@HeartSaVioR, you are right, I did not notice the `DisruptorQueueTest` 
failure.
@srdo, Thank you for putting up #1421. It's ironic to see `LocalNimbusTest` 
fail for that PR. :) I am bumping this on up.
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1844: Make netty unit test use STORM_TES...

[kishorvpatil]

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1421#issuecomment-219741847
  
This should help us avoid more of those unwanted build failures.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1844: Make netty unit test use STORM_TES...

[kishorvpatil]

Github user kishorvpatil commented on the pull request:

https://github.com/apache/storm/pull/1421#issuecomment-219741678
  
+1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1845) use UTF-8 instead of default encoding

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286717#comment-15286717
 ] 

ASF GitHub Bot commented on STORM-1845:
---

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/1422#issuecomment-219735852
  
default: `lookupCharset(System.getProperty("file.encoding")) == null ? 
"UTF-8" : System.getProperty("file.encoding")`


> use UTF-8 instead of default encoding
> -
>
> Key: STORM-1845
> URL: https://issues.apache.org/jira/browse/STORM-1845
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hbase, storm-hdfs, storm-hive, storm-redis
>Affects Versions: 1.0.1
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1845: use UTF-8 instead of default encod...

[vesense]

Github user vesense commented on the pull request:

https://github.com/apache/storm/pull/1422#issuecomment-219735852
  
default: `lookupCharset(System.getProperty("file.encoding")) == null ? 
"UTF-8" : System.getProperty("file.encoding")`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286713#comment-15286713
 ] 

Robert Joseph Evans commented on STORM-1757:


[~satish.duggana],

I'm not totally sure if this is the right place to ask this or not.  If you 
want me to move it to STORM-1763 I ma happy to.  How do we plan on doing a 
coordinated restore using the current 
[State|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/state/State.java]
 API?  I supposed we can extend it so rollback optionally supports a commit id, 
and there is an API to release commits older than a give commit id that are not 
needed any more.  Should I file a separate JIRA for something like that?  
Unless we want to start out with the simpler one micro batch at a time 
processing.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Robert Joseph Evans (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286704#comment-15286704
 ] 

Robert Joseph Evans commented on STORM-1757:


[~satish.duggana],

That all sounds great.   So what is really left is coordination between the 
sources and a translation layer, followed by lots and lots of 
optimizations/testing.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1674) Idle KafkaSpout consumes more bandwidth than needed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286687#comment-15286687
 ] 

ASF GitHub Bot commented on STORM-1674:
---

Github user harshach commented on the pull request:

https://github.com/apache/storm/pull/1393#issuecomment-219730884
  
+1


> Idle KafkaSpout consumes more bandwidth than needed
> ---
>
> Key: STORM-1674
> URL: https://issues.apache.org/jira/browse/STORM-1674
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Robert Hastings
>
> Discovered 30 megabits of traffic flowing between a set of KafkaSpouts
> and our kafka servers even though no Kafka messages were moving.
> Using the wireshark kafka dissector, we were able to see that
> each FetchRequest had maxWait set to 1
> and minBytes set to 0. When binBytes is set to 0 the kafka server
> responds immediately when there are no messages. In turn the KafkaSpout
> polls without any delay causing a constant stream of FetchRequest/
> FetchResponse messages. Using a non-KafkaSpout client had a similar
> traffic pattern with two key differences
> 1) minBytes was 1
> 2) maxWait was 100
> With these FetchRequest parameters and no messages flowing,
> the kafka server delays the FetchResponse by 100 ms. This reduces
> the network traffic from megabits to the low kilobits. It also
> reduced the CPU utilization of our kafka server from 140% to 2%.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Satish Duggana (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286663#comment-15286663
 ] 

Satish Duggana commented on STORM-1757:
---

[~arunmahadevan] These features are done in core to have parity with Beam 
windowing features. Beam runner  implementation can reuse this code.
Trident will be enhanced later once these are done.

> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1847) add popular storm related projects(from github) in docs

[Xin Wang (JIRA)]

Xin Wang created STORM-1847:
---

 Summary: add popular storm related projects(from github) in docs
 Key: STORM-1847
 URL: https://issues.apache.org/jira/browse/STORM-1847
 Project: Apache Storm
  Issue Type: Improvement
  Components: documentation
Reporter: Xin Wang
Assignee: Xin Wang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: FluxShellBolt/FluxShellSpout can now emit into...

[Darkless012]

Github user Darkless012 commented on the pull request:

https://github.com/apache/storm/pull/1426#issuecomment-219719857
  
Or should this pull request be to branch 1.0.x ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1757) Apache Beam Runner for Storm

[Satish Duggana (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286623#comment-15286623
 ] 

Satish Duggana commented on STORM-1757:
---

[~revans2] Currently, windowing support exists for processing time/event time 
with watermarks with a single trigger/eviction policy other than count based 
policies.

I think we can enhance core windowing for supporting multiple active 
windows/panes with multiple triggers/custom eviction policies as mentioned in 
earlier 
[comment|https://issues.apache.org/jira/browse/STORM-1757?focusedCommentId=15282971=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15282971].
 This should include checkpointing the panes. I plan to work next few days on 
this and add initial thoughts in STORM-1763.

Windowing can take the applied functions like reducer/aggregator etc and make 
appropriate optimizations in maintaining the pane state. Panes with 
accumulating eviction policies can have optimizations for reducer/aggregator 
without the need to store all the tuples in the pane state but by storing only 
the result value with required context. For ex, Avg function needs to store 
only the avg value and no of elements, these are sufficient in computing the 
accumulated value for the next pane. We can explore ways to optimize wherever 
it is possible.



> Apache Beam Runner for Storm
> 
>
> Key: STORM-1757
> URL: https://issues.apache.org/jira/browse/STORM-1757
> Project: Apache Storm
>  Issue Type: Brainstorming
>Reporter: P. Taylor Goetz
>Priority: Minor
>
> This is a call for interested parties to collaborate on an Apache Beam [1] 
> runner for Storm, and express their thoughts and opinions.
> Given the addition of the Windowing API to Apache Storm, we should be able to 
> map naturally to the Beam API. If not, it may be indicative of shortcomings 
> of the Storm API that should be addressed.
> [1] http://beam.incubator.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286594#comment-15286594
 ] 

Jungtaek Lim commented on STORM-1842:
-

[~barrywhart] 
Here's custom build of Storm 1.0.2 SNAPSHOT + STORM-1842 patch.
http://people.apache.org/~kabhwan/storm-core-1.0.2-SNAPSHOT.jar

If it incurs other issues please let me know that I just apply STORM-1842 with 
Storm 1.0.1 and rebuild.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: FluxShellBolt/FluxShellSpout can now emit into...

[Darkless012]

GitHub user Darkless012 opened a pull request:

https://github.com/apache/storm/pull/1426

FluxShellBolt/FluxShellSpout can now emit into named streams + YAML DSL 
example

I need ability for FluxShellBolt/FluxShellSpout to emit into named streams.

This pull request maintains backward compatibility, while adding 2 new 
methods to set stream output.

In Flux YAML DLS the named streams can be created either the old way 
(default stream only) or by using the new methods:

```
bolts:
- className: org.apache.storm.flux.wrappers.bolts.FluxShellBolt
  id: my_bolt
  constructorArgs:
  - [python, my_bolt.py]
  configMethods:
  - name: setNamedStream
args:
- first
- [word, count]
```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Darkless012/storm master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1426.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1426


commit 25dd41ed1e03a02aeb82bea7bc7e3f11da3867a6
Author: Pavel Grochal 
Date:   2016-05-17T13:11:29Z

FluxShellBolt can emit into custom stream.

commit 8a175701fa797d4bb4c90814e49f7dbac287caa3
Author: Pavel Grochal 
Date:   2016-05-17T13:12:13Z

Fixed imports

commit d97cab9118268b2af51cec77ae31ca6dc2a62220
Author: Pavel Grochal 
Date:   2016-05-17T13:17:50Z

FluxShellSpout can emit into streams.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1730) LocalCluster#shutdown() does not terminate all storm threads/thread pools.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286589#comment-15286589
 ] 

ASF GitHub Bot commented on STORM-1730:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1400#issuecomment-219713855
  
@fbyrne Sorry we don't have it explicitly for Java now, we only have 
Clojure style guide.
Part of the plan for Storm 2.0.0 is porting Clojure to Java. After that we 
would have style guide / formatter for Java.
For now let's see style of origin code and adopt ourselves.
What I pointed out is indentation difference between L74, L75, L79. Could 
you address to let indentation consistent?


> LocalCluster#shutdown() does not terminate all storm threads/thread pools.
> --
>
> Key: STORM-1730
> URL: https://issues.apache.org/jira/browse/STORM-1730
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows 7 x64
> Oracle Java 1.8.0 u92 x64
>Reporter: Fergus Byrne
>Assignee: Fergus Byrne
>Priority: Minor
> Attachments: Thread Pool '47' remaining..png, storm-shutdown-issue.zip
>
>
> When using the LocalCluster in test setup.  LocalCluster#shutdown() does not 
> shutdown all executor services it starts.  In my test case, there is a single 
> thread pool executor service that is not shutdown and not daemon.  This keeps 
> the jvm alive when it is expected to terminate.
> Please see attached test case.  In my example, thread pool 47 is not 
> shutdown.  Naming here is conditional on threading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: This closes #STORM-1730 #1730 LocalCluster#shu...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1400#issuecomment-219713855
  
@fbyrne Sorry we don't have it explicitly for Java now, we only have 
Clojure style guide.
Part of the plan for Storm 2.0.0 is porting Clojure to Java. After that we 
would have style guide / formatter for Java.
For now let's see style of origin code and adopt ourselves.
What I pointed out is indentation difference between L74, L75, L79. Could 
you address to let indentation consistent?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Barry Hart (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286584#comment-15286584
 ] 

Barry Hart commented on STORM-1842:
---

Done. STORM-1846

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1846) Need a smoke test for the generated Python code

[Barry Hart (JIRA)]

Barry Hart created STORM-1846:
-

 Summary: Need a smoke test for the generated Python code
 Key: STORM-1846
 URL: https://issues.apache.org/jira/browse/STORM-1846
 Project: Apache Storm
  Issue Type: Test
  Components: storm-core
Affects Versions: 1.0.0
Reporter: Barry Hart


This was inspired by STORM-1842. For that issue, simply attempting to import 
ttypes.py would've detected the error. I'm guessing there is no such test. It 
would be good to add one in order to help avoid problems like that in the 
future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286580#comment-15286580
 ] 

Jungtaek Lim commented on STORM-1842:
-

Yeah, good to have a new JIRA issue, and if you would contribute that I'd be 
really appreciated! 

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286563#comment-15286563
 ] 

ASF GitHub Bot commented on STORM-1842:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219709570
  
@barrywhart 
https://repository.apache.org/releases/ returns 503 now, which makes Travis 
CI failed.


> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 Forward references in storm.thrift ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219709570
  
@barrywhart 
https://repository.apache.org/releases/ returns 503 now, which makes Travis 
CI failed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Barry Hart (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286561#comment-15286561
 ] 

Barry Hart commented on STORM-1842:
---

More of a general question -- are there any automated tests for the generated 
Python code? Simply attempting to import ttypes.py would've detected this 
error. If there is no such test, adding one could help avoid problems like this 
in the future.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Barry Hart (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286549#comment-15286549
 ] 

Barry Hart commented on STORM-1842:
---

[~kabhwan]: Can you send me a custom package? I'm not familiar with building 
Storm from source.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286544#comment-15286544
 ] 

ASF GitHub Bot commented on STORM-1842:
---

Github user barrywhart commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219707172
  
The change looks good to me.

Are the build failures related to this change?

https://travis-ci.org/apache/storm/jobs/130816708#L1162

It looks like something unrelated:
```[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.2.1:process (default) 
on project storm-hive: Error resolving project artifact: Could not transfer 
artifact org.pentaho:pentaho-aggdesigner-algorithm:pom:5.1.3-jhyde from/to 
sonatype-apache (https://repository.apache.org/releases/): Connect to 
repository.apache.org:443 [repository.apache.org/140.211.11.112] failed: 
Connection timed out for project 
org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde -> [Help 1]```


> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 Forward references in storm.thrift ...

[barrywhart]

Github user barrywhart commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219707172
  
The change looks good to me.

Are the build failures related to this change?

https://travis-ci.org/apache/storm/jobs/130816708#L1162

It looks like something unrelated:
```[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-remote-resources-plugin:1.2.1:process (default) 
on project storm-hive: Error resolving project artifact: Could not transfer 
artifact org.pentaho:pentaho-aggdesigner-algorithm:pom:5.1.3-jhyde from/to 
sonatype-apache (https://repository.apache.org/releases/): Connect to 
repository.apache.org:443 [repository.apache.org/140.211.11.112] failed: 
Connection timed out for project 
org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde -> [Help 1]```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Jungtaek Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286491#comment-15286491
 ] 

Jungtaek Lim commented on STORM-1842:
-

[~barrywhart] [~CogumelosMaravilha]
I submitted pull requests for reordering structs to get rid of forward 
references.

While I'm not sure that breaks compatibility, you may want to build and package 
your own storm-core by applying this pull request and try to see it helps.

If you want to try testing but don't want to build your own, please ping to me 
so that I can pass custom package to you.

Thanks!

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim reassigned STORM-1842:
---

Assignee: Jungtaek Lim

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Assignee: Jungtaek Lim
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286483#comment-15286483
 ] 

ASF GitHub Bot commented on STORM-1842:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1425

STORM-1842 (1.0.x) Forward references in storm.thrift cause tooling issues

For master: #1423
For 1.x: #1424 

Please refer STORM-1842 for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842-1.0.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1425


commit 7bdb1fe92f85435e86c436669760ee8df1e1c7c6
Author: Jungtaek Lim 
Date:   2016-05-17T11:57:15Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286481#comment-15286481
 ] 

ASF GitHub Bot commented on STORM-1842:
---

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219698986
  
I'm not sure it breaks backward compatibility, so if anyone checks it 
doesn't break compatibility that would be great.


> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 (1.0.x) Forward references in storm...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1425

STORM-1842 (1.0.x) Forward references in storm.thrift cause tooling issues

For master: #1423
For 1.x: #1424 

Please refer STORM-1842 for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842-1.0.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1425.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1425


commit 7bdb1fe92f85435e86c436669760ee8df1e1c7c6
Author: Jungtaek Lim 
Date:   2016-05-17T11:57:15Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: STORM-1842 Forward references in storm.thrift ...

[HeartSaVioR]

Github user HeartSaVioR commented on the pull request:

https://github.com/apache/storm/pull/1423#issuecomment-219698986
  
I'm not sure it breaks backward compatibility, so if anyone checks it 
doesn't break compatibility that would be great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286476#comment-15286476
 ] 

ASF GitHub Bot commented on STORM-1842:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1424

STORM-1842 (1.x) Forward references in storm.thrift cause tooling issues

For master: #1423 

Please refer STORM-1842 for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1424


commit 4c7117d75f80b820da110c3abee88134ca6ecaaa
Author: Jungtaek Lim 
Date:   2016-05-17T11:45:03Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 (1.x) Forward references in storm.t...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1424

STORM-1842 (1.x) Forward references in storm.thrift cause tooling issues

For master: #1423 

Please refer STORM-1842 for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1424.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1424


commit 4c7117d75f80b820da110c3abee88134ca6ecaaa
Author: Jungtaek Lim 
Date:   2016-05-17T11:45:03Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286473#comment-15286473
 ] 

ASF GitHub Bot commented on STORM-1842:
---

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1423

STORM-1842 Forward references in storm.thrift cause tooling issues

Please refer [STORM-1842](https://issues.apache.org/jira/browse/STORM-1842) 
for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1423


commit ad46a470c173c8daf2bd76bc10ecbfc6fd08865b
Author: Jungtaek Lim 
Date:   2016-05-17T11:23:38Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1842 Forward references in storm.thrift ...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/1423

STORM-1842 Forward references in storm.thrift cause tooling issues

Please refer [STORM-1842](https://issues.apache.org/jira/browse/STORM-1842) 
for more details.

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-1842

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1423.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1423


commit ad46a470c173c8daf2bd76bc10ecbfc6fd08865b
Author: Jungtaek Lim 
Date:   2016-05-17T11:23:38Z

STORM-1842 Forward references in storm.thrift cause tooling issues

* python doesn't support forward reference so ttypes.py has a problem 
regarding this issue
* reorder HBPulse, HBRecords, HBNodes to resolve forward reference issue




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1845) use UTF-8 instead of default encoding

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286362#comment-15286362
 ] 

ASF GitHub Bot commented on STORM-1845:
---

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/1422#issuecomment-219677080
  
Hi  @vesense If I don't specified charset which one will be used ?



> use UTF-8 instead of default encoding
> -
>
> Key: STORM-1845
> URL: https://issues.apache.org/jira/browse/STORM-1845
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hbase, storm-hdfs, storm-hive, storm-redis
>Affects Versions: 1.0.1
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1845: use UTF-8 instead of default encod...

[darionyaphet]

Github user darionyaphet commented on the pull request:

https://github.com/apache/storm/pull/1422#issuecomment-219677080
  
Hi  @vesense If I don't specified charset which one will be used ?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1845) use UTF-8 instead of default encoding

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286352#comment-15286352
 ] 

ASF GitHub Bot commented on STORM-1845:
---

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/1422

STORM-1845: use UTF-8 instead of default encoding



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-1845

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1422


commit e8d20ca26976428021d38d8de16716c876801238
Author: Xin Wang 
Date:   2016-05-17T09:46:09Z

STORM-1845: use UTF-8 instead of default encoding




> use UTF-8 instead of default encoding
> -
>
> Key: STORM-1845
> URL: https://issues.apache.org/jira/browse/STORM-1845
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hbase, storm-hdfs, storm-hive, storm-redis
>Affects Versions: 1.0.1
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1845: use UTF-8 instead of default encod...

[vesense]

GitHub user vesense opened a pull request:

https://github.com/apache/storm/pull/1422

STORM-1845: use UTF-8 instead of default encoding



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vesense/storm STORM-1845

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1422.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1422


commit e8d20ca26976428021d38d8de16716c876801238
Author: Xin Wang 
Date:   2016-05-17T09:46:09Z

STORM-1845: use UTF-8 instead of default encoding




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1837) Running local clusters without simulating time breaks Testing.completeTopology, and may cause message loss

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286344#comment-15286344
 ] 

ASF GitHub Bot commented on STORM-1837:
---

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1417#issuecomment-219674715
  
Something like this? https://github.com/apache/storm/pull/1421


> Running local clusters without simulating time breaks 
> Testing.completeTopology, and may cause message loss
> --
>
> Key: STORM-1837
> URL: https://issues.apache.org/jira/browse/STORM-1837
> Project: Apache Storm
>  Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.0.1
>Reporter: Stig Rohde Døssing
>Priority: Minor
>
> Since https://github.com/apache/storm/pull/810 it is no longer possible to 
> call Testing.completeTopology when time is not simulating, because a call to 
> advance-cluster-time is made from the function, which calls Time/advanceTime. 
> advance-cluster-time should only be called if time is simulating.
> Since https://github.com/apache/storm/pull/830 a local cluster run without 
> time simulation may lose messages. When a worker emits messages for a worker 
> that hasn't started yet, the message is lost. This can happen because spouts 
> may start emitting before all workers have started, when time simulation is 
> disabled. Local clusters usually run without message timeouts, so this will 
> make tests relying on Testing.withLocalCluster flaky.
> The problem is that there are no longer any queues to store messages for 
> workers that haven't started yet. See 
> https://github.com/apache/storm/pull/830/files#diff-c6ff4208ef84c7a5a1a6b8b6bd1f7d19R104.
>  A queue should be added for messages for workers that haven't registered a 
> receive callback yet.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1837: Fix complete-topology and prevent ...

[srdo]

Github user srdo commented on the pull request:

https://github.com/apache/storm/pull/1417#issuecomment-219674715
  
Something like this? https://github.com/apache/storm/pull/1421


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (STORM-1845) use UTF-8 instead of default encoding

[Xin Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xin Wang updated STORM-1845:

Component/s: storm-redis

> use UTF-8 instead of default encoding
> -
>
> Key: STORM-1845
> URL: https://issues.apache.org/jira/browse/STORM-1845
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-hbase, storm-hdfs, storm-hive, storm-redis
>Affects Versions: 1.0.1
>Reporter: Xin Wang
>Assignee: Xin Wang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1842) Forward references in storm.thrift cause tooling issues

[Cogumelos Maravilha (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286313#comment-15286313
 ] 

Cogumelos Maravilha commented on STORM-1842:


I'm waiting for this to upgrade my Apache Storm.

> Forward references in storm.thrift cause tooling issues
> ---
>
> Key: STORM-1842
> URL: https://issues.apache.org/jira/browse/STORM-1842
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Barry Hart
>Priority: Critical
> Attachments: ttypes.py
>
>
> In recent versions of Storm (starting with version 1.0.0, I believe), 
> storm.thrift uses several types before they are declared:
> * HBPulse
> * HBRecords
> * HBNodes
> These types are used in the definition of HBMessageData.
> This causes issues with downstream tools. For example, generating Python 
> wrappers for the Storm Thrift types creates a module that will not import 
> successfully.
> Would it be possible to reorder this code to define the types before using 
> them? This appears to be a simple change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (STORM-1845) use UTF-8 instead of default encoding

[Xin Wang (JIRA)]

Xin Wang created STORM-1845:
---

 Summary: use UTF-8 instead of default encoding
 Key: STORM-1845
 URL: https://issues.apache.org/jira/browse/STORM-1845
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-hbase, storm-hdfs, storm-hive
Affects Versions: 1.0.1
Reporter: Xin Wang
Assignee: Xin Wang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1844) Some tests are flaky due to low timeout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286294#comment-15286294
 ] 

ASF GitHub Bot commented on STORM-1844:
---

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1421

STORM-1844: Make netty unit test use STORM_TEST_TIMEOUT_MS, increase …

…STORM_TEST_TIMEOUT_MS to 10 seconds, increase DisruptorQueueTest timeout 
to 5 seconds from 1

Link for convenience https://issues.apache.org/jira/browse/STORM-1844

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1844

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1421


commit 653465a163f382cdcfd1672a97b5280abc1465c2
Author: Stig Rohde Døssing 
Date:   2016-05-17T09:31:08Z

STORM-1844: Make netty unit test use STORM_TEST_TIMEOUT_MS, increase 
STORM_TEST_TIMEOUT_MS to 10 seconds, increase DisruptorQueueTest timeout to 5 
seconds from 1




> Some tests are flaky due to low timeout
> ---
>
> Key: STORM-1844
> URL: https://issues.apache.org/jira/browse/STORM-1844
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 2.0.0
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Minor
>
> We saw a few test failures likely due to low timeouts 
> https://github.com/apache/storm/pull/1417#issuecomment-219545865. The 
> timeouts for these tests should be increased, and the test mentioned by 
> [~kishorvpatil] should reflect STORM_TEST_TIMEOUT_MS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1844: Make netty unit test use STORM_TES...

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1421

STORM-1844: Make netty unit test use STORM_TEST_TIMEOUT_MS, increase …

…STORM_TEST_TIMEOUT_MS to 10 seconds, increase DisruptorQueueTest timeout 
to 5 seconds from 1

Link for convenience https://issues.apache.org/jira/browse/STORM-1844

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-1844

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1421.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1421


commit 653465a163f382cdcfd1672a97b5280abc1465c2
Author: Stig Rohde Døssing 
Date:   2016-05-17T09:31:08Z

STORM-1844: Make netty unit test use STORM_TEST_TIMEOUT_MS, increase 
STORM_TEST_TIMEOUT_MS to 10 seconds, increase DisruptorQueueTest timeout to 5 
seconds from 1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: remove unnecessary registerSerialization about...

[satishd]

Github user satishd commented on the pull request:

https://github.com/apache/storm/pull/1278#issuecomment-219667126
  
@hustfxj That was a good catch. We may need to keep 
`org.apache.storm.trident.topology.TransactionAttempt` registered  as it is 
serialized/deserialized over the wire in trident topologies. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (STORM-1844) Some tests are flaky due to low timeout

[JIRA]

Stig Rohde Døssing created STORM-1844:
-

 Summary: Some tests are flaky due to low timeout
 Key: STORM-1844
 URL: https://issues.apache.org/jira/browse/STORM-1844
 Project: Apache Storm
  Issue Type: Bug
  Components: storm-core
Affects Versions: 2.0.0
Reporter: Stig Rohde Døssing
Assignee: Stig Rohde Døssing
Priority: Minor


We saw a few test failures likely due to low timeouts 
https://github.com/apache/storm/pull/1417#issuecomment-219545865. The timeouts 
for these tests should be increased, and the test mentioned by [~kishorvpatil] 
should reflect STORM_TEST_TIMEOUT_MS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (STORM-1730) LocalCluster#shutdown() does not terminate all storm threads/thread pools.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286271#comment-15286271
 ] 

ASF GitHub Bot commented on STORM-1730:
---

Github user fbyrne commented on the pull request:

https://github.com/apache/storm/pull/1400#issuecomment-219662853
  
Sure, that's no problem.  I haven't been able to find the Java style guide 
on the contribute readme.

Can you point me to this or if there is an eclipse formatter configuration 
you can forward to me.



> LocalCluster#shutdown() does not terminate all storm threads/thread pools.
> --
>
> Key: STORM-1730
> URL: https://issues.apache.org/jira/browse/STORM-1730
> Project: Apache Storm
>  Issue Type: Bug
>  Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows 7 x64
> Oracle Java 1.8.0 u92 x64
>Reporter: Fergus Byrne
>Assignee: Fergus Byrne
>Priority: Minor
> Attachments: Thread Pool '47' remaining..png, storm-shutdown-issue.zip
>
>
> When using the LocalCluster in test setup.  LocalCluster#shutdown() does not 
> shutdown all executor services it starts.  In my test case, there is a single 
> thread pool executor service that is not shutdown and not daemon.  This keeps 
> the jvm alive when it is expected to terminate.
> Please see attached test case.  In my example, thread pool 47 is not 
> shutdown.  Naming here is conditional on threading.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: This closes #STORM-1730 #1730 LocalCluster#shu...

[fbyrne]

Github user fbyrne commented on the pull request:

https://github.com/apache/storm/pull/1400#issuecomment-219662853
  
Sure, that's no problem.  I haven't been able to find the Java style guide 
on the contribute readme.

Can you point me to this or if there is an eclipse formatter configuration 
you can forward to me.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request: remove unnecessary registerSerialization about...

[hustfxj]

Github user hustfxj commented on the pull request:

https://github.com/apache/storm/pull/1278#issuecomment-219647600
  
we register the class "org.apache.storm.transactional.TransactionAttempt" 
in SerializationFactory , But we register the class 
"org.apache.storm.trident.topology.TransactionAttempt" in 
MasterBatchCoordinator. So it will occur serialization errors when we run 
trident topology. Do you Think ? @revans2 @satishd  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (STORM-1745) Add partition to PartitionManager logs where it's missing

[Jungtaek Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/STORM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim resolved STORM-1745.
-
   Resolution: Fixed
Fix Version/s: 1.0.2
   2.0.0

Thanks [~Srdo], I merged into master, 1.x, 1.0.x branches.

> Add partition to PartitionManager logs where it's missing
> -
>
> Key: STORM-1745
> URL: https://issues.apache.org/jira/browse/STORM-1745
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Trivial
> Fix For: 2.0.0, 1.0.2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] storm pull request: STORM-1745: Add partition to log output in Par...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1380


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (STORM-1745) Add partition to PartitionManager logs where it's missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/STORM-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286153#comment-15286153
 ] 

ASF GitHub Bot commented on STORM-1745:
---

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/1380


> Add partition to PartitionManager logs where it's missing
> -
>
> Key: STORM-1745
> URL: https://issues.apache.org/jira/browse/STORM-1745
> Project: Apache Storm
>  Issue Type: Improvement
>  Components: storm-kafka
>Reporter: Stig Rohde Døssing
>Assignee: Stig Rohde Døssing
>Priority: Trivial
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)