Problem on ThriftAccessLogger logs on localhost mode #storm 1.0.2

[tsantalos-spamia]

There is a problem an you cannot set the logging level on localhost mode ehen 
using DRPC. It constantly sends logging messages such as "access from:  
principal:  operation: fetchRequest" that cannot be deactivated, making 
debugging impossible...

[GitHub] storm issue #1914: STORM-2334 - Join Bolt implementation with unit tests

[arunmahadevan]

Github user arunmahadevan commented on the issue:

https://github.com/apache/storm/pull/1914
  
+1, will wait for a day to see if there are any comments from other 
reviewers and will merge this after that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1914: STORM-2334 - Join Bolt implementation with unit tests

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/1914
  
@arunmahadevan ... PR has been revised. thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [Discuss] Storm hdfs spout improvements

[Arun Mahadevan]

Can you please raise a pull request with your proposal? That way it will be 
easier to review and comment.

Thanks,
Arun


On 2/15/17, 9:04 AM, "Sachin Pasalkar"  wrote:

>Can any one take a look at this? I have attached my code in JIRA.
>
>On 14/02/17, 7:38 AM, "Sachin Pasalkar" 
>wrote:
>
>>I have created JIRA for this
>>https://issues.apache.org/jira/browse/STORM-2358.
>>For point 1:
>>
>>Its specific use case just to support why it needs to be public
>>
>>For point 2:
>>We are limiting code to be very specific to these 2 implementations we
>>should have generic implementation. I see there is another check-in
>>happening for ZippedTextFileReader.I have attached my code changes in
>>JIRA, please take a look, where you need to provide class.
>>
>>For point 3:
>>
>>Lets assume I have multiple topologies with different readers. So I
>>defined the a base topology class with HDFSSpout in it. Now I always needs
>>to pass the outputFields as separate array. This actually can be part of
>>every reader class as its very specific to it.
>>
>>Thanks,
>>Sachin
>>
>>On 14/02/17, 4:52 AM, "Roshan Naik"  wrote:
>>
>>>
>>>
>>>On 2/13/17, 12:14 PM, "Sachin Pasalkar" 
>>>wrote:
>>>
I have attached updated source code of HDFSSpout for more reference. I
have updated respective classes (not attached)
>>>
>>>
>>>Don¹t see any attachment. Answers are below. Better to do this discussion
>>>on a JIRA.
>>>
>>>
>>>On 2/13/17, 8:32 AM, "Sachin Pasalkar" 
>>>wrote:
>>>
Hi,

I was looking at storm hdfs spout code in 1.x branch, I found below
improvements can be made in below code.

  1.  Make org.apache.storm.hdfs.spout.AbstractFileReader as public so
that it can be used in generics.
>>>
>>>Java generics and making a class public are unrelated to my knowledge.
>>>But
>>>making it public sounds ok to me if its useful for "user defined² readers
>>>Š although it doesn¹t really have that much going on in it. For future
>>>built-in reader types it is immaterial as they can derive from it anyway
>>>just like the existing ones. HdfsSpout class itself doesn¹t care about
>>>the
>>>ŒAbstractFileReader¹ type. For that there is the ŒFileReader¹ interface.
>>>
>>>
>>>
  2.  org.apache.storm.hdfs.spout.HdfsSpout requires readerType as
String. It will be great to have class
readerType; So we will not use Class.forName at multiple places also it
will help in below point.
>>>
>>>The reason it is a string, is that, for built-in readers,  we wanted to
>>>support Œshort aliases¹ like Œtext¹ and Œseq¹ instead of FQCN..
>>>
>>>
  3.  HdfsSpout also needs to provide outFields which are declared as
constants in each reader(e.g.SequenceFileReader). We can have abstract
API AbstractFileReader in which return them to user to make it generic.
>>>
>>>
>>>These consts can¹t go into the AbstractFileReader as they are reader
>>>specific.
>>>
>>>They are there just for convenience.  Users can call withOutputFields()
>>>on
>>>the spout and set it to these predefined names or anything else.
>>>
>>>
>>>-Roshan
>>>
>>
>

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[pasalkarsachin1]

Github user pasalkarsachin1 commented on the issue:

https://github.com/apache/storm/pull/1939
  
I haven't changed anything it was I guess previous one had style. You can 
ignore white space by appending ?w=1 to url. e.g. 
https://github.com/apache/storm/pull/1939/files?w=1 This shows only updated code


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1939: STORM-1363: TridentKafkaState should handle null values f...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/1939
  
@pasalkarsachin1 Could you preserve the style? I'm sorry we don't have 
clear style guide but most of changeset is from spaces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] storm pull request #1939: STORM-1363: TridentKafkaState should handle null v...

[Sachin Pasalkar]

Please review if possible. Its very small change.

On 14/02/17, 3:58 PM, "pasalkarsachin1"  wrote:

>GitHub user pasalkarsachin1 opened a pull request:
>
>
>https://clicktime.symantec.com/a/1/DBEpMKi7S8ttDWQWBLAX77jjfwHUyy4Njl5cZ3H
>b8zI=?d=hjW3hU4fseqlJeekQsKUtQDmn-71nGoCmEGzTXJ7cVpl3Xhyc3GnsK37akkU6NuBPl
>pqnC9IyIIr2cmRHSEuxFgBZDez_zHyWRlstrsFpNFXaxWFVlmrYqwSUBKCDOIRL691ETp_6G_y
>axpHEDvjKXpDAKpQnQLZm6Sv6an3fYB4D61QZST5BCQBfxAlTFVaY8cliJ2lpd6g7m7-hHc9Me
>P6eGE_UjLIwX0g1QtRnEPoy7utwfYRsHl_iH5eOtjExULV8tKKFnZGnZI-q2ZQDnQcRZ-LsJfO
>t1NT_pkPjRGreRpCBQzNQ7wksgkYGpRhe_WYP3KMFdofrq19DmK0R5VsAPum4BjxhharZ2gziV
>WGO2PM_Dc63s4xX0p1eoIIlOWXI6learycQjJ1NKS0E0KDJ7PhcBAPzNjjZHGxLFuTnltKJ-bE
>7W59hXKaBHdzlHIUcuvC3uH_CMhgRyB3ohF6_ulMFFP4wngS=https%3A%2F%2Fgithub.co
>m%2Fapache%2Fstorm%2Fpull%2F1939
>
>STORM-1363: TridentKafkaState should handle null values from
>TridentTupleToKafkaMapper.getMessageFromTuple()
>
>In case null value comes from the mapper it will print warning
>messages.
>Added log to print the time taken to emit number of messages.
>
>You can merge this pull request into a Git repository by running:
>
>$ git pull 
>https://clicktime.symantec.com/a/1/SernQA8KSlmNZQjIPv0ZkiF9eDlGpTWJdhaUYHo
>t3DY=?d=hjW3hU4fseqlJeekQsKUtQDmn-71nGoCmEGzTXJ7cVpl3Xhyc3GnsK37akkU6NuBPl
>pqnC9IyIIr2cmRHSEuxFgBZDez_zHyWRlstrsFpNFXaxWFVlmrYqwSUBKCDOIRL691ETp_6G_y
>axpHEDvjKXpDAKpQnQLZm6Sv6an3fYB4D61QZST5BCQBfxAlTFVaY8cliJ2lpd6g7m7-hHc9Me
>P6eGE_UjLIwX0g1QtRnEPoy7utwfYRsHl_iH5eOtjExULV8tKKFnZGnZI-q2ZQDnQcRZ-LsJfO
>t1NT_pkPjRGreRpCBQzNQ7wksgkYGpRhe_WYP3KMFdofrq19DmK0R5VsAPum4BjxhharZ2gziV
>WGO2PM_Dc63s4xX0p1eoIIlOWXI6learycQjJ1NKS0E0KDJ7PhcBAPzNjjZHGxLFuTnltKJ-bE
>7W59hXKaBHdzlHIUcuvC3uH_CMhgRyB3ohF6_ulMFFP4wngS=https%3A%2F%2Fgithub.co
>m%2Fpasalkarsachin1%2Fstorm STORM-1363
>
>Alternatively you can review and apply these changes as the patch at:
>
>
>https://clicktime.symantec.com/a/1/xOUqJRyJnT_tV6kmFKnezykmhm7Nlr271eJ8nnz
>K7Lo=?d=hjW3hU4fseqlJeekQsKUtQDmn-71nGoCmEGzTXJ7cVpl3Xhyc3GnsK37akkU6NuBPl
>pqnC9IyIIr2cmRHSEuxFgBZDez_zHyWRlstrsFpNFXaxWFVlmrYqwSUBKCDOIRL691ETp_6G_y
>axpHEDvjKXpDAKpQnQLZm6Sv6an3fYB4D61QZST5BCQBfxAlTFVaY8cliJ2lpd6g7m7-hHc9Me
>P6eGE_UjLIwX0g1QtRnEPoy7utwfYRsHl_iH5eOtjExULV8tKKFnZGnZI-q2ZQDnQcRZ-LsJfO
>t1NT_pkPjRGreRpCBQzNQ7wksgkYGpRhe_WYP3KMFdofrq19DmK0R5VsAPum4BjxhharZ2gziV
>WGO2PM_Dc63s4xX0p1eoIIlOWXI6learycQjJ1NKS0E0KDJ7PhcBAPzNjjZHGxLFuTnltKJ-bE
>7W59hXKaBHdzlHIUcuvC3uH_CMhgRyB3ohF6_ulMFFP4wngS=https%3A%2F%2Fgithub.co
>m%2Fapache%2Fstorm%2Fpull%2F1939.patch
>
>To close this pull request, make a commit to your master/trunk branch
>with (at least) the following in the commit message:
>
>This closes #1939
>
>
>commit f78760c7ed94b14312d0eae1c7b5688c7eb4e96d
>Author: Sachin Pasalkar 
>Date:   2017-02-14T10:24:23Z
>
>STORM-1363: TridentKafkaState should handle null values from
>TridentTupleToKafkaMapper.getMessageFromTuple()
>
>Incase null value comes from the mapper it will print warning
>messages also added the time take to emit number od messages in logs
>
>
>
>
>---
>If your project is set up for it, you can reply to this email and have
>your
>reply appear on GitHub as well. If your project does not have this feature
>enabled and wishes so, or if the feature is enabled but not working,
>please
>contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
>with INFRA.
>---

Re: [Discuss] Storm hdfs spout improvements

[Sachin Pasalkar]

Can any one take a look at this? I have attached my code in JIRA.

On 14/02/17, 7:38 AM, "Sachin Pasalkar" 
wrote:

>I have created JIRA for this
>https://issues.apache.org/jira/browse/STORM-2358.
>For point 1:
>
>Its specific use case just to support why it needs to be public
>
>For point 2:
>We are limiting code to be very specific to these 2 implementations we
>should have generic implementation. I see there is another check-in
>happening for ZippedTextFileReader.I have attached my code changes in
>JIRA, please take a look, where you need to provide class.
>
>For point 3:
>
>Lets assume I have multiple topologies with different readers. So I
>defined the a base topology class with HDFSSpout in it. Now I always needs
>to pass the outputFields as separate array. This actually can be part of
>every reader class as its very specific to it.
>
>Thanks,
>Sachin
>
>On 14/02/17, 4:52 AM, "Roshan Naik"  wrote:
>
>>
>>
>>On 2/13/17, 12:14 PM, "Sachin Pasalkar" 
>>wrote:
>>
>>>I have attached updated source code of HDFSSpout for more reference. I
>>>have updated respective classes (not attached)
>>
>>
>>Don¹t see any attachment. Answers are below. Better to do this discussion
>>on a JIRA.
>>
>>
>>On 2/13/17, 8:32 AM, "Sachin Pasalkar" 
>>wrote:
>>
>>>Hi,
>>>
>>>I was looking at storm hdfs spout code in 1.x branch, I found below
>>>improvements can be made in below code.
>>>
>>>  1.  Make org.apache.storm.hdfs.spout.AbstractFileReader as public so
>>>that it can be used in generics.
>>
>>Java generics and making a class public are unrelated to my knowledge.
>>But
>>making it public sounds ok to me if its useful for "user defined² readers
>>Š although it doesn¹t really have that much going on in it. For future
>>built-in reader types it is immaterial as they can derive from it anyway
>>just like the existing ones. HdfsSpout class itself doesn¹t care about
>>the
>>ŒAbstractFileReader¹ type. For that there is the ŒFileReader¹ interface.
>>
>>
>>
>>>  2.  org.apache.storm.hdfs.spout.HdfsSpout requires readerType as
>>>String. It will be great to have class
>>>readerType; So we will not use Class.forName at multiple places also it
>>>will help in below point.
>>
>>The reason it is a string, is that, for built-in readers,  we wanted to
>>support Œshort aliases¹ like Œtext¹ and Œseq¹ instead of FQCN..
>>
>>
>>>  3.  HdfsSpout also needs to provide outFields which are declared as
>>>constants in each reader(e.g.SequenceFileReader). We can have abstract
>>>API AbstractFileReader in which return them to user to make it generic.
>>
>>
>>These consts can¹t go into the AbstractFileReader as they are reader
>>specific.
>>
>>They are there just for convenience.  Users can call withOutputFields()
>>on
>>the spout and set it to these predefined names or anything else.
>>
>>
>>-Roshan
>>
>

Re: [ANNOUNCE] Apache Storm 1.0.3 Released

[P. Taylor Goetz]

GitHub releases are created every time you create a tag, for example during the 
release candidate process. They should not be considered official releases. 
Official releases are also always signed with the GPG signature of an Apache 
committer.

It’s always best to download releases from an official mirror.

-Taylor



> On Feb 14, 2017, at 4:50 PM, Andrew Xor  wrote:
> 
> from my understanding you can always fall back to github releases 
>  as well
> 
> Best,
> 
> A.
> 
> ​On Tue, Feb 14, 2017 at 9:44 PM, P. Taylor Goetz  > wrote:
> My apologies for sending this out early, not all web servers are in sync yet, 
> so you may get a 404 depending on which server you hit.
> 
> Direct link for download: 
> http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.3/ 
> 
> 
> Changelog: https://github.com/apache/storm/blob/v1.0.3/CHANGELOG.md 
> 
> 
> -Taylor
> 
> > On Feb 14, 2017, at 3:43 PM, P. Taylor Goetz  > > wrote:
> >
> > The Apache Storm community is pleased to announce the release of Apache 
> > Storm version 1.0.3.
> >
> > Storm is a distributed, fault-tolerant, and high-performance realtime 
> > computation system that provides strong guarantees on the processing of 
> > data. You can read more about Storm on the project website:
> >
> > http://storm.apache.org 
> >
> > Downloads of source and binary distributions are listed in our download
> > section:
> >
> > http://storm.apache.org/downloads.html 
> > 
> >
> > You can read more about this release in the following blog post:
> >
> > http://storm.apache.org/2017/02/14/storm103-released.html 
> > 
> >
> > Distribution artifacts are available in Maven Central at the following 
> > coordinates:
> >
> > groupId: org.apache.storm
> > artifactId: storm-core
> > version: 1.0.3
> >
> > The full list of changes is available here[1]. Please let us know [2] if 
> > you encounter any problems.
> >
> > Regards,
> >
> > The Apache Storm Team
> >
> > [1]: https://github.com/apache/storm/blob/v1.0.3/CHANGELOG.md 
> > 
> > [2]: https://issues.apache.org/jira/browse/STORM 
> > 
> 
> 

Re: [ANNOUNCE] Apache Storm 1.0.3 Released

[P. Taylor Goetz]

My apologies for sending this out early, not all web servers are in sync yet, 
so you may get a 404 depending on which server you hit.

Direct link for download: 
http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.0.3/

Changelog: https://github.com/apache/storm/blob/v1.0.3/CHANGELOG.md

-Taylor

> On Feb 14, 2017, at 3:43 PM, P. Taylor Goetz  wrote:
> 
> The Apache Storm community is pleased to announce the release of Apache Storm 
> version 1.0.3.
> 
> Storm is a distributed, fault-tolerant, and high-performance realtime 
> computation system that provides strong guarantees on the processing of data. 
> You can read more about Storm on the project website:
> 
> http://storm.apache.org
> 
> Downloads of source and binary distributions are listed in our download
> section:
> 
> http://storm.apache.org/downloads.html
> 
> You can read more about this release in the following blog post:
> 
> http://storm.apache.org/2017/02/14/storm103-released.html
> 
> Distribution artifacts are available in Maven Central at the following 
> coordinates:
> 
> groupId: org.apache.storm
> artifactId: storm-core
> version: 1.0.3
> 
> The full list of changes is available here[1]. Please let us know [2] if you 
> encounter any problems.
> 
> Regards,
> 
> The Apache Storm Team
> 
> [1]: https://github.com/apache/storm/blob/v1.0.3/CHANGELOG.md
> [2]: https://issues.apache.org/jira/browse/STORM

[GitHub] storm issue #1832: STORM-2250: Kafka Spout Refactoring to Increase Modularit...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/1832
  
Squashed this and pushed 1.x version at 
https://github.com/apache/storm/pull/1941. Thanks all for good feedback :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1941: STORM-2250: Kafka spout refactoring to increase mo...

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/1941

STORM-2250: Kafka spout refactoring to increase modularity and testab…

…ility. Also support nanoseconds in Storm time simulation

1.x version of https://github.com/apache/storm/pull/1832

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-2250-1.x

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1941.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1941






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[ANNOUNCE] Apache Storm 1.0.3 Released

[P. Taylor Goetz]

The Apache Storm community is pleased to announce the release of Apache Storm 
version 1.0.3.

Storm is a distributed, fault-tolerant, and high-performance realtime 
computation system that provides strong guarantees on the processing of data. 
You can read more about Storm on the project website:

http://storm.apache.org

Downloads of source and binary distributions are listed in our download
section:

http://storm.apache.org/downloads.html

You can read more about this release in the following blog post:

http://storm.apache.org/2017/02/14/storm103-released.html

Distribution artifacts are available in Maven Central at the following 
coordinates:

groupId: org.apache.storm
artifactId: storm-core
version: 1.0.3

The full list of changes is available here[1]. Please let us know [2] if you 
encounter any problems.

Regards,

The Apache Storm Team

[1]: https://github.com/apache/storm/blob/v1.0.3/CHANGELOG.md
[2]: https://issues.apache.org/jira/browse/STORM

[GitHub] storm issue #1832: STORM-2250: Kafka Spout Refactoring to Increase Modularit...

[hmcl]

Github user hmcl commented on the issue:

https://github.com/apache/storm/pull/1832
  
+1 for all the KafkaSpout related code
+1 also for the SimulatedTime changes in the sense that the minor 
refactoring of this class does not change the semantics of the pre-existing 
code.

@srdo can you please squash all the commits and create a PR for 1.x-branch, 
such that we can backport these changes. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1940: STORM-2361 Kafka spout - after leader change, it s...

[ernisv]

GitHub user ernisv opened a pull request:

https://github.com/apache/storm/pull/1940

STORM-2361 Kafka spout - after leader change, it stops committing offsets 
to ZK

Try to route acks/fails to recreated partition managers if manager is not 
found by exact (broker, partition, topic) combination

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ernisv/storm 
kafka-spout-leader-change-bug-2361

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1940.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1940


commit fe141fd79a0c7f1ec9a191359f9ccd6eb7a34f4b
Author: Ernestas Vaiciukevicius 
Date:   2017-02-14T15:41:02Z

Try to route acks/fails to recreated partition managers if manager is not 
found by exact (broker, partition, topic) combination




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1911: STORM-2336 Close Localizer and AsyncLocalizer when superv...

[tibkiss]

Github user tibkiss commented on the issue:

https://github.com/apache/storm/pull/1911
  
For future reference:
This issue has been committed with a typo in the issue id:
STORM-2236 was used as opposed to the correct STORM-2336

`64a9dd0 2017-02-01 Jungtaek Lim  STORM-2236 Close Localizer and 
AsyncLocalizer when supervisor is shutting down`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Sending periodic statistics to Spout from Bolts

[Bobby Evans]

That makes a lot of since.


- Bobby

On Monday, February 13, 2017, 5:02:53 PM CST, Anis Nasir  
wrote:Dear Bobby,

Thank you for the feedback. I will start looking at the source code now.

I would prefer the downstream operators to take care of these parameters
locally and only send a message to the upstream operator to *increase load*
or *decrease load.*

Given this feature, upstream operator will be able to react to each request
(i.e., *increase load* or *decrease load)*  rather than collecting all the
stats and solving the optimal assignment problem.

Therefore, I just need an interface to interact with upstream operator.

Regards,
Anis





On Tue, Feb 14, 2017 at 7:48 AM, Bobby Evans 
wrote:

> First off if you don't know clojure you are in luck. On the master branch
> all of the core code except for the UI, shell submission and a few classes
> needed to support them are in java.  There are still several tests that
> also need to move over to java but it should not be too big of an issue for
> you.
>
> q-length is fairly straight forward to collect.CPU utilization is hard
> Java does expose this through JMX on a per thread basis, so you might be
> able to get this for the executor thread of a given bolt/spout.memory
> utilization is on a per worker basis, but is fairly simple to get through
> JMXservice time vs idle time are things you will probably need to write
> yourself, but are probably not too difficult to do.  Be careful though this
> is on the data path and can impact the performance of all topologies.
>
>
> "on-need basis" is the hard part.  This is because the downstream
> components need to be able to know that an upstream component needs
> specific metrics.  I think the best way would be to broadcast it at a low
> frequency, but have thresholds where it would send it again if something
> changed drastically.
>
>
> - Bobby
>
> On Monday, February 13, 2017, 4:25:50 PM CST, Anis Nasir <
> aadi.a...@gmail.com> wrote:Dear Bobby,
>
> In this case, how can we enable such configuration?
>
> I am not very familiar with clojure. However, I would like the downstream
> operators to report various parameters on-need basis to the upstream
> operators, like service time, queue length, CPU utilization, memory
> utilization, idle time, etc.
>
> Regards,
> Anis
>
>
>
> On Tue, Feb 14, 2017 at 12:36 AM, Bobby Evans  >
> wrote:
>
> > Yes makes perfect since.
> >
> >
> > - Bobby
> >
> > On Friday, February 10, 2017, 4:36:22 PM CST, Anis Nasir <
> > aadi.a...@gmail.com> wrote:Dear Bobby,
> >
> > Thank you very much for your reply.
> >
> > In real deployments, it is often the case that executors are heterogenous
> > and execution time per tuple is non-uniform (as discussed in the JIRA).
> In
> > such cases, the workload and capacity (of executors) distributions are
> > often unknown at the upstream operator and it is required to infer the
> > capacity of each worker and the assigned workload.
> >
> > For such scenarios, I would like to design a grouping scheme that allows
> > upstream operators to change the assignments by knowing both the workload
> > and the capacities of the machine.
> >
> > Also, i would prefer that each downstream operator can send this message
> > on-need basis, rather than broadcasting it across the whole set of
> > operators.
> >
> > Does it makes sense?
> >
> > Regards,
> > Anis
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Feb 10, 2017 at 11:54 PM, Bobby Evans
>  > >
> > wrote:
> >
> > > Anis,
> > > We already have the q-length being reported up stream.
> > > https://issues.apache.org/jira/browse/STORM-162
> > > It works well, except when a topology gets really big the amount of
> > > metrics being collected can negatively impact the performance of the
> > > topology.  By really big I mean several thousand workers.
> > > There has also been a push to redo the metrics system in storm so it is
> > > more scalable and so that nimbus can query it.  That is what I
> personally
> > > think would be a good long term solution for features like elasticity.
> > But
> > > I am not really sure what you mean by load aware scheduling.
> > >
> > > - Bobby
> > >
> > > On Thursday, February 9, 2017, 10:34:29 PM CST, Anis Nasir <
> > > aadi.a...@gmail.com> wrote:Dear All,
> > >
> > > I have been trying to implement load aware scheduling for Apache Storm.
> > >
> > > For this purpose, I need to send periodic statistics from downstream
> > > operators to upstream operators.
> > >
> > > Is there a standard way of sending such statistics to upstream
> operator,
> > > e.g., a bolt periodically reporting it's local queue length to the
> > upstream
> > > spout.
> > >
> > > Thanking you in advance.
> > >
> > > Regards,
> > > Anis
> > >
> >
>

[GitHub] storm issue #1781: STORM-1369: Add MapState implementation to storm-cassandr...

[mkoch1]

Github user mkoch1 commented on the issue:

https://github.com/apache/storm/pull/1781
  
Rebase is done, sorry for the delay.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1939: STORM-1363: TridentKafkaState should handle null v...

[pasalkarsachin1]

GitHub user pasalkarsachin1 opened a pull request:

https://github.com/apache/storm/pull/1939

STORM-1363: TridentKafkaState should handle null values from 
TridentTupleToKafkaMapper.getMessageFromTuple()

In case null value comes from the mapper it will print warning messages.
Added log to print the time taken to emit number of messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pasalkarsachin1/storm STORM-1363

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1939.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1939


commit f78760c7ed94b14312d0eae1c7b5688c7eb4e96d
Author: Sachin Pasalkar 
Date:   2017-02-14T10:24:23Z

STORM-1363: TridentKafkaState should handle null values from 
TridentTupleToKafkaMapper.getMessageFromTuple()

Incase null value comes from the mapper it will print warning messages also 
added the time take to emit number od messages in logs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1938: STORM-2360: Storm-Hive: Thrift version mismatch wi...

[tibkiss]

GitHub user tibkiss opened a pull request:

https://github.com/apache/storm/pull/1938

STORM-2360: Storm-Hive: Thrift version mismatch with storm-core



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tibkiss/storm STORM-2360

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/1938.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1938


commit 28556a198e87c8f7c006699294216f78c54f9a7c
Author: Tibor Kiss 
Date:   2017-02-14T08:13:16Z

STORM-2360: Storm-Hive: Thrift version mismatch with storm-core




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---