[GitHub] storm pull request #2561: STORM-2958 - Use new wait strategy model for Spout...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2561


---

Re: [DISCUSS] Plan on Storm 2.0.0

[Jungtaek Lim]

I think that's good idea. "blockers" should have priority as blocker (so
they're self-classifiable), but may be better to classify "very desirable"
and "good to have".
Would we want to break down epic issue, or determine with other field
(priority, or label)?

2018년 2월 20일 (화) 오전 9:49, Roshan Naik 님이 작성:

> Would be good to callout which of issues listed in Storm 2.0 epic are
> considered “blockers” vs “very desirable” vs “good to have”
>
> -roshan
>
>
> On 2/19/18, 3:29 PM, "Jungtaek Lim"  wrote:
>
> Hi devs,
>
> We've just released Storm 1.2.1 (not officially announced but vote
> passed
> anyway) and we're in agreement that no further minor version on 1.x
> version
> line, so it's time to focus on Storm 2.0.0 so that we can bring far
> long
> undetermined milestone to reality.
>
> Here is the epic issue for Storm 2.0.0:
> https://issues.apache.org/jira/browse/STORM-2714
>
> We have no restriction to file issues and submit patches so there're
> more
> changes available outside of epic issue, but once we reach consensus to
> bring Storm 2.0.0 fairly sooner, I propose we (team, or individual)
> add all
> the issues to epic which we target to include to Storm 2.0.0, so that
> we
> can track all the remaining items from the epic issue, and decide to
> postpone some non-blocker dragging items to out of Storm 2.0.0. Makes
> sense?
>
> I have some backlog issues for myself which would need somewhat huge
> efforts so not sure they can be included to Storm 2.0.0. I'll add some
> to
> epic if I think I can find time to do. If you have items which are
> planned
> to be included to Storm 2.0.0, please add them to epic issue.
>
> I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
> left): even if it doesn't happen, we could release beta version of
> Storm
> 2.0.0 in 1Q and we say "feature freeze" and concentrate on stabilizing
> the
> release and making Storm 2.0.0 happen sooner than later. If you have
> items
> for Storm 2.0.0 which requires more than 1 month to be finished, I
> think it
> would be worth to share the items and discuss.
>
> Please add your voice on anything about Storm 2.0.0 plan. It would be
> much
> appreciated if someone puts efforts on testing and stabilizing the
> current
> master branch (it would take much time and efforts and more hands are
> definitely better).
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
>

Verification about my observation for current State implementation

[Jungtaek Lim]

Hi,

I'd like to verify my observation on current State implementation is
correct, so that we could fix them if necessary and make plan for
improvement.

1. State is stored with namespace prefix which typically composes to
(component id, task id) pair and it doesn't look like having classification
for topology. Is this correct observation? If then I think that's worth to
call it as 'critical' and it must be fixed.

2. We're allowing end-users to put key of state, and also no restriction
for grouping on stateful component. I feel such flexibility breaks the
possibility to reshard state and end-users are required to implement their
own reshard tool according to their topology state key distribution logic.
I expect it will not happen on streams API (since it should be done with
keyed stream) but wouldn't it better to also restrict such flexibility also
for core API?

3. Suppose we are going to support state resharding (for allowing change of
parallelism) and we restrict to apply field grouping with key while
connecting stateful component.
Then key-value can be moved based on key (though finding and replacing task
id may not be trivial if component name has '-'... we have same issue on
metric name, so maybe time to restrict characters on topology name as well
as component name?).
Is it also true for window/partition/windowsystem state? I didn't take a
deep look on window state (I would find a time) but it would be great if
someone knowing the detail makes it clear.

Thanks in advance,
Jungtaek Lim (HeartSaVioR)

[ANNOUNCE] Apache Storm 1.2.1 Released

[P. Taylor Goetz]

The Apache Storm community is pleased to announce the release of Apache Storm 
version 1.2.1.

Storm is a distributed, fault-tolerant, and high-performance realtime 
computation system that provides strong guarantees on the processing of data. 
You can read more about Storm on the project website:

http://storm.apache.org

Downloads of source and binary distributions are listed in our download
section:

http://storm.apache.org/downloads.html

You can read more about this release in the following blog post:

http://storm.apache.org/2018/02/19/storm121-released.html

Distribution artifacts are available in Maven Central at the following 
coordinates:

groupId: org.apache.storm
artifactId: storm-core
version: 1.2.1

The full list of changes is available here[1]. Please let us know [2] if you 
encounter any problems.

Regards,

The Apache Storm Team

[1]: http://www.us.apache.org/dist/storm/apache-storm-1.2.1/RELEASE_NOTES.html
[2]: https://issues.apache.org/jira/browse/STORM

Re: [DISCUSS] Plan on Storm 2.0.0

[Roshan Naik]

Would be good to callout which of issues listed in Storm 2.0 epic are 
considered “blockers” vs “very desirable” vs “good to have”

-roshan


On 2/19/18, 3:29 PM, "Jungtaek Lim"  wrote:

Hi devs,

We've just released Storm 1.2.1 (not officially announced but vote passed
anyway) and we're in agreement that no further minor version on 1.x version
line, so it's time to focus on Storm 2.0.0 so that we can bring far long
undetermined milestone to reality.

Here is the epic issue for Storm 2.0.0:
https://issues.apache.org/jira/browse/STORM-2714

We have no restriction to file issues and submit patches so there're more
changes available outside of epic issue, but once we reach consensus to
bring Storm 2.0.0 fairly sooner, I propose we (team, or individual) add all
the issues to epic which we target to include to Storm 2.0.0, so that we
can track all the remaining items from the epic issue, and decide to
postpone some non-blocker dragging items to out of Storm 2.0.0. Makes sense?

I have some backlog issues for myself which would need somewhat huge
efforts so not sure they can be included to Storm 2.0.0. I'll add some to
epic if I think I can find time to do. If you have items which are planned
to be included to Storm 2.0.0, please add them to epic issue.

I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
left): even if it doesn't happen, we could release beta version of Storm
2.0.0 in 1Q and we say "feature freeze" and concentrate on stabilizing the
release and making Storm 2.0.0 happen sooner than later. If you have items
for Storm 2.0.0 which requires more than 1 month to be finished, I think it
would be worth to share the items and discuss.

Please add your voice on anything about Storm 2.0.0 plan. It would be much
appreciated if someone puts efforts on testing and stabilizing the current
master branch (it would take much time and efforts and more hands are
definitely better).

Thanks,
Jungtaek Lim (HeartSaVioR)

[GitHub] storm issue #2561: STORM-2958 - Use new wait strategy model for Spout as wel...

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2561
  
thanks for catching that. fixed it.


---

[ANNOUNCE] Apache Storm 1.2.1 Released

[P. Taylor Goetz]

The Apache Storm community is pleased to announce the release of Apache Storm 
version 1.2.1.

Storm is a distributed, fault-tolerant, and high-performance realtime 
computation system that provides strong guarantees on the processing of data. 
You can read more about Storm on the project website:

http://storm.apache.org

Downloads of source and binary distributions are listed in our download
section:

http://storm.apache.org/downloads.html

You can read more about this release in the following blog post:

http://storm.apache.org/2018/02/19/storm121-released.html

Distribution artifacts are available in Maven Central at the following 
coordinates:

groupId: org.apache.storm
artifactId: storm-core
version: 1.2.1

The full list of changes is available here[1]. Please let us know [2] if you 
encounter any problems.

Regards,

The Apache Storm Team

[1]: http://www.us.apache.org/dist/storm/apache-storm-1.2.1/RELEASE_NOTES.html
[2]: https://issues.apache.org/jira/browse/STORM

[DISCUSS] Plan on Storm 2.0.0

[Jungtaek Lim]

Hi devs,

We've just released Storm 1.2.1 (not officially announced but vote passed
anyway) and we're in agreement that no further minor version on 1.x version
line, so it's time to focus on Storm 2.0.0 so that we can bring far long
undetermined milestone to reality.

Here is the epic issue for Storm 2.0.0:
https://issues.apache.org/jira/browse/STORM-2714

We have no restriction to file issues and submit patches so there're more
changes available outside of epic issue, but once we reach consensus to
bring Storm 2.0.0 fairly sooner, I propose we (team, or individual) add all
the issues to epic which we target to include to Storm 2.0.0, so that we
can track all the remaining items from the epic issue, and decide to
postpone some non-blocker dragging items to out of Storm 2.0.0. Makes sense?

I have some backlog issues for myself which would need somewhat huge
efforts so not sure they can be included to Storm 2.0.0. I'll add some to
epic if I think I can find time to do. If you have items which are planned
to be included to Storm 2.0.0, please add them to epic issue.

I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
left): even if it doesn't happen, we could release beta version of Storm
2.0.0 in 1Q and we say "feature freeze" and concentrate on stabilizing the
release and making Storm 2.0.0 happen sooner than later. If you have items
for Storm 2.0.0 which requires more than 1 month to be finished, I think it
would be worth to share the items and discuss.

Please add your voice on anything about Storm 2.0.0 plan. It would be much
appreciated if someone puts efforts on testing and stabilizing the current
master branch (it would take much time and efforts and more hands are
definitely better).

Thanks,
Jungtaek Lim (HeartSaVioR)

[GitHub] storm issue #2563: [STORM-2961] Refactor addResource and addResources Method...

[kishorvpatil]

Github user kishorvpatil commented on the issue:

https://github.com/apache/storm/pull/2563
  
@HeartSaVioR Thank you, I fixed the indentation.


---

[GitHub] storm pull request #2560: STORM-2896: Add tool to help users migrate offsets...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2560#discussion_r169182620
  
--- Diff: 
external/storm-kafka-migration/src/main/java/org/apache/storm/kafka/migration/KafkaTridentSpoutMigration.java
 ---
@@ -69,24 +70,40 @@ public String toString() {
 }
 }
 
+/**
+ * Get value for key. Error if value is null or not the expected type.
+ */
+public static  T getOrError(Map conf, String key) {
--- End diff --

Not a big deal (because the storm-kafka-migration module is just a 
utility), but getOrError() looks like better to place to separate class to 
decouple two independent classes.  Just a 2 cents and I'm OK even we keep it as 
it is.


---

[GitHub] storm pull request #2565: STORM-2954 add HBase metricstore implementation

[agresch]

GitHub user agresch opened a pull request:

https://github.com/apache/storm/pull/2565

STORM-2954 add HBase metricstore implementation

This provides an HBase metricstore implementation to optionally replace the 
default RocksDB version.  It allows both Nimbus and Supervisors to insert 
metrics directly into HBase.

The code should allow for multiple hosts/threads per host to insert into 
HBase.  Where new metadata or aggregated metrics are added, HBase checkandput 
calls are made with retries to make sure the latest data is fetched and stored.

The docs provide info on the configuration options, schema, and how to 
create a table for usage.

I ran my unit tests against HBase 1.1.12, and tested on a secure 1.3 (with 
modified internal code) HBase setup.




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agresch/storm agresch_hbase_metricstore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2565.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2565


commit 24cfa6a11eeebbb8105294ed14c2d2e4cf4dd482
Author: Aaron Gresch 
Date:   2018-02-19T22:16:45Z

STORM-2954 add HBase metricstore implementation




---

[GitHub] storm pull request #2560: STORM-2896: Add tool to help users migrate offsets...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2560#discussion_r169182743
  
--- Diff: 
external/storm-kafka-migration/src/main/java/org/apache/storm/kafka/migration/KafkaSpoutMigration.java
 ---
@@ -54,25 +55,30 @@
 private int zkRetryIntervalMs;
 }
 
+/**
+ * Migrates offsets from the Zookeeper store used by the storm-kafka 
non-Trident spouts, to Kafka's offset store used by the
+ * storm-kafka-client non-Trident spout.
+ */
 public static void main(String[] args) throws Exception {
 if (args.length != 1) {
 System.err.println("Args: confFile");
+return;
--- End diff --

You may want to exit with explicit return code: it will help scripts which 
leverages this module. Same for trident migration.


---

[GitHub] storm pull request #2563: [STORM-2961] Refactor addResource and addResources...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2563#discussion_r169180931
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java
 ---
@@ -779,30 +769,20 @@ public BoltDeclarer addConfigurations(Map conf) {
 return this;
 }
 
+/**
+ * return the current component configuration.
+ *
+ * @return the current configuration.
+ */
 @Override
-public BoltDeclarer addResources(Map resources) {
-if (resources != null) {
-Map currentResources = (Map) component.componentConf.computeIfAbsent(
-Config.TOPOLOGY_COMPONENT_RESOURCES_MAP, (k) -> new 
HashMap<>());
-currentResources.putAll(resources);
-}
-return this;
+public Map getComponentConfiguration() {
+return component.componentConf;
 }
 
-@Override
+   @Override
--- End diff --

nit: broken indent


---

[GitHub] storm issue #2562: STORM-2960 Stress importance of setting up proper OS acco...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2562
  
+1


---

[GitHub] storm pull request #2562: STORM-2960 Stress importance of setting up proper ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2562#discussion_r169179737
  
--- Diff: docs/SECURITY.md ---
@@ -17,6 +17,9 @@ Authentication and Authorization. But to do so usually 
requires
 configuring your Operating System to restrict the operations that can be 
done.
 This is generally a good idea even if you plan on running your cluster 
with Auth.
 
+Meaning to say, Storm's OS level security is based on running Storm 
processes with proper OS account,
--- End diff --

Updated.


---

[GitHub] storm pull request #2562: STORM-2960 Stress importance of setting up proper ...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2562#discussion_r169179563
  
--- Diff: docs/SECURITY.md ---
@@ -17,6 +17,9 @@ Authentication and Authorization. But to do so usually 
requires
 configuring your Operating System to restrict the operations that can be 
done.
 This is generally a good idea even if you plan on running your cluster 
with Auth.
 
+Meaning to say, Storm's OS level security is based on running Storm 
processes with proper OS account,
--- End diff --

I think that's clearer. Thanks for suggestion.


---

[GitHub] storm pull request #2564: STORM-2896 master

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/2564

STORM-2896 master

Master version of 
https://github.com/apache/storm/compare/master...srdo:STORM-2896-master?expand=1

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-2896-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2564.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2564


commit fe19434065bd9b5ac2e1049e6a266e9e94acee37
Author: Stig Rohde Døssing 
Date:   2018-02-17T21:05:30Z

STORM-2896: Add tool to help users migrate offsets from storm-kafka to 
storm-kafka-client

commit d565bba319a88669e1677d65920482befd88ab72
Author: Stig Rohde Døssing 
Date:   2018-02-19T18:41:10Z

Make a few improvements while porting to master

commit ace1936ac02fae5753ca0f01f2e57b5ca3306a7a
Author: Stig Rohde Døssing 
Date:   2018-02-19T18:46:18Z

Port to master




---

[GitHub] storm issue #2560: STORM-2896: Add tool to help users migrate offsets from s...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2560
  
Thanks, will make a master branch version.


---

[GitHub] storm pull request #2562: STORM-2960 Stress importance of setting up proper ...

[srdo]

Github user srdo commented on a diff in the pull request:

https://github.com/apache/storm/pull/2562#discussion_r16911
  
--- Diff: docs/SECURITY.md ---
@@ -17,6 +17,9 @@ Authentication and Authorization. But to do so usually 
requires
 configuring your Operating System to restrict the operations that can be 
done.
 This is generally a good idea even if you plan on running your cluster 
with Auth.
 
+Meaning to say, Storm's OS level security is based on running Storm 
processes with proper OS account,
--- End diff --

I think this is a little hard to understand. What do you think about 
"Storm's OS level security relies on running Storm processes using OS accounts 
that have only the permissions they need. Note that workers run under the same 
OS account as the Supervisor daemon by default"?


---

[GitHub] storm pull request #2563: [STORM-2961] Refactor addResource and addResources...

[kishorvpatil]

GitHub user kishorvpatil opened a pull request:

https://github.com/apache/storm/pull/2563

[STORM-2961] Refactor addResource and addResources Methods in 
BaseConfigurationDeclarer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kishorvpatil/incubator-storm refactorDeclarers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2563.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2563


commit 0267d549a96c1e7aefa3cb5d23e1cca559dd8e74
Author: Kishor Patil 
Date:   2018-02-17T00:23:17Z

Refactor addResource and addResources Method and avoid NPEs




---

[GitHub] storm pull request #2562: STORM-2960 Stress importance of setting up proper ...

[HeartSaVioR]

GitHub user HeartSaVioR opened a pull request:

https://github.com/apache/storm/pull/2562

STORM-2960 Stress importance of setting up proper OS account for Storm 
processes

While the explanation of last statement implicitly says that setting up 
proper OS account is needed, but explicitly mentioning it should help end-users 
to understand the importance while setting up OS account for Storm cluster.

This change should be clean cherry-pick for 1.x version lines.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HeartSaVioR/storm STORM-2960

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2562.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2562


commit 482e94559ed81620cdca3885163dcd1565f85c4e
Author: Jungtaek Lim 
Date:   2018-02-19T11:31:36Z

STORM-2960 Stress importance of setting up proper OS account for Storm 
processes




---

[GitHub] storm issue #2561: STORM-2958 - Use new wait strategy model for Spout as wel...

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2561
  
@roshannaik 
You may want to update `Performance.md` as well in this patch, since it 
explains old spout wait strategy.


---