[GitHub] storm issue #2502: new PR for STORM-2306 : Messaging subsystem redesign

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2502
  
I'm running performance test (TVL) to see there's any regression here. As 
@revans2 already did  performance test, I wouldn't spend the time going through 
too deeply (just couple of tests). I would provide +1 after things are going 
well.


---

[GitHub] storm pull request #2544: [STORM-2932] the naming of topology localityaware ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2544


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168057328
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

Done.


---

[GitHub] storm issue #2556: STORM-2946: Upgrade to HBase 2.0

[ptgoetz]

Github user ptgoetz commented on the issue:

https://github.com/apache/storm/pull/2556
  
> Did you test the patch manually with HBase 2.0 beta 1?

Not extensively. The point on this JIRA/PR is to raise awareness that we 
should look at targeting new ecosystem versions. This holds true for HDFS, 
Hive, etc.

> I guess there's little chance for HBase community to introduce API change 
after beta, but the possibility is still open. Moreover that's not ideal to 
depend on beta version. Do you intend to open this for the official release of 
HBase 2.0 and update the version/API change once it happens?
Does HBase 2 client be compatible with HBase 1.x? At a glance of HBase 2 
project doc 

I don't have much influence over the Apache HBase community, so this PR is 
dependent on what they do. It's an effort to align with where they are going.

> Does HBase 2 client be compatible with HBase 1.x? At a glance of HBase 2 
project doc 
(https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#)
 they claim admin interface is incompatible and HBase 1 client cannot 
administrate HBase 2. I'm wondering about opposite case (HBase 2 client to 
administrate HBase 1 server), and same case for other things. In short, I'd 
like to determine needs to have separate hbase module for HBase 1.x and HBase 
2.x.

Not that I'm aware of. I don't think this change would be 
backward-compatible, hence my comment about versioning.




---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168046999
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

yes please thats the intent.


---

[GitHub] storm issue #2556: STORM-2946: Upgrade to HBase 2.0

[HeartSaVioR]

Github user HeartSaVioR commented on the issue:

https://github.com/apache/storm/pull/2556
  
I'm expecting that the patch is just for the update of API change, hence 
questions look more important than the code change.

Several questions: 
1. Did you test the patch manually with HBase 2.0 beta 1?
2. I guess there's little chance for HBase community to introduce API 
change after beta, but the possibility is still open. Moreover that's not ideal 
to depend on beta version. Do you intend to open this for the official release 
of HBase 2.0 and update the version/API change once it happens?
3. Does HBase 2 client be compatible with HBase 1.x? At a glance of HBase 2 
project doc 
(https://docs.google.com/document/d/1WCsVlnHjJeKUcl7wHwqb4z9iEu_ktczrlKHK8N4SZzs/edit#)
 they claim admin interface is incompatible and HBase 1 client cannot 
administrate HBase 2. I'm wondering about opposite case (HBase 2 client to 
administrate HBase 1 server), and same case for other things. In short, I'd 
like to determine needs to have separate hbase module for HBase 1.x and HBase 
2.x.


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168044307
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

OK. Please let me know if you plan to figure out in time frame of Storm 
2.0.0. I'll add it in epic of releasing Storm 2.0.0. Thanks!


---

[GitHub] storm pull request #2556: STORM-2946: Upgrade to HBase 2.0

[ptgoetz]

GitHub user ptgoetz opened a pull request:

https://github.com/apache/storm/pull/2556

STORM-2946: Upgrade to HBase 2.0

https://issues.apache.org/jira/browse/STORM-2946

We may want to discuss changes like this in the email thread about 
independently versioned "external" components.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptgoetz/storm STORM-2946

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2556.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2556


commit 278f01bf09ee6d0bbe71315a69118961405b19fe
Author: P. Taylor Goetz 
Date:   2018-01-26T18:24:27Z

Merge branch 'STORM-2912' of github.com:HeartSaVioR/storm

commit f991dae41641eeb9101ec832dd2fd29b5ed3e059
Author: P. Taylor Goetz 
Date:   2018-02-07T19:02:54Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/storm

commit 5c3a911c0aed922421f4fab387dfad550ec90b31
Author: P. Taylor Goetz 
Date:   2018-02-13T17:13:02Z

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/storm

commit 06f7f66db3bee17257e7610ff67178fb4ddab49d
Author: P. Taylor Goetz 
Date:   2018-02-13T21:56:12Z

STORM-2946: Upgrade to HBase 2.0

commit 74439146a49aecf42fc129138e641c5d77ec0d76
Author: P. Taylor Goetz 
Date:   2018-02-13T23:52:54Z

STORM-2946: checkstyle cleanup




---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168037754
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

It is to figure out what will have for Storm 2.0... since we cannot make 
any breaking changes even if we like to thereafter until 3.0.  


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168034925
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

I thought STORM-2945 was filed to find the way to support background emit 
without external synchronization, so likely having the chance to keep 
unresolved in 2.x. If you intended to document how to enable background emit 
with current state in STORM-2945, please ignore the comment here.


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168033940
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

To nail down and document the concurrent emits semantics I had opened 
[STORM-2945](https://issues.apache.org/jira/browse/STORM-2945)


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168033466
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/messaging/netty/BackPressureStatus.java 
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.messaging.netty;
+
+import org.apache.storm.serialization.KryoValuesDeserializer;
+import org.apache.storm.serialization.KryoValuesSerializer;
+import org.jboss.netty.buffer.ChannelBuffer;
+import org.jboss.netty.buffer.ChannelBuffers;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
+
+// Instances of this type are sent from NettyWorker to upstream 
WorkerTransfer to indicate BackPressure situation
+public class BackPressureStatus {
+public static final short IDENTIFIER = (short)-600;
+private static final int SIZE_OF_ID = 2; // size if IDENTIFIER
+private static final int SIZE_OF_INT = 4;
+
+private static AtomicLong bpCount = new AtomicLong(0);
+
+public String workerId;
+public final long id;   // monotonically 
increasing id
--- End diff --

OK thanks for clarification. I thought it as some kinds of guarantee we 
should ensure. Maybe better to clear out that that's not a requirement and only 
for debugging purpose.


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168032846
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -22,19 +22,23 @@
 import org.apache.storm.executor.TupleInfo;
 import org.apache.storm.spout.ISpout;
 import org.apache.storm.spout.ISpoutOutputCollector;
+import org.apache.storm.tuple.AddressedTuple;
 import org.apache.storm.tuple.MessageId;
 import org.apache.storm.tuple.TupleImpl;
 import org.apache.storm.tuple.Values;
-import org.apache.storm.utils.Utils;
 import org.apache.storm.utils.MutableLong;
 import org.apache.storm.utils.RotatingMap;
+import org.apache.storm.utils.Utils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Random;
 
+// Methods are not thread safe. Each thread expected to have a separate 
instance, or else synchronize externally
--- End diff --

nit: better to make it as javadoc so that it can be exposed to more ways.

As @revans2 stated, I also think this is a good assumption for a spout, but 
even better to update the restriction if we have documented any. That's just a 
2 cents, not blocker.


---

Re: [DISCUSS] consider EOL for version lines

[P. Taylor Goetz]

This (3 version lines) is effectively what we’ve been doing — when a new 
version comes out major/minor version comes out, we stop updating/supporting 
the oldest previous version. For example, if we release 2.0 or 1.3 
(hypothetical), we would EOL 1.0.x. In some cases, like a security 
vulnerability, we may revisit an older release, but that’s fairly rare.

I agree that making it more explicit would help users. There’s not much to do 
other than update the downloads page and clean up dist.a.o.

-Taylor

> On Feb 13, 2018, at 4:45 PM, Jungtaek Lim  wrote:
> 
> We're in discussion to split out storm-kafka-client to have separate
> release version and cycle. Since we have applied necessary changes to
> storm-core of 1.1.x/1.0.x, unless storm-kafka-client needs more changes on
> storm-core, storm-core 1.1.x/1.0.x should be compatible with
> storm-kafka-client be compatible with storm-core 1.x.
> 
> I'm OK to keep the 1.0.x version lines specifically for storm-mesos (and
> then we would end up keeping 1.1.x version line too) if we can't avoid the
> case, but I couldn't guarantee all the bug fixes should be ported back to
> 1.0.x. We even often don't port back the bug fixes to 1.0.x version lines
> when 1.1 is the active minor version of Storm unless the fixes are
> critical. That means we may not port back the bug fixes to 1.1.x after
> announcing 1.2.0 and 1.2.x-branch is cut. As we all know, it greatly
> reduces the maintenance cost. It does make sense for storm-kafka-client to
> do the backport for 1.1.x/1.0.x branches because huge changes are just
> applied (though I guess the experiment described above may resolve such
> issue), but not for others including storm-core except critical/blocker
> issues.
> 
> So in storm-mesos view, it would be better to catch up with highest
> possible version (that's why I hope to adopt storm-mesos in storm repo,
> maybe not likely to happen for now), and I understand storm-mesos can't for
> now because of Storm's issue. I wish the investigation of "Storm on X"
> would occur actively sooner than later, so that it can be included as
> earlier version of Storm 2.x (ideally 2.0.0 but it is just an ideal view).
> 
> Anyway, looks like there is no objection to announce 0.10.x/0.9.x
> explicitly EOL.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2018년 2월 14일 (수) 오전 6:02, Erik Weathers 님이
> 작성:
> 
>> Thanks for keeping storm-mesos in mind Stig. :)  I'd be most worried about
>> any issues we might see with the backported storm-kafka-client and how we
>> *might* need to fix bugs in 1.0.x.  At least it should be easy to
>> cherry-pick fixes back into 1.0.x after the backport-stomping of
>> STORM-2937.
>> 
>> Look forward to working with Bobby to get a long term plan for storm to run
>> on mesos in 2.x+.
>> 
>> - Erik
>> 
>> On Tue, Feb 13, 2018 at 11:26 AM, Stig Rohde Døssing <
>> stigdoess...@gmail.com
>>> wrote:
>> 
>>> +1 to maintain 3 version lines, though we may want to look at what we can
>>> do for storm mesos, which I think it currently stuck on 1.0.x.
>>> 
>>> 2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro :
>>> 
 +1 to maintain 3 version lines. Let’s properly announce that in our
>>> portal
 and users list such that users know what’s coming.
 
 Agree with focusing on 2.0 which has a lot of improvements, rather than
 1.x, x >= 3.
 
> On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
 avermeerber...@gmail.com> wrote:
> 
> +1 (non binding) to maintaining less version lines, provided that
> 1.2.x branch is maintained long enough to allow progressive adoption
> of 2.x
> 
> Alexandre Vermeerbergen
> 
> 2018-02-13 19:38 GMT+01:00 Priyank Shah :
>> +1 to maintaining 3 version lines as suggested by Jungtaek.
>> 
>> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" <
 ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
>> 
>>   +1 to maintain 3 version lines.
>> 
>>   I think the next focus should be 2.0.0 than 1.3.0.
>> 
>> 
>> 
>> 
>>>   On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
>>> 
>>> Hi devs,
>>> 
>>> I've noticed that we are providing 4 different version lines
>> (1.1.x,
 1.0.x,
>>> 0.10.x, 0.9.x) in download page, and I expect we will add one more
>>> for
>>> 1.2.0. Moreover, we have one more develop version line (2.0.0 -
>>> master)
>>> which most of development happens there.
>>> 
>>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
>>> simultaneously and it took heavy effort to track all the RCs and
 verify all
>>> of them. I guess release manager would take more overhead of
 releasing, and
>>> it doesn't make sense for me if we continue maintaining all of
>> them.
>>> 
>>> Ideally I'd like to propose maintaining three version lines: 2.0.0

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r168026607
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/messaging/netty/BackPressureStatus.java 
---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.messaging.netty;
+
+import org.apache.storm.serialization.KryoValuesDeserializer;
+import org.apache.storm.serialization.KryoValuesSerializer;
+import org.jboss.netty.buffer.ChannelBuffer;
+import org.jboss.netty.buffer.ChannelBuffers;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
+
+// Instances of this type are sent from NettyWorker to upstream 
WorkerTransfer to indicate BackPressure situation
+public class BackPressureStatus {
+public static final short IDENTIFIER = (short)-600;
+private static final int SIZE_OF_ID = 2; // size if IDENTIFIER
+private static final int SIZE_OF_INT = 4;
+
+private static AtomicLong bpCount = new AtomicLong(0);
+
+public String workerId;
+public final long id;   // monotonically 
increasing id
--- End diff --

Its only for debugging purposes.. so that we can co-relate sent & recvd 
msgs. I have used it to measure latency involved in transmission of 
BackPressureStatus msgs. 


---

Re: [DISCUSS] consider EOL for version lines

[Jungtaek Lim]

We're in discussion to split out storm-kafka-client to have separate
release version and cycle. Since we have applied necessary changes to
storm-core of 1.1.x/1.0.x, unless storm-kafka-client needs more changes on
storm-core, storm-core 1.1.x/1.0.x should be compatible with
storm-kafka-client be compatible with storm-core 1.x.

I'm OK to keep the 1.0.x version lines specifically for storm-mesos (and
then we would end up keeping 1.1.x version line too) if we can't avoid the
case, but I couldn't guarantee all the bug fixes should be ported back to
1.0.x. We even often don't port back the bug fixes to 1.0.x version lines
when 1.1 is the active minor version of Storm unless the fixes are
critical. That means we may not port back the bug fixes to 1.1.x after
announcing 1.2.0 and 1.2.x-branch is cut. As we all know, it greatly
reduces the maintenance cost. It does make sense for storm-kafka-client to
do the backport for 1.1.x/1.0.x branches because huge changes are just
applied (though I guess the experiment described above may resolve such
issue), but not for others including storm-core except critical/blocker
issues.

So in storm-mesos view, it would be better to catch up with highest
possible version (that's why I hope to adopt storm-mesos in storm repo,
maybe not likely to happen for now), and I understand storm-mesos can't for
now because of Storm's issue. I wish the investigation of "Storm on X"
would occur actively sooner than later, so that it can be included as
earlier version of Storm 2.x (ideally 2.0.0 but it is just an ideal view).

Anyway, looks like there is no objection to announce 0.10.x/0.9.x
explicitly EOL.

- Jungtaek Lim (HeartSaVioR)

2018년 2월 14일 (수) 오전 6:02, Erik Weathers 님이
작성:

> Thanks for keeping storm-mesos in mind Stig. :)  I'd be most worried about
> any issues we might see with the backported storm-kafka-client and how we
> *might* need to fix bugs in 1.0.x.  At least it should be easy to
> cherry-pick fixes back into 1.0.x after the backport-stomping of
> STORM-2937.
>
> Look forward to working with Bobby to get a long term plan for storm to run
> on mesos in 2.x+.
>
> - Erik
>
> On Tue, Feb 13, 2018 at 11:26 AM, Stig Rohde Døssing <
> stigdoess...@gmail.com
> > wrote:
>
> > +1 to maintain 3 version lines, though we may want to look at what we can
> > do for storm mesos, which I think it currently stuck on 1.0.x.
> >
> > 2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro :
> >
> > > +1 to maintain 3 version lines. Let’s properly announce that in our
> > portal
> > > and users list such that users know what’s coming.
> > >
> > > Agree with focusing on 2.0 which has a lot of improvements, rather than
> > > 1.x, x >= 3.
> > >
> > > > On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com> wrote:
> > > >
> > > > +1 (non binding) to maintaining less version lines, provided that
> > > > 1.2.x branch is maintained long enough to allow progressive adoption
> > > > of 2.x
> > > >
> > > > Alexandre Vermeerbergen
> > > >
> > > > 2018-02-13 19:38 GMT+01:00 Priyank Shah :
> > > >> +1 to maintaining 3 version lines as suggested by Jungtaek.
> > > >>
> > > >> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" <
> > > ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
> > > >>
> > > >>+1 to maintain 3 version lines.
> > > >>
> > > >>I think the next focus should be 2.0.0 than 1.3.0.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
> > > >>
> > > >>> Hi devs,
> > > >>>
> > > >>> I've noticed that we are providing 4 different version lines
> (1.1.x,
> > > 1.0.x,
> > > >>> 0.10.x, 0.9.x) in download page, and I expect we will add one more
> > for
> > > >>> 1.2.0. Moreover, we have one more develop version line (2.0.0 -
> > master)
> > > >>> which most of development happens there.
> > > >>>
> > > >>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
> > > >>> simultaneously and it took heavy effort to track all the RCs and
> > > verify all
> > > >>> of them. I guess release manager would take more overhead of
> > > releasing, and
> > > >>> it doesn't make sense for me if we continue maintaining all of
> them.
> > > >>>
> > > >>> Ideally I'd like to propose maintaining three version lines: 2.0.0
> > > (next
> > > >>> major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix)
> > and
> > > >>> making others EOL (that respects semantic versioning and even other
> > > >>> projects tend to maintain only two version lines), but if someone
> > > feels too
> > > >>> aggressive, I propose at least we explicitly announce EOL to 0.x
> > > version
> > > >>> lines and get rid of any supports (downloads) for them.
> > > >>>
> > > >>> Would like to hear your opinion.
> > > >>>
> > > >>> Thanks,
> > > >>> Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] consider EOL for version lines

[Erik Weathers]

Thanks for keeping storm-mesos in mind Stig. :)  I'd be most worried about
any issues we might see with the backported storm-kafka-client and how we
*might* need to fix bugs in 1.0.x.  At least it should be easy to
cherry-pick fixes back into 1.0.x after the backport-stomping of STORM-2937.

Look forward to working with Bobby to get a long term plan for storm to run
on mesos in 2.x+.

- Erik

On Tue, Feb 13, 2018 at 11:26 AM, Stig Rohde Døssing  wrote:

> +1 to maintain 3 version lines, though we may want to look at what we can
> do for storm mesos, which I think it currently stuck on 1.0.x.
>
> 2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro :
>
> > +1 to maintain 3 version lines. Let’s properly announce that in our
> portal
> > and users list such that users know what’s coming.
> >
> > Agree with focusing on 2.0 which has a lot of improvements, rather than
> > 1.x, x >= 3.
> >
> > > On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
> > avermeerber...@gmail.com> wrote:
> > >
> > > +1 (non binding) to maintaining less version lines, provided that
> > > 1.2.x branch is maintained long enough to allow progressive adoption
> > > of 2.x
> > >
> > > Alexandre Vermeerbergen
> > >
> > > 2018-02-13 19:38 GMT+01:00 Priyank Shah :
> > >> +1 to maintaining 3 version lines as suggested by Jungtaek.
> > >>
> > >> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" <
> > ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
> > >>
> > >>+1 to maintain 3 version lines.
> > >>
> > >>I think the next focus should be 2.0.0 than 1.3.0.
> > >>
> > >>
> > >>
> > >>
> > >>On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
> > >>
> > >>> Hi devs,
> > >>>
> > >>> I've noticed that we are providing 4 different version lines (1.1.x,
> > 1.0.x,
> > >>> 0.10.x, 0.9.x) in download page, and I expect we will add one more
> for
> > >>> 1.2.0. Moreover, we have one more develop version line (2.0.0 -
> master)
> > >>> which most of development happens there.
> > >>>
> > >>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
> > >>> simultaneously and it took heavy effort to track all the RCs and
> > verify all
> > >>> of them. I guess release manager would take more overhead of
> > releasing, and
> > >>> it doesn't make sense for me if we continue maintaining all of them.
> > >>>
> > >>> Ideally I'd like to propose maintaining three version lines: 2.0.0
> > (next
> > >>> major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix)
> and
> > >>> making others EOL (that respects semantic versioning and even other
> > >>> projects tend to maintain only two version lines), but if someone
> > feels too
> > >>> aggressive, I propose at least we explicitly announce EOL to 0.x
> > version
> > >>> lines and get rid of any supports (downloads) for them.
> > >>>
> > >>> Would like to hear your opinion.
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>
> > >>
> > >>
> > >
> >
> >
>

[GitHub] storm pull request #2554: STORM-2939 add WorkerMetricsProcessor interface

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2554


---

Re: [DISCUSS] consider EOL for version lines

[Stig Rohde Døssing]

+1 to maintain 3 version lines, though we may want to look at what we can
do for storm mesos, which I think it currently stuck on 1.0.x.

2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro :

> +1 to maintain 3 version lines. Let’s properly announce that in our portal
> and users list such that users know what’s coming.
>
> Agree with focusing on 2.0 which has a lot of improvements, rather than
> 1.x, x >= 3.
>
> > On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
> >
> > +1 (non binding) to maintaining less version lines, provided that
> > 1.2.x branch is maintained long enough to allow progressive adoption
> > of 2.x
> >
> > Alexandre Vermeerbergen
> >
> > 2018-02-13 19:38 GMT+01:00 Priyank Shah :
> >> +1 to maintaining 3 version lines as suggested by Jungtaek.
> >>
> >> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" <
> ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
> >>
> >>+1 to maintain 3 version lines.
> >>
> >>I think the next focus should be 2.0.0 than 1.3.0.
> >>
> >>
> >>
> >>
> >>On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
> >>
> >>> Hi devs,
> >>>
> >>> I've noticed that we are providing 4 different version lines (1.1.x,
> 1.0.x,
> >>> 0.10.x, 0.9.x) in download page, and I expect we will add one more for
> >>> 1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
> >>> which most of development happens there.
> >>>
> >>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
> >>> simultaneously and it took heavy effort to track all the RCs and
> verify all
> >>> of them. I guess release manager would take more overhead of
> releasing, and
> >>> it doesn't make sense for me if we continue maintaining all of them.
> >>>
> >>> Ideally I'd like to propose maintaining three version lines: 2.0.0
> (next
> >>> major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
> >>> making others EOL (that respects semantic versioning and even other
> >>> projects tend to maintain only two version lines), but if someone
> feels too
> >>> aggressive, I propose at least we explicitly announce EOL to 0.x
> version
> >>> lines and get rid of any supports (downloads) for them.
> >>>
> >>> Would like to hear your opinion.
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>
> >>
> >>
> >
>
>

Re: [DISCUSS] consider EOL for version lines

[Hugo Da Cruz Louro]

+1 to maintain 3 version lines. Let’s properly announce that in our portal and 
users list such that users know what’s coming.

Agree with focusing on 2.0 which has a lot of improvements, rather than 1.x, x 
>= 3.

> On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen 
>  wrote:
> 
> +1 (non binding) to maintaining less version lines, provided that
> 1.2.x branch is maintained long enough to allow progressive adoption
> of 2.x
> 
> Alexandre Vermeerbergen
> 
> 2018-02-13 19:38 GMT+01:00 Priyank Shah :
>> +1 to maintaining 3 version lines as suggested by Jungtaek.
>> 
>> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" 
>>  wrote:
>> 
>>+1 to maintain 3 version lines.
>> 
>>I think the next focus should be 2.0.0 than 1.3.0.
>> 
>> 
>> 
>> 
>>On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
>> 
>>> Hi devs,
>>> 
>>> I've noticed that we are providing 4 different version lines (1.1.x, 1.0.x,
>>> 0.10.x, 0.9.x) in download page, and I expect we will add one more for
>>> 1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
>>> which most of development happens there.
>>> 
>>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
>>> simultaneously and it took heavy effort to track all the RCs and verify all
>>> of them. I guess release manager would take more overhead of releasing, and
>>> it doesn't make sense for me if we continue maintaining all of them.
>>> 
>>> Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
>>> major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
>>> making others EOL (that respects semantic versioning and even other
>>> projects tend to maintain only two version lines), but if someone feels too
>>> aggressive, I propose at least we explicitly announce EOL to 0.x version
>>> lines and get rid of any supports (downloads) for them.
>>> 
>>> Would like to hear your opinion.
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>> 
>> 
>> 
> 

Re: [DISCUSS] consider EOL for version lines

[Alexandre Vermeerbergen]

+1 (non binding) to maintaining less version lines, provided that
1.2.x branch is maintained long enough to allow progressive adoption
of 2.x

Alexandre Vermeerbergen

2018-02-13 19:38 GMT+01:00 Priyank Shah :
> +1 to maintaining 3 version lines as suggested by Jungtaek.
>
> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" 
>  wrote:
>
> +1 to maintain 3 version lines.
>
> I think the next focus should be 2.0.0 than 1.3.0.
>
>
>
>
> On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:
>
> >Hi devs,
> >
> >I've noticed that we are providing 4 different version lines (1.1.x, 
> 1.0.x,
> >0.10.x, 0.9.x) in download page, and I expect we will add one more for
> >1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
> >which most of development happens there.
> >
> >Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
> >simultaneously and it took heavy effort to track all the RCs and verify 
> all
> >of them. I guess release manager would take more overhead of releasing, 
> and
> >it doesn't make sense for me if we continue maintaining all of them.
> >
> >Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
> >major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
> >making others EOL (that respects semantic versioning and even other
> >projects tend to maintain only two version lines), but if someone feels 
> too
> >aggressive, I propose at least we explicitly announce EOL to 0.x version
> >lines and get rid of any supports (downloads) for them.
> >
> >Would like to hear your opinion.
> >
> >Thanks,
> >Jungtaek Lim (HeartSaVioR)
>
>
>

Re: [DISCUSS] consider EOL for version lines

[Priyank Shah]

+1 to maintaining 3 version lines as suggested by Jungtaek.

On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" 
 wrote:

+1 to maintain 3 version lines.

I think the next focus should be 2.0.0 than 1.3.0.




On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:

>Hi devs,
>
>I've noticed that we are providing 4 different version lines (1.1.x, 1.0.x,
>0.10.x, 0.9.x) in download page, and I expect we will add one more for
>1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
>which most of development happens there.
>
>Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
>simultaneously and it took heavy effort to track all the RCs and verify all
>of them. I guess release manager would take more overhead of releasing, and
>it doesn't make sense for me if we continue maintaining all of them.
>
>Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
>major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
>making others EOL (that respects semantic versioning and even other
>projects tend to maintain only two version lines), but if someone feels too
>aggressive, I propose at least we explicitly announce EOL to 0.x version
>lines and get rid of any supports (downloads) for them.
>
>Would like to hear your opinion.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] consider EOL for version lines

[Arun Mahadevan]

+1 to maintain 3 version lines.

I think the next focus should be 2.0.0 than 1.3.0.




On 2/12/18, 11:40 PM, "Jungtaek Lim"  wrote:

>Hi devs,
>
>I've noticed that we are providing 4 different version lines (1.1.x, 1.0.x,
>0.10.x, 0.9.x) in download page, and I expect we will add one more for
>1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
>which most of development happens there.
>
>Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
>simultaneously and it took heavy effort to track all the RCs and verify all
>of them. I guess release manager would take more overhead of releasing, and
>it doesn't make sense for me if we continue maintaining all of them.
>
>Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
>major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
>making others EOL (that respects semantic versioning and even other
>projects tend to maintain only two version lines), but if someone feels too
>aggressive, I propose at least we explicitly announce EOL to 0.x version
>lines and get rid of any supports (downloads) for them.
>
>Would like to hear your opinion.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)

[GitHub] storm issue #2502: new PR for STORM-2306 : Messaging subsystem redesign

[roshannaik]

Github user roshannaik commented on the issue:

https://github.com/apache/storm/pull/2502
  
I think I have addressed all the major and minor issues as well. 


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r167814459
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/messaging/netty/BackPressureStatus.java 
---
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License
+ */
+
+package org.apache.storm.messaging.netty;
+
+import org.apache.storm.serialization.KryoValuesDeserializer;
+import org.apache.storm.serialization.KryoValuesSerializer;
+import org.jboss.netty.buffer.ChannelBuffer;
+import org.jboss.netty.buffer.ChannelBuffers;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.concurrent.atomic.AtomicLong;
+
+// Instances of this type are sent from NettyWorker to upstream 
WorkerTransfer to indicate BackPressure situation
+public class BackPressureStatus implements java.io.Serializable {
--- End diff --

Thanks for the detailed comment. Made the changes and tested them as well.


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r167807269
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/executor/spout/SpoutOutputCollectorImpl.java
 ---
@@ -44,14 +48,15 @@
 private final Boolean isEventLoggers;
 private final Boolean isDebug;
 private final RotatingMap pending;
+private TupleInfo globalTupleInfo = new TupleInfo();  // thread 
safety: assumes Collector.emit*() calls are externally synchronized (if needed).
--- End diff --

To get the id of the current thread  involves a call to 
Thread.currentThread() which is quite 
[expensive](!http://www.jutils.com/checks/performance.html)... so not good to 
use to determine whether or not to use fast path.

I am introducing that check if topology.debug is enabled as a compromise. 
This mode could 
 be used mode to do any checks in dev mode that are unnecessary or 
expensive to do repeatedly in production.

I have opened: 
[STORM-2945](!https://issues.apache.org/jira/browse/STORM-2945) to nail down 
and document background emits support.. we can document both spout & bolt 
support semantics together in the same document.


---

[GitHub] storm pull request #2502: new PR for STORM-2306 : Messaging subsystem redesi...

[roshannaik]

Github user roshannaik commented on a diff in the pull request:

https://github.com/apache/storm/pull/2502#discussion_r167800249
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/trident/spout/RichSpoutBatchExecutor.java 
---
@@ -194,7 +194,12 @@ public void reportError(Throwable t) {
 public void emitDirect(int task, String stream, List 
values, Object id) {
 throw new UnsupportedOperationException("Trident does not 
support direct streams");
 }
-
+
+@Override
+public void flush() {
+//NOOP   //TODO: Roshan: validate if this is OK
--- End diff --

needs to flush.


---

[GitHub] storm pull request #2555: STORM-2841 Use extended class instead of partial m...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2555


---