[GitHub] [storm] agresch opened a new pull request #3125: STORM-3506 prevent topo conf from overriding some system properties

2019-09-05 Thread GitBox 
agresch opened a new pull request #3125: STORM-3506 prevent topo conf from 
overriding some system properties
URL: https://github.com/apache/storm/pull/3125
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm merged pull request #3122: [STORM-3503] add unit test for BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_…

2019-09-05 Thread GitBox 
Ethanlm merged pull request #3122: [STORM-3503] add unit test for 
BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_…
URL: https://github.com/apache/storm/pull/3122
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm merged pull request #3113: STORM-3482 Implement One Worker Per Component Option

2019-09-05 Thread GitBox 
Ethanlm merged pull request #3113: STORM-3482 Implement One Worker Per 
Component Option
URL: https://github.com/apache/storm/pull/3113
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] RuiLi8080 commented on issue #3122: [STORM-3503] add unit test for BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_…

2019-09-05 Thread GitBox 
RuiLi8080 commented on issue #3122: [STORM-3503] add unit test for 
BLACKLIST_SCHEDULER_ASSUME_SUPERVISOR_…
URL: https://github.com/apache/storm/pull/3122#issuecomment-528574126
 
 
   Addressed comments. Very good advice. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm merged pull request #3123: STORM-3488 Scheduling can cause RAS_Node resources to become negative

2019-09-05 Thread GitBox 
Ethanlm merged pull request #3123: STORM-3488 Scheduling can cause RAS_Node 
resources to become negative
URL: https://github.com/apache/storm/pull/3123
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on a change in pull request #3113: STORM-3482 Implement One Worker Per Component Option

2019-09-05 Thread GitBox 
Ethanlm commented on a change in pull request #3113: STORM-3482 Implement One 
Worker Per Component Option
URL: https://github.com/apache/storm/pull/3113#discussion_r321406815
 
 

 ##
 File path: storm-client/src/jvm/org/apache/storm/Config.java
 ##
 @@ -318,9 +318,19 @@
 /**
  * Whether to limit each worker to one executor. This is useful for 
debugging topologies to clearly identify workers that
  * are slow/crashing and for estimating resource requirements and capacity.
+ * If both {@link #TOPOLOGY_RAS_ONE_EXECUTOR_PER_WORKER} and {@link 
#TOPOLOGY_RAS_ONE_COMPONENT_PER_WORKER} are enabled,
+ * {@link #TOPOLOGY_RAS_ONE_COMPONENT_PER_WORKER} is ignored.
  */
 @IsBoolean
 public static final String TOPOLOGY_RAS_ONE_EXECUTOR_PER_WORKER = 
"topology.ras.one.executor.per.worker";
+/**
+ * Whether to limit each worker to one component. This is useful for 
debugging topologies to clearly identify workers that
+ * are slow/crashing and for estimating resource requirements and capacity.
+ * If both TOPOLOGY_RAS_ONE_EXECUTOR_PER_WORKER and 
TOPOLOGY_RAS_ONE_COMPONENT_PER_WORKER are enabled,
 
 Review comment:
   Ideally we want to use javadoc style here too. But it's okay


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on a change in pull request #3113: STORM-3482 Implement One Worker Per Component Option

2019-09-05 Thread GitBox 
Ethanlm commented on a change in pull request #3113: STORM-3482 Implement One 
Worker Per Component Option
URL: https://github.com/apache/storm/pull/3113#discussion_r321357142
 
 

 ##
 File path: 
storm-server/src/main/java/org/apache/storm/scheduler/resource/RasNode.java
 ##
 @@ -362,15 +364,40 @@ public void assignSingleExecutor(WorkerSlot ws, 
ExecutorDetails exec, TopologyDe
  * @return true if it would fit else false
  */
 public boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, 
TopologyDetails td) {
-assert nodeId.equals(ws.getNodeId()) : "Slot " + ws + " is not a part 
of this node " + nodeId;
 
 Review comment:
   I didn't notice this before. Why do we want to delete this line


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node resources to become negative

2019-09-05 Thread GitBox 
Ethanlm commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node 
resources to become negative
URL: https://github.com/apache/storm/pull/3123#issuecomment-528431929
 
 
   @srdo  Thank you very much. I really appreciated it


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] srdo commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node resources to become negative

2019-09-05 Thread GitBox 
srdo commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node 
resources to become negative
URL: https://github.com/apache/storm/pull/3123#issuecomment-528428513
 
 
   Ok, I just wanted to be sure you knew. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] Ethanlm commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node resources to become negative

2019-09-05 Thread GitBox 
Ethanlm commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node 
resources to become negative
URL: https://github.com/apache/storm/pull/3123#issuecomment-528424261
 
 
   @srdo  Thanks for the information. I tried to cherry-pick this commit but 
there were conflicts. So I asked @dandsager1 to file a separate PR so we can 
review.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [storm] srdo commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node resources to become negative

2019-09-05 Thread GitBox 
srdo commented on issue #3123: STORM-3488 Scheduling can cause RAS_Node 
resources to become negative
URL: https://github.com/apache/storm/pull/3123#issuecomment-528415492
 
 
   @Ethanlm I can't remember if this is described anywhere, so just in case no 
one told you: Generally when cherry picking back changes, unless there are 
conflicts with the base branch I think you can do the cherry pick without a 
PR/review and without waiting for the 24 hour waiting period. I think we have 
been considering the original branch review sufficient when there aren't 
conflicts. Just in case you don't want to go through the delay :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Re: [VOTE] Release Apache Storm 2.1.0 (rc3)

2019-09-05 Thread Stig Rohde Døssing 
Sounds great, thanks Ethan.

Den tor. 5. sep. 2019 kl. 16.55 skrev Ethan Li :

> Once another bug fix https://github.com/apache/storm/pull/3123 <
> https://github.com/apache/storm/pull/3123> is merged, I think we can run
> another release candidate.
>
>
>
> > On Aug 28, 2019, at 9:27 AM, Ethan Li  wrote:
> >
> > Thanks for finding the issue.
> >
> > I just uploaded my key to http://pgp.mit.edu/ .
> It’s a good idea to link to https://www.apache.org/info/verification.html
>   on download page.
> >
> >> On Aug 27, 2019, at 3:35 PM, Stig Rohde Døssing  > wrote:
> >>
> >> -1, I think this issue is a blocker. It's caused by
> >> https://github.com/apache/storm/pull/3093/files <
> https://github.com/apache/storm/pull/3093/files>, which prevents the
> >> localizer from deleting blobs that are deleted from Nimbus, even if
> those
> >> blobs are only scheduled for download.
> >>
> >> Have opened https://github.com/apache/storm/pull/3119 <
> https://github.com/apache/storm/pull/3119>
> >>
> >> Den tir. 27. aug. 2019 kl. 19.54 skrev Stig Rohde Døssing <
> >> stigdoess...@gmail.com >:
> >>
> >>> I think there may be an issue with topology cleanup
> >>> I ran WordCountTopology for a bit, then killed it. The supervisor
> seems to
> >>> keep trying to download the topology jar indefinitely.
> >>>
> >>> 2019-08-27 19:49:53.459 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor
> - 1
> >>> [WARN] Failed to download blob LOCAL TOPO BLOB TOPO_JAR
> >>> word-count-1-1566926844 will try again in 100 ms
> >>> org.apache.storm.generated.KeyNotFoundException: null
> >>> at
> >>>
> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25887)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:25818)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
> >>> ~[storm-shaded-deps-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.generated.Nimbus$Client.recv_getBlobMeta(Nimbus.java:794)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> org.apache.storm.generated.Nimbus$Client.getBlobMeta(Nimbus.java:781)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.blobstore.NimbusBlobStore.getBlobMeta(NimbusBlobStore.java:85)
> >>> ~[storm-client-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:127)
> >>> ~[storm-server-2.1.0.jar:2.1.0]
> >>> at
> >>>
> org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:264)
> >>> ~[storm-server-2.1.0.jar:2.1.0]
> >>> at
> >>>
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> >>> [?:1.8.0_144]
> >>> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>> [?:1.8.0_144]
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> >>> [?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >>> [?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>> [?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>> [?:1.8.0_144]
> >>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> >>>
> >>> These repeat, intermittently broken up by
> >>>
> >>> Caused by: java.lang.RuntimeException: Could not download...
> >>> at
> >>>
> org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:285)
> >>> ~[storm-server-2.1.0.jar:2.1.0]
> >>> at
> >>>
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> >>> ~[?:1.8.0_144]
> >>> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>> ~[?:1.8.0_144]
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> >>> ~[?:1.8.0_144]
> >>> at
> >>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >>> ~[?:1.8.0_144]
> >>> ... 3 more
> >>> Caused by: org.apache.storm.generated.KeyNotFoundException
> >>> at
> >>>
> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919)
> >>>

Re: [VOTE] Release Apache Storm 2.1.0 (rc3)

2019-09-05 Thread Ethan Li 
Once another bug fix https://github.com/apache/storm/pull/3123 
 is merged, I think we can run 
another release candidate. 



> On Aug 28, 2019, at 9:27 AM, Ethan Li  wrote:
> 
> Thanks for finding the issue. 
> 
> I just uploaded my key to http://pgp.mit.edu/ . It’s a 
> good idea to link to https://www.apache.org/info/verification.html 
>   on download page.
> 
>> On Aug 27, 2019, at 3:35 PM, Stig Rohde Døssing > > wrote:
>> 
>> -1, I think this issue is a blocker. It's caused by
>> https://github.com/apache/storm/pull/3093/files 
>> , which prevents the
>> localizer from deleting blobs that are deleted from Nimbus, even if those
>> blobs are only scheduled for download.
>> 
>> Have opened https://github.com/apache/storm/pull/3119 
>> 
>> 
>> Den tir. 27. aug. 2019 kl. 19.54 skrev Stig Rohde Døssing <
>> stigdoess...@gmail.com >:
>> 
>>> I think there may be an issue with topology cleanup
>>> I ran WordCountTopology for a bit, then killed it. The supervisor seems to
>>> keep trying to download the topology jar indefinitely.
>>> 
>>> 2019-08-27 19:49:53.459 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor - 1
>>> [WARN] Failed to download blob LOCAL TOPO BLOB TOPO_JAR
>>> word-count-1-1566926844 will try again in 100 ms
>>> org.apache.storm.generated.KeyNotFoundException: null
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25887)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:25818)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:88)
>>> ~[storm-shaded-deps-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.generated.Nimbus$Client.recv_getBlobMeta(Nimbus.java:794)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at org.apache.storm.generated.Nimbus$Client.getBlobMeta(Nimbus.java:781)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.blobstore.NimbusBlobStore.getBlobMeta(NimbusBlobStore.java:85)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:127)
>>> ~[storm-server-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:264)
>>> ~[storm-server-2.1.0.jar:2.1.0]
>>> at
>>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>>> [?:1.8.0_144]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> [?:1.8.0_144]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> [?:1.8.0_144]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> [?:1.8.0_144]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>> [?:1.8.0_144]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>> [?:1.8.0_144]
>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
>>> 
>>> These repeat, intermittently broken up by
>>> 
>>> Caused by: java.lang.RuntimeException: Could not download...
>>> at
>>> org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:285)
>>> ~[storm-server-2.1.0.jar:2.1.0]
>>> at
>>> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
>>> ~[?:1.8.0_144]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> ~[?:1.8.0_144]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> ~[?:1.8.0_144]
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> ~[?:1.8.0_144]
>>> ... 3 more
>>> Caused by: org.apache.storm.generated.KeyNotFoundException
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25887)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>> org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:25818)
>>> ~[storm-client-2.1.0.jar:2.1.0]
>>> at
>>>