[GitHub] storm pull request #2842: STORM-3229: Add in better error reporting
GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2842 STORM-3229: Add in better error reporting You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm STORM-3229 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2842.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2842 commit b1a535de35b8813a84d1cf6d95bc811fa0a58381 Author: Robert (Bobby) Evans Date: 2018-09-18T22:03:39Z STORM-3229: Add in better error reporting ---
Re: Regarding releasing Apache Storm 2.0.0
I’m ready to release when everything is ready to go. Since we haven’t released from the 2.0-based master branch, I wouldn’t be surprised if I run into release issues, but I’ll slog through it. -Taylor > On Sep 18, 2018, at 10:46 AM, Bobby Evans wrote: > > Great work everyone. We are really close on this. We have everything in > except for https://github.com/apache/storm/pull/2719, but there has been no > movement there, so I will try and put up an alternative pull request. > > Also We noticed that a recent merge broke some things fairly badly so we > need to get https://github.com/apache/storm/pull/2839 in, but that is just > a matter of waiting a few more hours for the 24 hours to be up. > > Great work everyone, hopefully we will have an RC up for a vote a little > over a day from now. > > Thanks, > > Bobby > > P.S. Taylor, You have put up all of the release candidates in the past and > done all of the votes for them. If you want to continue the trend that is > fine with me, but if not I am happy to do it, but I might have to bug you > to be sure I do it all correctly. > > On Mon, Sep 17, 2018 at 9:13 AM Bobby Evans wrote: > >> I think we are really close on this and I would love to see us get an RC >> out ASAP. >> >> We are still missing some things that Stig called out. >> >> https://github.com/apache/storm/pull/2719 has a build issue, not sure if >> we need to make an alternative patch or not. >> https://github.com/apache/storm/pull/2800 has a newer alternative patch >> https://github.com/apache/storm/pull/2836 please take a look. >> https://github.com/apache/storm/pull/2805 has some merge conflicts >> currently, but everyone please take a chance to review it. >> >> Thanks, >> >> Bobby >> >> On Fri, Sep 14, 2018 at 2:57 AM Jungtaek Lim wrote: >> >>> I have sought the name of client artifact from some of streaming >>> frameworks. Please refer below: >>> >>> Spark: spark-core >>> Kafka: kafka-clients >>> Flink: flink-clients >>> Heron: heron-api >>> >>> Based on divergence, I don't see the reason "storm-core" is the only name >>> which avoid confusion. Actually, if my understanding is right, we need to >>> let end users including "storm-server" when running local cluster, then >>> "storm-core" vs "storm-server" would give real confusion. I guess we >>> already discussed about the naming, and given that we don't rename it we >>> are OK with renamed artifacts. >>> >>> 2018년 9월 14일 (금) 오후 4:07, Roshan Naik 님이 >>> 작성: >>> Happy to see consensus in moving fwd with 2.0 soon. I will try to get a minor patch (STORM-3205) within 24 hours ... as it seems like it has potential to deliver a decent perf boost and energy savings. One thing I am hoping we can address before releasing Storm 2 is... to >>> fix the naming of the storm-client.jar. Its such a core jar really, it >>> should have been really called storm-core or something like that... but unfortunately we already have another jar with that name. Retaining the 'client' name for this new jar would be confusing and give wrong impressions to users and any new devs IMO. -roshan On Thursday, September 13, 2018, 2:12:40 PM PDT, Govind Menon wrote: STORM-3217 and STORM-3221 have been fixed - +1 from me for 2.0 RC. On Wed, Sep 12, 2018 at 10:01 AM Govind Menon wrote: > Hi all, > > There are some regressions that I introduced as part of STORM-1311 >>> which > I'm working on as part of https://issues.apache.org/jira/browse/STORM-3217 > and https://issues.apache.org/jira/browse/STORM-3221. These should be > fixed before a 2.x release > > I have code working on the Yahoo internal branch and should have PRs >>> up > for them in community soon. > > I apologize for slowing things up. > > Thanks, > Govind. > > On Tue, Sep 11, 2018 at 3:31 PM Arun Mahadevan >>> wrote: > >> +1 for releasing 2.0. >> >> May be the RC can be cut once critical patches are merged. >> >> On Tue, 11 Sep 2018 at 10:28, Stig Rohde Døssing < stigdoess...@gmail.com> >> wrote: >> >>> +1 to cut an RC. >>> >>> Here are a couple of PRs that could maybe go in >>> >>> https://github.com/apache/storm/pull/2719 >>> https://github.com/apache/storm/pull/2800 (this one requires some >> changes, >>> but we should be able to fix it pretty quickly) >>> also would like to get https://github.com/apache/storm/pull/2805 >> reviewed, >>> it might change some public methods. >>> >>> Other than that, we should try to remove as much deprecated code >>> as we >> can >>> before release >>> >>> https://issues.apache.org/jira/browse/STORM-2947 >>> >>> Den man. 10. sep. 2018 kl. 21.59 skrev Alexandre Vermeerbergen < >>> avermeerber...@gmail.com>: >>> +1 for an Storm 2.0 as soon as
[GitHub] storm issue #2841: STORM-3105: upgrade version of hive
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2841 The build failures look like issues with downloading the JDK/slowness in travis. https://travis-ci.org/revans2/incubator-storm/builds/430134233 is my version that passed. ---
[GitHub] storm pull request #2839: STORM-3228 allow refernce counting of differing Po...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2839 ---
[GitHub] storm issue #2839: STORM-3228 allow refernce counting of differing PortAndAs...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2839 Actually these are all comments so I'll just take care of it on checkin. ---
[GitHub] storm pull request #2839: STORM-3228 allow refernce counting of differing Po...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2839#discussion_r218564458
--- Diff:
storm-server/src/main/java/org/apache/storm/localizer/TimePortAndAssignment.java
---
@@ -48,4 +48,29 @@ public LocalAssignment getAssignment() {
public void complete() {
stopTiming();
}
+
+@Override
+public String toString() {
+return "TimePortAndAssignment{" +
getAssignment().get_topology_id() + " on " + getPort() + "}";
+}
+
+/**
+ * All implementations of PortAndAssignment should implement the same
hashCode() method
+ */
+@Override
+public int hashCode() {
+return (17 * getPort()) + getAssignment().hashCode();
+}
+
+/**
+ * All implementations of PortAndAssignment should implement the same
equals() method
--- End diff --
Checkstyle: missing period here too.
---
[GitHub] storm pull request #2839: STORM-3228 allow refernce counting of differing Po...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2839#discussion_r218564297
--- Diff:
storm-server/src/main/java/org/apache/storm/localizer/PortAndAssignmentImpl.java
---
@@ -45,14 +48,17 @@ public String getOwner() {
return assignment.get_owner();
}
+/**
+ * All implementations of PortAndAssignment should implement the same
hashCode() method
--- End diff --
checkstyle: missing period at the end of this.
---
[GitHub] storm pull request #2839: STORM-3228 allow refernce counting of differing Po...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2839#discussion_r218564194
--- Diff:
storm-server/src/main/java/org/apache/storm/localizer/PortAndAssignmentImpl.java
---
@@ -26,13 +26,16 @@ public PortAndAssignmentImpl(int port, LocalAssignment
assignment) {
this.assignment = assignment;
}
+/**
+ * All implementations of PortAndAssignment should implement the same
equals() method
--- End diff --
checkstyle: missing period at the end of this.
---
[GitHub] storm pull request #2839: STORM-3228 allow refernce counting of differing Po...
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2839#discussion_r218564410
--- Diff:
storm-server/src/main/java/org/apache/storm/localizer/TimePortAndAssignment.java
---
@@ -48,4 +48,29 @@ public LocalAssignment getAssignment() {
public void complete() {
stopTiming();
}
+
+@Override
+public String toString() {
+return "TimePortAndAssignment{" +
getAssignment().get_topology_id() + " on " + getPort() + "}";
+}
+
+/**
+ * All implementations of PortAndAssignment should implement the same
hashCode() method
--- End diff --
checkstyle: missing period here too.
---
[GitHub] storm issue #2839: STORM-3228 allow refernce counting of differing PortAndAs...
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2839 There are some new checkstyle violations that we missed before. ---
[GitHub] storm pull request #2838: STORM-3227: Only push credentials if going to expe...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2838 ---
[GitHub] storm issue #2840: STORM-3147: Add metrics based on ClusterSummary
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2840 I'm hoping @zd-project will help review this, since this is a rebase of his PR. I don't really have the context to know whether these added metrics are useful. ---
[GitHub] storm issue #2841: STORM-3105: upgrade version of hive
Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2841 The build is failing do not merge yet... ---
[GitHub] storm pull request #2840: STORM-3147: Add metrics based on ClusterSummary
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2840#discussion_r218485913
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -4767,4 +4797,205 @@ public IScheduler getForcedScheduler() {
}
+private static class ClusterSummaryMetrics implements MetricSet {
+private static final String SUMMARY = "summary";
+private final Map metrics =
new HashMap<>();
+
+public com.codahale.metrics.Metric put(String key,
com.codahale.metrics.Metric value) {
+return metrics.put(MetricRegistry.name(SUMMARY, key), value);
+}
+
+@Override
+public Map getMetrics() {
+return metrics;
+}
+}
+
+private class ClusterSummaryMetricSet implements Runnable {
+private static final int CACHING_WINDOW = 5;
+
+private final ClusterSummaryMetrics clusterSummaryMetrics = new
ClusterSummaryMetrics();
+
+private final Function registerHistogram =
(name) -> {
+//This histogram reflects the data distribution across only
one ClusterSummary, i.e.,
+// data distribution across all entities of a type (e.g., data
from all nimbus/topologies) at one moment.
+// Hence we use half of the CACHING_WINDOW time to ensure it
retains only data from the most recent update
+final Histogram histogram = new Histogram(new
SlidingTimeWindowReservoir(CACHING_WINDOW / 2, TimeUnit.SECONDS));
+clusterSummaryMetrics.put(name, histogram);
+return histogram;
+};
+private volatile boolean active = false;
+
+//NImbus metrics distribution
+private final Histogram nimbusUptime =
registerHistogram.apply("nimbuses:uptime-secs");
+
+//Supervisor metrics distribution
+private final Histogram supervisorsUptime =
registerHistogram.apply("supervisors:uptime-secs");
+private final Histogram supervisorsNumWorkers =
registerHistogram.apply("supervisors:num-workers");
+private final Histogram supervisorsNumUsedWorkers =
registerHistogram.apply("supervisors:num-used-workers");
+private final Histogram supervisorsUsedMem =
registerHistogram.apply("supervisors:used-mem");
+private final Histogram supervisorsUsedCpu =
registerHistogram.apply("supervisors:used-cpu");
+private final Histogram supervisorsFragmentedMem =
registerHistogram.apply("supervisors:fragmented-mem");
+private final Histogram supervisorsFragmentedCpu =
registerHistogram.apply("supervisors:fragmented-cpu");
+
+//Topology metrics distribution
+private final Histogram topologiesNumTasks =
registerHistogram.apply("topologies:num-tasks");
+private final Histogram topologiesNumExecutors =
registerHistogram.apply("topologies:num-executors");
+private final Histogram topologiesNumWorker =
registerHistogram.apply("topologies:num-workers");
+private final Histogram topologiesUptime =
registerHistogram.apply("topologies:uptime-secs");
+private final Histogram topologiesReplicationCount =
registerHistogram.apply("topologies:replication-count");
+private final Histogram topologiesRequestedMemOnHeap =
registerHistogram.apply("topologies:requested-mem-on-heap");
+private final Histogram topologiesRequestedMemOffHeap =
registerHistogram.apply("topologies:requested-mem-off-heap");
+private final Histogram topologiesRequestedCpu =
registerHistogram.apply("topologies:requested-cpu");
+private final Histogram topologiesAssignedMemOnHeap =
registerHistogram.apply("topologies:assigned-mem-on-heap");
+private final Histogram topologiesAssignedMemOffHeap =
registerHistogram.apply("topologies:assigned-mem-off-heap");
+private final Histogram topologiesAssignedCpu =
registerHistogram.apply("topologies:assigned-cpu");
--- End diff --
I am sorry, but I am really having a hard time of seeing a histogram of
these being valuable, but perhaps I am wrong. I would much rather see
dimensions/tags being used, but since those are not available yet, I am okay
with this.
---
[GitHub] storm pull request #2840: STORM-3147: Add metrics based on ClusterSummary
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/2840#discussion_r218484299
--- Diff:
storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java ---
@@ -2922,19 +2948,20 @@ public void launchServer() throws Exception {
}
});
-metricsRegistry.registerGauge("nimbus:num-supervisors", () ->
state.supervisors(null).size());
-metricsRegistry.registerGauge("nimbus:fragmented-memory",
this::fragmentedMemory);
-metricsRegistry.registerGauge("nimbus:fragmented-cpu",
this::fragmentedCpu);
-metricsRegistry.registerGauge("nimbus:available-memory", () ->
nodeIdToResources.get().values().parallelStream()
-.mapToDouble(SupervisorResources::getAvailableMem)
+
metricsRegistry.registerGauge("nimbus:total-available-memory-non-negative", ()
-> nodeIdToResources.get().values()
--- End diff --
There appears to be some oddness with the naming of the metrics. It would
be nice to have some kind of convention for them.
Perhaps we can have something like
* 'nimbus:cluster:available-memory-mb' to show that this came from nimbus,
it is about the cluster and it is the available memory in MB.
* 'nimbus:cluster:available-cpu-pct'
* 'nimbus:cluster:total-memory-mb'
* 'nimbus:cluster:total-cpu-pct'
etc.
---
[GitHub] storm pull request #2841: STORM-3105: upgrade version of hive
GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2841 STORM-3105: upgrade version of hive You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm STORM-3105 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2841.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2841 commit 16d3910101ae4be0db66cb6972c614b98572d098 Author: Abhishek Date: 2018-06-14T16:35:27Z Update hive.version to last stable hive release. commit 5742d2e2a590f274cbcecddeb56f7e2bb551865e Author: Robert (Bobby) Evans Date: 2018-09-18T14:47:41Z Merge branch 'STORM-3105' of https://github.com/aandis/storm into STORM-3105 This closes #2719 commit f9f123367cd00106562c949c32044fe413e30418 Author: Robert (Bobby) Evans Date: 2018-09-18T15:19:37Z STORM-3105: Added exclude to let build continue ---
Re: Regarding releasing Apache Storm 2.0.0
Great work everyone. We are really close on this. We have everything in except for https://github.com/apache/storm/pull/2719, but there has been no movement there, so I will try and put up an alternative pull request. Also We noticed that a recent merge broke some things fairly badly so we need to get https://github.com/apache/storm/pull/2839 in, but that is just a matter of waiting a few more hours for the 24 hours to be up. Great work everyone, hopefully we will have an RC up for a vote a little over a day from now. Thanks, Bobby P.S. Taylor, You have put up all of the release candidates in the past and done all of the votes for them. If you want to continue the trend that is fine with me, but if not I am happy to do it, but I might have to bug you to be sure I do it all correctly. On Mon, Sep 17, 2018 at 9:13 AM Bobby Evans wrote: > I think we are really close on this and I would love to see us get an RC > out ASAP. > > We are still missing some things that Stig called out. > > https://github.com/apache/storm/pull/2719 has a build issue, not sure if > we need to make an alternative patch or not. > https://github.com/apache/storm/pull/2800 has a newer alternative patch > https://github.com/apache/storm/pull/2836 please take a look. > https://github.com/apache/storm/pull/2805 has some merge conflicts > currently, but everyone please take a chance to review it. > > Thanks, > > Bobby > > On Fri, Sep 14, 2018 at 2:57 AM Jungtaek Lim wrote: > >> I have sought the name of client artifact from some of streaming >> frameworks. Please refer below: >> >> Spark: spark-core >> Kafka: kafka-clients >> Flink: flink-clients >> Heron: heron-api >> >> Based on divergence, I don't see the reason "storm-core" is the only name >> which avoid confusion. Actually, if my understanding is right, we need to >> let end users including "storm-server" when running local cluster, then >> "storm-core" vs "storm-server" would give real confusion. I guess we >> already discussed about the naming, and given that we don't rename it we >> are OK with renamed artifacts. >> >> 2018년 9월 14일 (금) 오후 4:07, Roshan Naik 님이 >> 작성: >> >> > Happy to see consensus in moving fwd with 2.0 soon. >> > I will try to get a minor patch (STORM-3205) within 24 hours ... as it >> > seems like it has potential to deliver a decent perf boost and energy >> > savings. >> > One thing I am hoping we can address before releasing Storm 2 is... to >> fix >> > the naming of the storm-client.jar. Its such a core jar really, it >> should >> > have been really called storm-core or something like that... but >> > unfortunately we already have another jar with that name. Retaining the >> > 'client' name for this new jar would be confusing and give wrong >> > impressions to users and any new devs IMO. >> > -roshan >> > >> > On Thursday, September 13, 2018, 2:12:40 PM PDT, Govind Menon >> > wrote: >> > >> > STORM-3217 and STORM-3221 have been fixed - +1 from me for 2.0 RC. >> > >> > On Wed, Sep 12, 2018 at 10:01 AM Govind Menon wrote: >> > >> > > Hi all, >> > > >> > > There are some regressions that I introduced as part of STORM-1311 >> which >> > > I'm working on as part of >> > https://issues.apache.org/jira/browse/STORM-3217 >> > > and https://issues.apache.org/jira/browse/STORM-3221. These should be >> > > fixed before a 2.x release >> > > >> > > I have code working on the Yahoo internal branch and should have PRs >> up >> > > for them in community soon. >> > > >> > > I apologize for slowing things up. >> > > >> > > Thanks, >> > > Govind. >> > > >> > > On Tue, Sep 11, 2018 at 3:31 PM Arun Mahadevan >> wrote: >> > > >> > >> +1 for releasing 2.0. >> > >> >> > >> May be the RC can be cut once critical patches are merged. >> > >> >> > >> On Tue, 11 Sep 2018 at 10:28, Stig Rohde Døssing < >> > stigdoess...@gmail.com> >> > >> wrote: >> > >> >> > >> > +1 to cut an RC. >> > >> > >> > >> > Here are a couple of PRs that could maybe go in >> > >> > >> > >> > https://github.com/apache/storm/pull/2719 >> > >> > https://github.com/apache/storm/pull/2800 (this one requires some >> > >> changes, >> > >> > but we should be able to fix it pretty quickly) >> > >> > also would like to get https://github.com/apache/storm/pull/2805 >> > >> reviewed, >> > >> > it might change some public methods. >> > >> > >> > >> > Other than that, we should try to remove as much deprecated code >> as we >> > >> can >> > >> > before release >> > >> > >> > >> > https://issues.apache.org/jira/browse/STORM-2947 >> > >> > >> > >> > Den man. 10. sep. 2018 kl. 21.59 skrev Alexandre Vermeerbergen < >> > >> > avermeerber...@gmail.com>: >> > >> > >> > >> > > +1 for an Storm 2.0 as soon as possible, let's jump into the >> future >> > :) >> > >> > > Le lun. 10 sept. 2018 à 21:50, Kishorkumar Patil >> > >> > > a écrit : >> > >> > > > >> > >> > > > Looking into all issues reported under epic >> > >> > > > https://issues.apache.org/jira/browse/STORM-2714 are >> > >> resolved/closed. >> > >> > I >>
[GitHub] storm pull request #1337: STORM-1475: Add storm-elasticsearch2 module
Github user dossett closed the pull request at: https://github.com/apache/storm/pull/1337 ---
