[jira] [Created] (STORM-2936) Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch (targeted for 1.1.2)
Jungtaek Lim created STORM-2936: --- Summary: Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch (targeted for 1.1.2) Key: STORM-2936 URL: https://issues.apache.org/jira/browse/STORM-2936 Project: Apache Storm Issue Type: Task Components: storm-kafka-client Affects Versions: 1.1.2 Reporter: Jungtaek Lim Assignee: Jungtaek Lim This is to track the effort of syncing up divergence between storm-kafka-client 1.x-branch and 1.1.x-branch so that critical fixes can be go in 1.1.x-branch as well. Linking discussion thread: [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2936) Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch
[ https://issues.apache.org/jira/browse/STORM-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2936: Summary: Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch (was: Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch (targeted for 1.1.2)) > Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch > - > > Key: STORM-2936 > URL: https://issues.apache.org/jira/browse/STORM-2936 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.1.2 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.1.x-branch so that critical fixes can be > go in 1.1.x-branch as well. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
Jungtaek Lim created STORM-2937: --- Summary: Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch Key: STORM-2937 URL: https://issues.apache.org/jira/browse/STORM-2937 Project: Apache Storm Issue Type: Task Components: storm-kafka-client Affects Versions: 1.0.6 Reporter: Jungtaek Lim This is to track the effort of syncing up divergence between storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be go in 1.0.x-branch as well. Note that it can modify storm-core as well (unlike backward-incompatible way but not 100% sure), so we should make a decision whether we allow the change in bugfix version line. Linking discussion thread: [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2913) STORM-2844 made autocommit and at-most-once storm-kafka-client spouts log warnings on every emit, because those modes don't commit the right metadata to Kafka
[ https://issues.apache.org/jira/browse/STORM-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2913. - Resolution: Fixed Fix Version/s: 1.2.0 > STORM-2844 made autocommit and at-most-once storm-kafka-client spouts log > warnings on every emit, because those modes don't commit the right metadata > to Kafka > -- > > Key: STORM-2913 > URL: https://issues.apache.org/jira/browse/STORM-2913 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Blocker > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Time Spent: 5.5h > Remaining Estimate: 0h > > The mechanism added in https://issues.apache.org/jira/browse/STORM-2844 to > allow us to check whether a committed offset was committed by the currently > running topology requires that we commit some metadata along with the offset. > We are using this metadata for two things: Only applying the > FirstPollOffsetStrategy when the topology is deployed, rather than when the > worker is restarted, and an (IMO fairly unimportant) runtime check that the > spout offset tracking is not in a bad state. > Autocommit spouts don't include this metadata, and we also don't include it > when committing offsets in at-most-once mode. We can fix at-most-once by > switching to committing a custom OffsetAndMetadata, rather than using the > no-arg commitSync variant. > I'm not sure what we should do to fix the autocommit case. There doesn't seem > to be a way to include metadata in autocommits, so I don't think we can > support this mechanism for autocommits. > If we can't fix the autocommit case, I see two options for fixing this: > * Make doSeek have the old behavior for autocommits only (i.e. apply the > FirstPollOffsetStrategy on every worker restart), and keep the new behavior > for at-least-once/at-most-once. I think this behavior could be a little > confusing. > * Revert doSeek to the old behavior in all cases, and throw out the runtime > check that uses the metadata. This also isn't a great option, because the new > seek behavior is more useful than restarting on every worker reboot. > What do you think [~hmclouro]? I'm leaning toward the first option. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2914) Remove enable.auto.commit support from storm-kafka-client
[ https://issues.apache.org/jira/browse/STORM-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2914. - Resolution: Fixed Fix Version/s: 1.2.0 > Remove enable.auto.commit support from storm-kafka-client > - > > Key: STORM-2914 > URL: https://issues.apache.org/jira/browse/STORM-2914 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Attachments: storm-kafka-modes.ods > > Time Spent: 4h 40m > Remaining Estimate: 0h > > The enable.auto.commit option causes the KafkaConsumer to periodically commit > the latest offsets it has returned from poll(). It is convenient for use > cases where messages are polled from Kafka and processed synchronously, in a > loop. > Due to https://issues.apache.org/jira/browse/STORM-2913 we'd really like to > store some metadata in Kafka when the spout commits. This is not possible > with enable.auto.commit. I took at look at what that setting actually does, > and it just causes the KafkaConsumer to call commitAsync during poll (and > during a few other operations, e.g. close and assign) with some interval. > Ideally I'd like to get rid of ProcessingGuarantee.NONE, since I think > ProcessingGuarantee.AT_MOST_ONCE covers the same use cases, and is likely > almost as fast. The primary difference between them is that AT_MOST_ONCE > commits synchronously. > If we really want to keep ProcessingGuarantee.NONE, I think we should make > our ProcessingGuarantee.NONE setting cause the spout to call commitAsync > after poll, and never use the enable.auto.commit option. This allows us to > include metadata in the commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2943) Binary distribution includes storm-kafka-monitor source/javadoc in toollib directory
Jungtaek Lim created STORM-2943:
---
Summary: Binary distribution includes storm-kafka-monitor
source/javadoc in toollib directory
Key: STORM-2943
URL: https://issues.apache.org/jira/browse/STORM-2943
Project: Apache Storm
Issue Type: Bug
Components: storm-kafka-monitor
Affects Versions: 1.1.1, 1.1.0, 1.2.0, 1.1.2
Reporter: Alexandre Vermeerbergen
Assignee: P. Taylor Goetz
Quoting Alexandre's RC3 vote:
{quote}I hate to be the one who always give bad news, but as a matter of
facts, Storm 1.2.0 RC3 installation from binary artifacts (both
apache-storm-1.2.0-src.tar.gz and apache-storm-1.2.0.zip) leads to "by
default KO Kafka monitor" in Nimbus UI (which dirty exceptions in
ui.log)
Here's for example what I get from apache-storm-1.2.0-src.tar.gz
downloaded from
[https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz]:
$ tar ztvf apache-storm-1.2.0.tar.gz apache-storm-1.2.0/toollib
-rwxrwxrwx ptgoetz/staff 16999 2018-02-06 21:22
apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-sources.jar
-rwxrwxrwx ptgoetz/staff 93461 2018-02-06 21:22
apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-javadoc.jar
-rwxrwxrwx ptgoetz/staff 21591320 2018-02-06 21:22
apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0.jar
And here's what I see in ui.log:
org.apache.storm.kafka.spout.KafkaSpout
2018-02-07 16:49:57.153 o.a.s.u.TopologySpoutLag qtp1997623038-18
[WARN] Exception message:Error: Could not find or load main class
.usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
org.apache.storm.utils.ShellUtils$ExitCodeException: Error: Could not
find or load main class
.usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
at org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:231)
~[storm-core-1.2.0.jar:1.2.0]
at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371)
~[storm-core-1.2.0.jar:1.2.0]
at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461)
~[storm-core-1.2.0.jar:1.2.0]
at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.utils.TopologySpoutLag.getLagResultForNewKafkaSpout(TopologySpoutLag.java:189)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:57)
~[storm-core-1.2.0.jar:1.2.0]
at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805)
~[storm-core-1.2.0.jar:1.2.0]
at org.apache.storm.ui.core$fn__9586.invoke(core.clj:1165)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.compojure.core$make_route$fn__5979.invoke(core.clj:100)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.compojure.core$if_route$fn__5967.invoke(core.clj:46)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.compojure.core$if_method$fn__5960.invoke(core.clj:31)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.compojure.core$routing$fn__5985.invoke(core.clj:113)
~[storm-core-1.2.0.jar:1.2.0]
at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113)
~[storm-core-1.2.0.jar:1.2.0]
at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
at
org.apache.storm.shade.compojure.core$routes$fn__5989.invoke(core.clj:118)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8894.invoke(cors.clj:149)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8841.invoke(json.clj:56)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7904.invoke(reload.clj:22)
~[storm-core-1.2.0.jar:1.2.0]
at
org.apache.storm.ui.helpers$requests_middleware$fn__6874.invoke(helpers.clj:52)
~[storm-core-1.2.0.jar:1.2.0]
Deleting the extraneous storm-kafka-monitor-1.2.0-sources.jar and
storm-kafka-monitor-1.2.0-javadoc.jar file, then restarting Nimbus
solves the issue.
However, binaries artifacts should be as clean as possible, isn't it ?
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354746#comment-16354746
]
Jungtaek Lim edited comment on STORM-2937 at 2/6/18 11:53 PM:
--
[~erikdw]
That was fairly simple for 1.1.x-branch, whereas I admit that would not be that
simple for 1.0.x-branch.
Step to copy:
# git checkout 1.x-branch
# mvn clean
# cd external
# tar cvfz storm-kafka-client.tar.gz storm-kafka-client
# mv storm-kafka-client.tar.gz ~
# ... same for examples/storm-kafka-client-examples ...
# git checkout 1.0.x-branch
# git checkout -b {{}}
# cd external
# rm -rf storm-kafka-client
# cp ~/storm-kafka-client.tar.gz .
# tar xvfz storm-kafka-client.tar.gz
# change parent version in storm-kafka-client/pom.xml
# rm -rf storm-kafka-client.tar.gz
# ... same for examples/storm-kafka-client-examples ...
Step to iterate "find and apply":
# mvn clean package -DskipTests (to save time)
# if it fails from compilation, look at the message and see which classes are
missing/methods are missing
# open browser and open github apache/storm, and find the class from 1.x-branch
# if the class is just missing, copy the content and create file in IDE (so I
was utilizing console/browser/IDE)
# if the method is missing or method signature is different, open blame from
github to find the commit to check whether it is solely modified along with
storm-kafka-client. And modify class manually via copying the content.
# build again (back to 1)
I haven't met the case which storm-kafka-client relies on storm-core which
changes are not introduced from storm-kafka-client yet. If there's a case,
that's the thing to discuss, since we are porting back other issues as well.
was (Author: kabhwan):
[~erikdw]
That was fairly simple for 1.1.x-branch, whereas I admit that would not be that
simple for 1.0.x-branch.
Step to copy:
# git checkout 1.x-branch
# mvn clean
# cd external
# tar cvfz storm-kafka-client.tar.gz storm-kafka-client
# mv storm-kafka-client.tar.gz ~
# ... same for examples/storm-kafka-client-examples ...
# git checkout 1.1.x-branch
# git checkout -b {{}}
# cd external
# rm -rf storm-kafka-client
# cp ~/storm-kafka-client.tar.gz .
# tar xvfz storm-kafka-client.tar.gz
# change parent version in storm-kafka-client/pom.xml
# rm -rf storm-kafka-client.tar.gz
# ... same for examples/storm-kafka-client-examples ...
Step to iterate "find and apply":
# mvn clean package -DskipTests (to save time)
# if it fails from compilation, look at the message and see which classes are
missing/methods are missing
# open browser and open github apache/storm, and find the class from 1.x-branch
# if the class is just missing, copy the content and create file in IDE (so I
was utilizing console/browser/IDE)
# if the method is missing or method signature is different, open blame from
github to find the commit to check whether it is solely modified along with
storm-kafka-client. And modify class manually via copying the content.
# build again (back to 1)
I haven't met the case which storm-kafka-client relies on storm-core which
changes are not introduced from storm-kafka-client yet. If there's a case,
that's the thing to discuss, since we are porting back other issues as well.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354746#comment-16354746
]
Jungtaek Lim edited comment on STORM-2937 at 2/7/18 12:12 AM:
--
[~erikdw]
That was fairly simple for 1.1.x-branch, whereas I admit that would not be that
simple for 1.0.x-branch.
Step to copy:
# git checkout 1.x-branch
# mvn clean
# cd external
# tar cvfz storm-kafka-client.tar.gz storm-kafka-client
# mv storm-kafka-client.tar.gz ~
# ... same for examples/storm-kafka-client-examples ...
# git checkout 1.0.x-branch
# git checkout -b {{}}
# cd external
# rm -rf storm-kafka-client
# cp ~/storm-kafka-client.tar.gz .
# tar xvfz storm-kafka-client.tar.gz
# change parent version in storm-kafka-client/pom.xml
# rm -rf storm-kafka-client.tar.gz
# ... same for examples/storm-kafka-client-examples ...
Step to iterate "find and apply":
# mvn clean package -DskipTests (to save time)
# if it fails from compilation, look at the message and see which classes are
missing/methods are missing
# open browser and open github apache/storm, and find the class from 1.x-branch
# if the class is just missing, open history from github to check whether it
is solely modified along with storm-kafka-client. copy the content and create
file in IDE (so I was utilizing console/browser/IDE)
# if the method is missing or method signature is different, open blame from
github to find the commit to check whether it is solely modified along with
storm-kafka-client. And modify class manually via copying the content.
# build again (back to 1)
I haven't met the case which storm-kafka-client relies on storm-core which
changes are not introduced from storm-kafka-client yet. If there's a case,
that's the thing to discuss, since we are porting back other issues as well.
was (Author: kabhwan):
[~erikdw]
That was fairly simple for 1.1.x-branch, whereas I admit that would not be that
simple for 1.0.x-branch.
Step to copy:
# git checkout 1.x-branch
# mvn clean
# cd external
# tar cvfz storm-kafka-client.tar.gz storm-kafka-client
# mv storm-kafka-client.tar.gz ~
# ... same for examples/storm-kafka-client-examples ...
# git checkout 1.0.x-branch
# git checkout -b {{}}
# cd external
# rm -rf storm-kafka-client
# cp ~/storm-kafka-client.tar.gz .
# tar xvfz storm-kafka-client.tar.gz
# change parent version in storm-kafka-client/pom.xml
# rm -rf storm-kafka-client.tar.gz
# ... same for examples/storm-kafka-client-examples ...
Step to iterate "find and apply":
# mvn clean package -DskipTests (to save time)
# if it fails from compilation, look at the message and see which classes are
missing/methods are missing
# open browser and open github apache/storm, and find the class from 1.x-branch
# if the class is just missing, copy the content and create file in IDE (so I
was utilizing console/browser/IDE)
# if the method is missing or method signature is different, open blame from
github to find the commit to check whether it is solely modified along with
storm-kafka-client. And modify class manually via copying the content.
# build again (back to 1)
I haven't met the case which storm-kafka-client relies on storm-core which
changes are not introduced from storm-kafka-client yet. If there's a case,
that's the thing to discuss, since we are porting back other issues as well.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2853) Deactivated topologies cause high cpu utilization
[ https://issues.apache.org/jira/browse/STORM-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354766#comment-16354766 ] Jungtaek Lim commented on STORM-2853: - [~vorin] I guess you're saying stateful implemented by user side: I don't think they can use intermediate result while crash occurs, since the result is not guaranteed to be not corrupted. If you're saying stateful supported by Storm, it doesn't leverage tick tuple from bolt side. Please elaborate your example a bit if my explanation doesn't fall into your case. > Deactivated topologies cause high cpu utilization > - > > Key: STORM-2853 > URL: https://issues.apache.org/jira/browse/STORM-2853 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.1.0 >Reporter: Stuart >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.6 > > Attachments: exclamation.zip > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The issue is there is high cpu usage for deactivated apache storm topologies. > I can reliably re-create the issue using the steps below but I haven't > identified the exact cause or a solution yet. > The environment is a storm cluster on which 1 topology is running (The > topology is extremely simple, I used the exclamation example). It is > INACTIVE. Initially there is normal CPU usage. However, when I kill all > topology JVM processes on all supervisors and let Storm restart them again, I > find that some time later (~9 hours) the CPU usage per JVM process rises to > nearly 100%. I have tested an ACTIVE topology and this does not happen with > it. I have also tested more than one topology and observe the same results > when they're in the INACTIVE state. > ***Steps to re-create:*** > 1. Run 1 topology on an Apache Storm cluster > 2. Deactivate it > 3. Kill **all** topology JVM processes on all supervisors (Storm will > restart them) > 4. Observe the CPU usage on Supervisors rise to nearly 100% for all > **INACTIVE** topology JVM processes. > ***Environment*** > Apache Storm 1.1.0 running on 3 VMs (1 nimbus and 2 supervisors). > Cluster Summary: > - Supervisors: 2 > - Used Slots: 2 > - Available Slots: 38 > - Total Slots: 40 > - Executors: 50 > - Tasks: 50 > the topology has 2 workers and 50 executors/tasks (threads). > ***Investigation so far:*** > Apart from being able to reliably re-create the issue, I have identified, for > the affected topology JVM process, the threads using the most CPU. There are > 102 threads total in the process, 97 blocked, 5 IN_NATIVE. The threads using > the most CPU are identical and there are 23 of them (all in BLOCKED state): > Thread 28558: (state = BLOCKED) > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > information may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, > line=338 (Compiled frame) > - com.lmax.disruptor.MultiProducerSequencer.next(int) @bci=82, line=136 > (Compiled frame) > - com.lmax.disruptor.RingBuffer.next(int) @bci=5, line=260 (Interpreted > frame) > - > org.apache.storm.utils.DisruptorQueue.publishDirect(java.util.ArrayList, > boolean) @bci=18, line=517 (Interpreted frame) > - > org.apache.storm.utils.DisruptorQueue.access$1000(org.apache.storm.utils.DisruptorQueue, > java.util.ArrayList, boolean) @bci=3, line=61 (Interpreted frame) > - > org.apache.storm.utils.DisruptorQueue$ThreadLocalBatcher.flush(boolean) > @bci=50, line=280 (Interpreted frame) > - org.apache.storm.utils.DisruptorQueue$Flusher.run() @bci=55, line=303 > (Interpreted frame) > - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 > (Compiled frame) > - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled > frame) > - > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=95, line=1142 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) > I identified this thread by using `jstack` to get a thread dump for the > process: > > jstack -F > jstack.txt > and `top` to identify the threads within the process using the most CPU: > top -H -p -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354781#comment-16354781
]
Jungtaek Lim commented on STORM-2937:
-
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame". For guaranteeing quality of commit message, we can
force PR to fill the template of PR and adding it to commit body, which is what
other projects are doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but commits are not in line with effective date. Commit log
of merge commit doesn't give informative thing, and I should find JIRA,
relevant commits which are merged in with merge commit. That is less handy than
squashing and push itself without adding merge commit.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354791#comment-16354791 ] Jungtaek Lim commented on STORM-2937: - [~erikdw] Ah you need to change root pom.xml to update kafka-client version as well for 1.0.x-branch. The branch supported 0.9, whereas it now requires 0.10.1.0. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354679#comment-16354679 ] Jungtaek Lim commented on STORM-2937: - And I'm sorry but "if possible", I'm eager to get this in ASAP in 1.0.6, since it is broken fairly long, and I don't think we should have 1.0.7 sooner. TBH, I don't want to maintain 1.0.x version line. We are now maintaining 4 version lines. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354685#comment-16354685 ] Jungtaek Lim commented on STORM-2937: - Unlike what I'm at least doing it with reading/writing, unfortunately I'm not good at listening/speaking English so if we can't understand the semantic with text, audio chat will not work too. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354746#comment-16354746
]
Jungtaek Lim commented on STORM-2937:
-
[~erikdw]
That was fairly simple for 1.1.x-branch, whereas I admit that would not be that
simple for 1.0.x-branch.
Step to copy:
# git checkout 1.x-branch
# mvn clean
# cd external
# tar cvfz storm-kafka-client.tar.gz storm-kafka-client
# mv storm-kafka-client.tar.gz ~
# ... same for examples/storm-kafka-client-examples ...
# git checkout 1.1.x-branch
# git checkout -b {{}}
# cd external
# rm -rf storm-kafka-client
# cp ~/storm-kafka-client.tar.gz .
# tar xvfz storm-kafka-client.tar.gz
# change parent version in storm-kafka-client/pom.xml
# rm -rf storm-kafka-client.tar.gz
# ... same for examples/storm-kafka-client-examples ...
Step to iterate "find and apply":
# mvn clean package -DskipTests (to save time)
# if it fails from compilation, look at the message and see which classes are
missing/methods are missing
# open browser and open github apache/storm, and find the class from 1.x-branch
# if the class is just missing, copy the content and create file in IDE (so I
was utilizing console/browser/IDE)
# if the method is missing or method signature is different, open blame from
github to find the commit to check whether it is solely modified along with
storm-kafka-client. And modify class manually via copying the content.
# build again (back to 1)
I haven't met the case which storm-kafka-client relies on storm-core which
changes are not introduced from storm-kafka-client yet. If there's a case,
that's the thing to discuss, since we are porting back other issues as well.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354781#comment-16354781
]
Jungtaek Lim edited comment on STORM-2937 at 2/7/18 12:31 AM:
--
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame", and composing a unit of commit is also what majority
of contributors/even PMCs are not doing well. Everyone has different standard
about the unit, hence that can be easily led to debate as well.
For guaranteeing quality of commit message, we can force PR to fill the
template of PR and adding it to commit body, which is what other projects are
doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but added commits under merge commit are not in line with
effective date. Commit log of merge commit doesn't give informative thing, and
I should find JIRA, relevant commits which are merged in with merge commit.
That is less handy than squashing and push itself without adding merge commit.
was (Author: kabhwan):
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame", and composing a unit of commit is also what majority
of contributors/even PMCs are not doing well. Everyone has different standard
about the unit, hence that can be easily led to debate as well.
For guaranteeing quality of commit message, we can force PR to fill the
template of PR and adding it to commit body, which is what other projects are
doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but commits are not in line with effective date. Commit log
of merge commit doesn't give informative thing, and I should find JIRA,
relevant commits which are merged in with merge commit. That is less handy than
squashing and push itself without adding merge commit.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354667#comment-16354667 ] Jungtaek Lim commented on STORM-2937: - [~erikdw] Suppose we squash all the commits being done for storm-kafka-client, it should be same as my one commit. What does it mean? I don't fix any other thing, but only find and apply divergence in storm-core between branches which is introduced from storm-kafka-client patches. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354752#comment-16354752 ] Jungtaek Lim edited comment on STORM-2937 at 2/6/18 11:59 PM: -- That's based on the assumption that modifying storm-core due to storm-kafka-client occurred less, and all of changes affect compilation. If not, the change would be close to bugfix, and the change should've filed and resolved separately. (I'm expecting we did.) was (Author: kabhwan): That's based on the assumption that modifying storm-core due to storm-kafka-client occurred less, and all of changes affect compilation. If not, the change would be close to bugfix, and the change should've filed and resolved separately. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354758#comment-16354758 ] Jungtaek Lim edited comment on STORM-2937 at 2/7/18 12:05 AM: -- That is just a sequence of steps I did which I feel very handy to do, but someone may have concerns with my assumptions and makes the steps guaranteeing more correctness but less handy. For me, any sequence of steps would be fine for me, once we confirm that storm-kafka-client/storm-kafka-client-example are copied correctly, and tests are succeed. was (Author: kabhwan): That is just a sequence of steps I did which I feel very handy to do, but someone may have concerns with my assumptions and make the steps guaranteeing more correctness but less handy. For me, any sequence of steps would be fine for me, once we confirm that storm-kafka-client/storm-kafka-client-example are copied correctly, and tests are succeed. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354758#comment-16354758 ] Jungtaek Lim commented on STORM-2937: - That is just a sequence of steps I did which I feel very handy to do, but someone may have concerns with my assumptions and make the steps guaranteeing more correctness but less handy. For me, any sequence of steps would be fine for me, once we confirm that storm-kafka-client/storm-kafka-client-example are copied correctly, and tests are succeed. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354752#comment-16354752 ] Jungtaek Lim commented on STORM-2937: - That's based on the assumption that modifying storm-core due to storm-kafka-client occurred less, and the change would affect compilation. If not the change would be close to bugfix, and the change should've filed and resolved separately. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[ https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354752#comment-16354752 ] Jungtaek Lim edited comment on STORM-2937 at 2/6/18 11:58 PM: -- That's based on the assumption that modifying storm-core due to storm-kafka-client occurred less, and all of changes affect compilation. If not, the change would be close to bugfix, and the change should've filed and resolved separately. was (Author: kabhwan): That's based on the assumption that modifying storm-core due to storm-kafka-client occurred less, and the change would affect compilation. If not the change would be close to bugfix, and the change should've filed and resolved separately. > Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch > - > > Key: STORM-2937 > URL: https://issues.apache.org/jira/browse/STORM-2937 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.0.6 >Reporter: Jungtaek Lim >Assignee: Erik Weathers >Priority: Blocker > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be > go in 1.0.x-branch as well. > Note that it can modify storm-core as well (unlikely in a > backwards-incompatible way but not 100% sure), so we should make a decision > whether we allow the change in bugfix version line. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354781#comment-16354781
]
Jungtaek Lim edited comment on STORM-2937 at 2/7/18 12:27 AM:
--
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame", and composing a unit of commit is also what majority
of contributors/even PMCs are not doing well. Everyone has different standard
about the unit, hence that can be easily led to debate as well.
For guaranteeing quality of commit message, we can force PR to fill the
template of PR and adding it to commit body, which is what other projects are
doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but commits are not in line with effective date. Commit log
of merge commit doesn't give informative thing, and I should find JIRA,
relevant commits which are merged in with merge commit. That is less handy than
squashing and push itself without adding merge commit.
was (Author: kabhwan):
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame". For guaranteeing quality of commit message, we can
force PR to fill the template of PR and adding it to commit body, which is what
other projects are doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but commits are not in line with effective date. Commit log
of merge commit doesn't give informative thing, and I should find JIRA,
relevant commits which are merged in with merge commit. That is less handy than
squashing and push itself without adding merge commit.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354781#comment-16354781
]
Jungtaek Lim edited comment on STORM-2937 at 2/7/18 12:32 AM:
--
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame", and composing a unit of commit is also what majority
of contributors/even PMCs are not doing well. Everyone has different standard
about the unit, hence that can be easily led to debate as well.
For guaranteeing quality of commit message, we can force PR to fill the
template of PR and adding it to commit body (via merge script), which is what
other projects are doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but added commits under merge commit are not in line with
effective date. Commit log of merge commit doesn't give informative thing, and
I should find JIRA, relevant commits which are merged in with merge commit.
That is less handy than squashing and push itself without adding merge commit.
was (Author: kabhwan):
{quote}Gigantic commits are often a problem for this, especially given how bad
the majority of folks are at writing commit messages and code comments.
{quote}
I'm in favor of squashing commits for the same reason. I don't think commits
like "addressing review comments" will help us finding the root cause while
debugging and "git blame", and composing a unit of commit is also what majority
of contributors/even PMCs are not doing well. Everyone has different standard
about the unit, hence that can be easily led to debate as well.
For guaranteeing quality of commit message, we can force PR to fill the
template of PR and adding it to commit body, which is what other projects are
doing and makes more sense than relying on contributors.
Similar feeling with merge commit, since it enables reverting whole the commits
at once (per PR), but added commits under merge commit are not in line with
effective date. Commit log of merge commit doesn't give informative thing, and
I should find JIRA, relevant commits which are merged in with merge commit.
That is less handy than squashing and push itself without adding merge commit.
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354814#comment-16354814
]
Jungtaek Lim commented on STORM-2937:
-
[~erikdw]
{quote}The main difference in my strategy being that I plan to have a commit of
what was manually copied back without thought or changes, and then a separate
commit of what I did to fix these various issues.
{quote}
That also comes from difference of the standard of unit of commit between you
and me. I wouldn't see commit which doesn't compile. Ideally I wouldn't see
commit either which doesn't pass tests, but life is not easy. :) Your commit
list is even OK for me while reviewing, but if I have a chance to merge
whatever way I want, I might request squashing at the end to meet the standard
for me.
I think that can be sorted out in review phase with hearing others voices as
well. Please take any approach you feel convenient. Thanks again for taking up
hard work!
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (STORM-2937) Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
[
https://issues.apache.org/jira/browse/STORM-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354814#comment-16354814
]
Jungtaek Lim edited comment on STORM-2937 at 2/7/18 1:04 AM:
-
[~erikdw]
{quote}The main difference in my strategy being that I plan to have a commit of
what was manually copied back without thought or changes, and then a separate
commit of what I did to fix these various issues.
{quote}
That also comes from difference of the standard of unit of commit between you
and me. I wouldn't accept commit which doesn't compile. Ideally I wouldn't
accept commit either which doesn't pass tests, but life is not easy. :) Your
commit list is even OK for me while reviewing, but if I have a chance to merge
whatever way I want, I might request squashing at the end to meet the standard
for me.
I think that can be sorted out in review phase with hearing others voices as
well. Please take any approach you feel convenient. Thanks again for taking up
hard work!
was (Author: kabhwan):
[~erikdw]
{quote}The main difference in my strategy being that I plan to have a commit of
what was manually copied back without thought or changes, and then a separate
commit of what I did to fix these various issues.
{quote}
That also comes from difference of the standard of unit of commit between you
and me. I wouldn't see commit which doesn't compile. Ideally I wouldn't see
commit either which doesn't pass tests, but life is not easy. :) Your commit
list is even OK for me while reviewing, but if I have a chance to merge
whatever way I want, I might request squashing at the end to meet the standard
for me.
I think that can be sorted out in review phase with hearing others voices as
well. Please take any approach you feel convenient. Thanks again for taking up
hard work!
> Overwrite storm-kafka-client 1.x-branch into 1.0.x-branch
> -
>
> Key: STORM-2937
> URL: https://issues.apache.org/jira/browse/STORM-2937
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka-client
>Affects Versions: 1.0.6
>Reporter: Jungtaek Lim
>Assignee: Erik Weathers
>Priority: Blocker
>
> This is to track the effort of syncing up divergence between
> storm-kafka-client 1.x-branch and 1.0.x-branch so that critical fixes can be
> go in 1.0.x-branch as well.
> Note that it can modify storm-core as well (unlikely in a
> backwards-incompatible way but not 100% sure), so we should make a decision
> whether we allow the change in bugfix version line.
> Linking discussion thread:
> [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-2944) Eliminate these deprecated methods in Storm 3
[
https://issues.apache.org/jira/browse/STORM-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2944:
Affects Version/s: 3.0.0
> Eliminate these deprecated methods in Storm 3
> -
>
> Key: STORM-2944
> URL: https://issues.apache.org/jira/browse/STORM-2944
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-client
>Affects Versions: 3.0.0
>Reporter: Roshan Naik
>Priority: Major
>
> In Storm 2 we deprecated these methods. These methods were retained to allow
> storm 2.x nimbus & supervisor to manage older Storm 1.x workers.
> These are the methods in IStormClusterState.java
> {code}
> /** @deprecated: In Storm 2.0. Retained for enabling transition from 1.x.
> Will be removed soon. */
> @Deprecated
> boolean topologyBackpressure(String stormId, long timeoutMs, Runnable
> callback);
> /** @deprecated: In Storm 2.0. Retained for enabling transition from 1.x.
> Will be removed soon. */
> @Deprecated
> void setupBackpressure(String stormId);
> /** @deprecated: In Storm 2.0. Retained for enabling transition from 1.x.
> Will be removed soon. */
> @Deprecated
> void removeBackpressure(String stormId);
> /** @deprecated: In Storm 2.0. Retained for enabling transition from 1.x.
> Will be removed soon. */
> @Deprecated
> void removeWorkerBackpressure(String stormId, String node, Long port);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (STORM-2841) testNoAcksIfFlushFails UT fails with NullPointerException
[
https://issues.apache.org/jira/browse/STORM-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned STORM-2841:
---
Assignee: Jungtaek Lim
> testNoAcksIfFlushFails UT fails with NullPointerException
> -
>
> Key: STORM-2841
> URL: https://issues.apache.org/jira/browse/STORM-2841
> Project: Apache Storm
> Issue Type: Test
>Reporter: Yesha Vora
>Assignee: Jungtaek Lim
>Priority: Major
>
> testNoAcksIfFlushFails UT fails with NPE
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.storm.hive.bolt.HiveBolt.flushAllWriters(HiveBolt.java:207)
> at
> org.apache.storm.hive.bolt.TestHiveBolt.testNoAcksIfFlushFails(TestHiveBolt.java:322){code}
> {code}
> Standard Output
> 60488 [main] INFO h.metastore - Mestastore configuration
> hive.metastore.filter.hook changed from
> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
> to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Shutting down the object
> store...
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Metastore shutdown complete.
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 60497 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/75acc3af-f90e-4ab2-888c-a8ad29383128_resources
> 60499 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60501 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60503 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128/_tmp_space.db
> 60793 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table_names_by_filter: db
> = testdb, filter =
> 60793 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table_names_by_filter: db = testdb, filter =
> 60847 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Opening raw store with
> implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 60848 [main] INFO o.a.h.h.m.ObjectStore - ObjectStore, initialize called
> 60864 [main] INFO D.Persistence - Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 60864 [main] INFO D.Persistence - Property datanucleus.cache.level2 unknown
> - will be ignored
> 61370 [main] INFO o.a.h.h.m.ObjectStore - Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 61711 [main] INFO o.a.h.h.m.MetaStoreDirectSql - Using direct SQL,
> underlying DB is DERBY
> 61711 [main] INFO o.a.h.h.m.ObjectStore - Initialized ObjectStore
> 61716 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table : db=testdb
> tbl=test_table
> 61716 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table : db=testdb tbl=test_table
> 61738 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_table : db=testdb
> tbl=test_table
> 61738 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_table : db=testdb tbl=test_table
> 62053 [main] INFO h.m.hivemetastoressimpl - deleting
> raw:/tmp/junit2860976527054408251/testdb.db/test_table
> 62062 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_database: testdb
> 62063 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_database: testdb
> 62068 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_database: testdb
> 62068 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_database: testdb
> 62069 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_all_tables: db=testdb
> 62069 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_all_tables: db=testdb
> 62073 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_functions: db=testdb pat=*
> 62073 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_functions: db=testdb pat=*
> 62103 [main] INFO o.a.h.h.m.ObjectStore - Dropping database testdb along
> with all tables
> 62141 [main] INFO h.m.hivemetastoressimpl - deleting
> raw:/tmp/junit2860976527054408251/testdb.db
> 62148 [main] INFO o.a.h.h.q.l.PerfLogger - from=org.apache.hadoop.hive.ql.Driver>
> 62148 [main] INFO o.a.h.h.q.l.PerfLogger - from=org.apache.hadoop.hive.ql.Driver>
> 62149 [main] INFO o.a.h.h.q.Driver - We are setting the hadoop caller
> context from to
[jira] [Commented] (STORM-2841) testNoAcksIfFlushFails UT fails with NullPointerException
[
https://issues.apache.org/jira/browse/STORM-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360448#comment-16360448
]
Jungtaek Lim commented on STORM-2841:
-
This is actually super odd, because it shouldn't call the real method while
mocking spy object partially but it is being called.
We could avoid NPE via calling prepare() earlier, but if the error indicates
that the mocking is not working in that case, the test will fail anyway.
Maybe we can just try out replacing partial mock with extending class, since
without mocking that method there's no need to spy the class.
> testNoAcksIfFlushFails UT fails with NullPointerException
> -
>
> Key: STORM-2841
> URL: https://issues.apache.org/jira/browse/STORM-2841
> Project: Apache Storm
> Issue Type: Test
>Reporter: Yesha Vora
>Priority: Major
>
> testNoAcksIfFlushFails UT fails with NPE
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.storm.hive.bolt.HiveBolt.flushAllWriters(HiveBolt.java:207)
> at
> org.apache.storm.hive.bolt.TestHiveBolt.testNoAcksIfFlushFails(TestHiveBolt.java:322){code}
> {code}
> Standard Output
> 60488 [main] INFO h.metastore - Mestastore configuration
> hive.metastore.filter.hook changed from
> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
> to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Shutting down the object
> store...
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Metastore shutdown complete.
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 60497 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/75acc3af-f90e-4ab2-888c-a8ad29383128_resources
> 60499 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60501 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60503 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128/_tmp_space.db
> 60793 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table_names_by_filter: db
> = testdb, filter =
> 60793 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table_names_by_filter: db = testdb, filter =
> 60847 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Opening raw store with
> implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 60848 [main] INFO o.a.h.h.m.ObjectStore - ObjectStore, initialize called
> 60864 [main] INFO D.Persistence - Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 60864 [main] INFO D.Persistence - Property datanucleus.cache.level2 unknown
> - will be ignored
> 61370 [main] INFO o.a.h.h.m.ObjectStore - Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 61711 [main] INFO o.a.h.h.m.MetaStoreDirectSql - Using direct SQL,
> underlying DB is DERBY
> 61711 [main] INFO o.a.h.h.m.ObjectStore - Initialized ObjectStore
> 61716 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table : db=testdb
> tbl=test_table
> 61716 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table : db=testdb tbl=test_table
> 61738 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_table : db=testdb
> tbl=test_table
> 61738 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_table : db=testdb tbl=test_table
> 62053 [main] INFO h.m.hivemetastoressimpl - deleting
> raw:/tmp/junit2860976527054408251/testdb.db/test_table
> 62062 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_database: testdb
> 62063 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_database: testdb
> 62068 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_database: testdb
> 62068 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_database: testdb
> 62069 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_all_tables: db=testdb
> 62069 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_all_tables: db=testdb
> 62073 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_functions: db=testdb pat=*
> 62073 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_functions: db=testdb pat=*
> 62103 [main] INFO o.a.h.h.m.ObjectStore - Dropping database testdb along
> with all tables
[jira] [Resolved] (STORM-2940) Increase the CAPACITY value in LoadAwareShuffleGrouping
[
https://issues.apache.org/jira/browse/STORM-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2940.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~ethanli], I merged into master.
> Increase the CAPACITY value in LoadAwareShuffleGrouping
> ---
>
> Key: STORM-2940
> URL: https://issues.apache.org/jira/browse/STORM-2940
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We have seen exceptions from workers
> {code:java}
> java.lang.IllegalArgumentException: bound must be positive
> {code}
> The stack trace points to LoadAwareShuffleGrouping.java#L234:
> {code:java}
> //in case we didn't fill in enough
> for (; currentIdx < CAPACITY; currentIdx++) {
> prepareChoices[currentIdx] = prepareChoices[random.nextInt(currentIdx)];
> }
> {code}
> This is because in some situation,
> {code:java}
> int count = (int) ((indexAndWeights.weight / (double) weightSum) * CAPACITY);
> {code}
> the above code will be 0 during the whole for-loop. For example, when there
> are more than 1000 (which is CAPACITY) downstream tasks, say 1001, and each
> task has weight 1. Then for every target task, count =(int) ( (1 / 1001.0) *
> 1000) = 0.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-2841) testNoAcksIfFlushFails UT fails with NullPointerException
[
https://issues.apache.org/jira/browse/STORM-2841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2841.
-
Resolution: Fixed
Fix Version/s: 2.0.0
1.0.7
1.1.3
1.2.1
Merged into master, 1.x, 1.1.x, 1.0.x branches.
> testNoAcksIfFlushFails UT fails with NullPointerException
> -
>
> Key: STORM-2841
> URL: https://issues.apache.org/jira/browse/STORM-2841
> Project: Apache Storm
> Issue Type: Test
>Reporter: Yesha Vora
>Assignee: Jungtaek Lim
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.2.1, 1.1.3, 1.0.7, 2.0.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> testNoAcksIfFlushFails UT fails with NPE
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.storm.hive.bolt.HiveBolt.flushAllWriters(HiveBolt.java:207)
> at
> org.apache.storm.hive.bolt.TestHiveBolt.testNoAcksIfFlushFails(TestHiveBolt.java:322){code}
> {code}
> Standard Output
> 60488 [main] INFO h.metastore - Mestastore configuration
> hive.metastore.filter.hook changed from
> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
> to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Shutting down the object
> store...
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 60489 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Metastore shutdown complete.
> 60489 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 60497 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/75acc3af-f90e-4ab2-888c-a8ad29383128_resources
> 60499 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60501 [main] INFO o.a.h.h.q.s.SessionState - Created local directory:
> /tmp/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128
> 60503 [main] INFO o.a.h.h.q.s.SessionState - Created HDFS directory:
> /tmp/hive/nobody/75acc3af-f90e-4ab2-888c-a8ad29383128/_tmp_space.db
> 60793 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table_names_by_filter: db
> = testdb, filter =
> 60793 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table_names_by_filter: db = testdb, filter =
> 60847 [main] INFO o.a.h.h.m.HiveMetaStore - 0: Opening raw store with
> implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 60848 [main] INFO o.a.h.h.m.ObjectStore - ObjectStore, initialize called
> 60864 [main] INFO D.Persistence - Property
> hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 60864 [main] INFO D.Persistence - Property datanucleus.cache.level2 unknown
> - will be ignored
> 61370 [main] INFO o.a.h.h.m.ObjectStore - Setting MetaStore object pin
> classes with
> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 61711 [main] INFO o.a.h.h.m.MetaStoreDirectSql - Using direct SQL,
> underlying DB is DERBY
> 61711 [main] INFO o.a.h.h.m.ObjectStore - Initialized ObjectStore
> 61716 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_table : db=testdb
> tbl=test_table
> 61716 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_table : db=testdb tbl=test_table
> 61738 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_table : db=testdb
> tbl=test_table
> 61738 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_table : db=testdb tbl=test_table
> 62053 [main] INFO h.m.hivemetastoressimpl - deleting
> raw:/tmp/junit2860976527054408251/testdb.db/test_table
> 62062 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_database: testdb
> 62063 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_database: testdb
> 62068 [main] INFO o.a.h.h.m.HiveMetaStore - 0: drop_database: testdb
> 62068 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=drop_database: testdb
> 62069 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_all_tables: db=testdb
> 62069 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_all_tables: db=testdb
> 62073 [main] INFO o.a.h.h.m.HiveMetaStore - 0: get_functions: db=testdb pat=*
> 62073 [main] INFO o.a.h.h.m.H.audit - ugi=nobody ip=unknown-ip-addr
> cmd=get_functions: db=testdb pat=*
> 62103 [main] INFO o.a.h.h.m.ObjectStore - Dropping database testdb along
> with all tables
> 62141 [main] INFO h.m.hivemetastoressimpl - deleting
>
[jira] [Updated] (STORM-2945) Nail down and document how to support background emits in Spouts and Bolts
[ https://issues.apache.org/jira/browse/STORM-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2945: Affects Version/s: 2.0.0 > Nail down and document how to support background emits in Spouts and Bolts > -- > > Key: STORM-2945 > URL: https://issues.apache.org/jira/browse/STORM-2945 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2945) Nail down and document how to support background emits in Spouts and Bolts
[ https://issues.apache.org/jira/browse/STORM-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2945: Fix Version/s: (was: 2.0.0) > Nail down and document how to support background emits in Spouts and Bolts > -- > > Key: STORM-2945 > URL: https://issues.apache.org/jira/browse/STORM-2945 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2932) topology.localityaware.higher/lower.bound.percent configs are confusing
[
https://issues.apache.org/jira/browse/STORM-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2932.
-
Resolution: Fixed
Assignee: Ethan Li
Fix Version/s: 2.0.0
Thanks [~ethanli], I merged into master.
> topology.localityaware.higher/lower.bound.percent configs are confusing
> ---
>
> Key: STORM-2932
> URL: https://issues.apache.org/jira/browse/STORM-2932
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> We have configs
> {code:java}
> topology.localityaware.higher.bound.percent: 0.8
> topology.localityaware.lower.bound.percent: 0.2{code}
> They are both called percent but 0.8/0.2 here actually means 80%/20%. I think
> it's better to remove "percent" here
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-2936) Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch
[ https://issues.apache.org/jira/browse/STORM-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2936. - Resolution: Fixed Fix Version/s: 1.2.0 > Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch > - > > Key: STORM-2936 > URL: https://issues.apache.org/jira/browse/STORM-2936 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.1.2 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > Labels: pull-request-available > Fix For: 1.2.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.1.x-branch so that critical fixes can be > go in 1.1.x-branch as well. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2936) Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch
[ https://issues.apache.org/jira/browse/STORM-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363545#comment-16363545 ] Jungtaek Lim commented on STORM-2936: - [~erikdw] Thanks for the correction! Just fixed. > Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch > - > > Key: STORM-2936 > URL: https://issues.apache.org/jira/browse/STORM-2936 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.1.2 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > Labels: pull-request-available > Fix For: 1.1.2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.1.x-branch so that critical fixes can be > go in 1.1.x-branch as well. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2947: Description: We've been deprecating the things but haven't have time to replace/get rid of them. It should be better if we have time to review and address them. was:We've been deprecating the things but haven't have time to replace/get rid of them. It should be better if we have time to review and address them. Component/s: storm-solr storm-kafka storm-hdfs > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363664#comment-16363664 ] Jungtaek Lim commented on STORM-2947: - Many of deprecated annotation don't have deprecated javadoc tag and don't guide how end-users can replace the usage with others. I'm not sure we could remove such things for Storm 2.0.0. Other things can be reviewed and removed in Storm 2.0.0, especially transactional API (does anyone still use it?) and ineffective configurations. > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2936) Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch
[ https://issues.apache.org/jira/browse/STORM-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2936: Fix Version/s: (was: 1.2.0) 1.1.2 > Overwrite storm-kafka-client 1.x-branch into 1.1.x-branch > - > > Key: STORM-2936 > URL: https://issues.apache.org/jira/browse/STORM-2936 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka-client >Affects Versions: 1.1.2 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Blocker > Labels: pull-request-available > Fix For: 1.1.2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > This is to track the effort of syncing up divergence between > storm-kafka-client 1.x-branch and 1.1.x-branch so that critical fixes can be > go in 1.1.x-branch as well. > Linking discussion thread: > [https://lists.apache.org/thread.html/0451fed132bb982b618d9e0780282a87554f1bc5747827599f276944@%3Cdev.storm.apache.org%3E] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363548#comment-16363548 ] Jungtaek Lim commented on STORM-2947: - I just wrote a page "Deprecated annotation in master branch (Storm 2.0.0)" which enumerates all the deprecated annotations. It doesn't enumerate the usage of deprecations. > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2761) JoinBolt.java 's paradigm is new model of stream join?
[ https://issues.apache.org/jira/browse/STORM-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2761. - Resolution: Not A Problem > JoinBolt.java 's paradigm is new model of stream join? > -- > > Key: STORM-2761 > URL: https://issues.apache.org/jira/browse/STORM-2761 > Project: Apache Storm > Issue Type: Question > Components: storm-client >Reporter: Fei Pan >Priority: Critical > > Hi, I am a researcher from University of Toronto and I am studying > acceleration on stream processing platform. I have a question about the model > of window-based stream join used in the JoinBolt.java. From my understanding, > when a new tuple arrived, we join this new tuple with all the tuples in the > window of the opposite stream. However, in the JoinBolt.java, not only the > new tuple, but the tuples in the entire local window will join with the > window of the opposite stream. This actually produces a lot of duplicated > results, since most of the old tuples in the local window have joined before. > I don't know if this is a new paradigm or the storm's team misunderstood the > model of stream join. Can someone help me to clarify this question? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-1732) Resources are deleted when worker dies
[
https://issues.apache.org/jira/browse/STORM-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-1732.
-
Resolution: Invalid
We introduced Supervisor V2 in 1.0.3 which completely renew the code on
Supervisor. We don't track old supervisor issue.
> Resources are deleted when worker dies
> --
>
> Key: STORM-1732
> URL: https://issues.apache.org/jira/browse/STORM-1732
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: gareth smith
>Priority: Critical
> Attachments: potentalFix.patch
>
>
> *Lets say a worker has been started by the supervisor*
> 2016-04-26 16:11:48.716 [o.a.s.d.supervisor] INFO: Launching worker with
> assignment {:storm-id "Lightning-1-1461683473", :executors [[12 12] [54 54]
> [42 42] [24 24] [18 18] [6 6] [48 48] [30 30] [36 36]], :resources
> #object[org.apache.storm.generated.WorkerResources 0x10bac1e4
> "WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)"]} for this
> supervisor 477ae22e-1a2b-4ea3-afd5-cb969f25e732 on port 6700 with id
> a5d51626-6e9f-4614-9ebb-a6263c140ca2
> 2016-04-26 16:11:48.727 [o.a.s.d.supervisor] INFO: Launching worker with
> command: 'C:\LightningDeployment\Java\bin\java' '-cp'
> 2016-04-26 16:11:48.910 [o.a.s.config] INFO: SET worker-user
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 LIGHTNINGVM14$
> *note this bit is is new for storm 1.0.0*
> 2016-04-26 16:11:49.405 [o.a.s.d.supervisor] INFO: Creating symlinks for
> worker-id: a5d51626-6e9f-4614-9ebb-a6263c140ca2 storm-id:
> Lightning-1-1461683473 to its port artifacts directory
> 2016-04-26 16:11:50.251 [o.a.s.d.supervisor] INFO: Creating symlinks for
> worker-id: a5d51626-6e9f-4614-9ebb-a6263c140ca2 storm-id:
> Lightning-1-1461683473 for files(1): ("resources")
> *When a worker dies we correctly see some clean up and a new worker
> started...*
> 2016-04-26 16:15:35.520 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 exited with code: 20
> 2016-04-26 16:15:39.674 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 has died!
> 2016-04-26 16:15:39.675 [o.a.s.d.supervisor] INFO: Shutting down and clearing
> state for id a5d51626-6e9f-4614-9ebb-a6263c140ca2. Current supervisor time:
> 1461683739. State: :timed-out, Heartbeat: {:time-secs 1461683734, :storm-id
> "Lightning-1-1461683473", :executors [[12 12] [54 54] [42 42] [24 24] [18 18]
> [6 6] [48 48] [30 30] [-1 -1] [36 36]], :port 6700}
> 2016-04-26 16:15:39.676 [o.a.s.d.supervisor] INFO: Shutting down
> 477ae22e-1a2b-4ea3-afd5-cb969f25e732:a5d51626-6e9f-4614-9ebb-a6263c140ca2
> 2016-04-26 16:15:39.676 [o.a.s.config] INFO: GET worker-user
> a5d51626-6e9f-4614-9ebb-a6263c140ca2
> 2016-04-26 16:15:39.677 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 has died!
> 2016-04-26 16:15:39.681 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 has died!
> 2016-04-26 16:15:39.857 [o.a.s.util] INFO: Error when trying to kill 1352.
> Process is probably already dead.
> 2016-04-26 16:15:39.955 [o.a.s.util] INFO: Error when trying to kill 2372.
> Process is probably already dead.
> 2016-04-26 16:15:40.009 [o.a.s.util] INFO: Error when trying to kill 4932.
> Process is probably already dead.
> 2016-04-26 16:15:40.009 [o.a.s.d.supervisor] INFO: Sleep 10 seconds for
> execution of cleanup threads on worker.
> 2016-04-26 16:15:49.677 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 has died!
> 2016-04-26 16:15:49.679 [o.a.s.d.supervisor] INFO: Worker Process
> a5d51626-6e9f-4614-9ebb-a6263c140ca2 has died!
> 2016-04-26 16:15:50.056 [o.a.s.util] INFO: Error when trying to kill 1352.
> Process is probably already dead.
> 2016-04-26 16:15:50.119 [o.a.s.util] INFO: Error when trying to kill 2372.
> Process is probably already dead.
> 2016-04-26 16:15:50.175 [o.a.s.util] INFO: Error when trying to kill 4932.
> Process is probably already dead.
> 2016-04-26 16:15:50.257 [o.a.s.config] INFO: REMOVE worker-user
> a5d51626-6e9f-4614-9ebb-a6263c140ca2
> 2016-04-26 16:15:50.257 [o.a.s.d.supervisor] INFO: Shut down
> 477ae22e-1a2b-4ea3-afd5-cb969f25e732:a5d51626-6e9f-4614-9ebb-a6263c140ca2
> 2016-04-26 16:15:50.257 [o.a.s.d.supervisor] INFO: Launching worker with
> assignment {:storm-id "Lightning-1-1461683473", :executors [[12 12] [54 54]
> [42 42] [24 24] [18 18] [6 6] [48 48] [30 30] [36 36]], :resources
> #object[org.apache.storm.generated.WorkerResources 0x20e1ad4f
> "WorkerResources(mem_on_heap:0.0, mem_off_heap:0.0, cpu:0.0)"]} for this
> supervisor 477ae22e-1a2b-4ea3-afd5-cb969f25e732 on port 6700 with id
> e413447b-c9ca-417d-8e55-e10dd0edc6a4
> *When the worker has
[jira] [Updated] (STORM-2878) Supervisor collapse continuously when there is a expired assignment for overdue storm
[
https://issues.apache.org/jira/browse/STORM-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2878:
Fix Version/s: (was: 2.0.0)
> Supervisor collapse continuously when there is a expired assignment for
> overdue storm
> -
>
> Key: STORM-2878
> URL: https://issues.apache.org/jira/browse/STORM-2878
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-server
>Affects Versions: 2.0.0, 1.x
>Reporter: Yuzhao Chen
>Priority: Critical
> Labels: patch
>
> For now, when a topology is reassigned or killed for a cluster, supervisor
> will delete 4 files for an overdue storm:
> - storm-code
> - storm-ser
> - storm-jar
> - LocalAssignment
> Slot.java
> static DynamicState cleanupCurrentContainer(DynamicState dynamicState,
> StaticState staticState, MachineState nextState) throws Exception {
> assert(dynamicState.container != null);
> assert(dynamicState.currentAssignment != null);
> assert(dynamicState.container.areAllProcessesDead());
>
> dynamicState.container.cleanUp();
> staticState.localizer.releaseSlotFor(dynamicState.currentAssignment,
> staticState.port);
> DynamicState ret = dynamicState.withCurrentAssignment(null, null);
> if (nextState != null) {
> ret = ret.withState(nextState);
> }
> return ret;
> }
> But we do not make a transaction to do this, if an exception occurred during
> deleting storm-code/ser/jar, an overdue local assignment will be left on disk.
> Then when supervisor restart from the exception above, the slots will be
> initial and container will be recovered from LocalAssignments, the blob store
> will fetch the files from Nimbus/Master, but will get a KeyNotFoundException,
> and supervisor collapses again.
> This will happens continuously and supervisor will never recover until we
> clean up all the local assignments manually.
> This is the stack:
> 2017-12-27 14:15:04.434 o.a.s.l.AsyncLocalizer [INFO] Cleaning up unused
> topologies in /opt/meituan/storm/data/supervisor/stormdist
> 2017-12-27 14:15:04.434 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
> /opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785
> 2017-12-27 14:15:04.445 o.a.s.d.s.Slot [INFO] STATE EMPTY msInState: 109 ->
> WAITING_FOR_BASIC_LOCALIZATION msInState: 1
> 2017-12-27 14:15:04.471 o.a.s.d.s.Supervisor [INFO] Starting supervisor with
> id 255d3fed-f3ee-4c7e-8a08-b693c9a6a072 at host gq-data-rt48.gq.sankuai.com.
> 2017-12-27 14:15:04.502 o.a.s.u.Utils [ERROR] An exception happened while
> downloading
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
> from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
> ~[storm-core-1.1.2-mt001.jar:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[?:1.7.0_76]
> at
>
[jira] [Resolved] (STORM-2436) Custom Log4j2 appender not working with storm
[
https://issues.apache.org/jira/browse/STORM-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2436.
-
Resolution: Won't Fix
The version is not we're tracking now. Closing.
> Custom Log4j2 appender not working with storm
> -
>
> Key: STORM-2436
> URL: https://issues.apache.org/jira/browse/STORM-2436
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 0.10.0
> Environment: Storm Mapr Cluster
>Reporter: Nishant Malhotra
>Priority: Blocker
>
> We have created below logger configuration in worker.xml of storm cluster
>
> filename="${sys:storm.log.dir}/userlogs/info-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/info-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
> filename="${sys:storm.log.dir}/userlogs/debug-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/debug-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
> filename="${sys:storm.log.dir}/userlogs/error-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/error-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
>
> We have created below logger configuration in worker.xml of storm cluster
>
> filename="${sys:storm.log.dir}/userlogs/info-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/info-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
> filename="${sys:storm.log.dir}/userlogs/debug-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/debug-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
> filename="${sys:storm.log.dir}/userlogs/error-${sys:logfile.name}"
>
> filepattern="${sys:storm.log.dir}/userlogs/error-${sys:logfile.name}.%d{-MM-dd-HH-mm}"
> append="true">
>
> ${patternFdpNew}
>
>
>
>
>
>
> and custom logger defined in below fashion
>
>
>
>
>
> In storm topology builder main class
>config.put("topology.classpath","/usr/local/Cellar/storm/mylogger.jar");
> and
> In spout
> private static org.slf4j.Logger _logger =LoggerFactory.getLogger("custom-
> logger");
> Now what happens it detect my jar and write one line of log in log file but
> post that doesn't log any line to the log file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-2382) log4j and slf4j conflicting libraries issue
[
https://issues.apache.org/jira/browse/STORM-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2382:
Priority: Minor (was: Blocker)
> log4j and slf4j conflicting libraries issue
> ---
>
> Key: STORM-2382
> URL: https://issues.apache.org/jira/browse/STORM-2382
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.0.1
> Environment: centos 7.1, oracle jdk1.8
>Reporter: Sree Vaddi
>Priority: Minor
>
> my project storm 1.0.1 job's dependencies (log4j & slf4j) conflict with
> apache storm 1.0.1 default libraries (STORM_HOME/lib) and is preventing
> submitting new storm job.
> * 1. shadow my storm job jar *
> I get the following error:
> {code:language=java}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/tmp/356865dafc1a11e69341ecb1d7ac1510.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.IllegalAccessError: tried to access
> method
> org.apache.logging.log4j.core.lookup.MapLookup.newMap(I)Ljava/util/HashMap;
> from class org.apache.logging.log4j.core.lookup.MainMapLookup
> at
> org.apache.logging.log4j.core.lookup.MainMapLookup.(MainMapLookup.java:37)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.logging.log4j.core.util.ReflectionUtil.instantiate(ReflectionUtil.java:185)
> at
> org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:65)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:346)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:161)
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:359)
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:420)
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
> at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
> at org.apache.storm.utils.LocalState.(LocalState.java:45)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at clojure.lang.RT.classForName(RT.java:2154)
> at clojure.lang.RT.classForName(RT.java:2163)
> at org.apache.storm.config__init.__init7(Unknown Source)
> at org.apache.storm.config__init.(Unknown Source)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at clojure.lang.RT.classForName(RT.java:2154)
> at clojure.lang.RT.classForName(RT.java:2163)
> at clojure.lang.RT.loadClassForName(RT.java:2182)
> at clojure.lang.RT.load(RT.java:436)
> at clojure.lang.RT.load(RT.java:412)
> at clojure.core$load$fn__5448.invoke(core.clj:5866)
> at clojure.core$load.doInvoke(core.clj:5865)
> at clojure.lang.RestFn.invoke(RestFn.java:408)
> at clojure.core$load_one.invoke(core.clj:5671)
> at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
> at clojure.core$load_lib.doInvoke(core.clj:5710)
> at clojure.lang.RestFn.applyTo(RestFn.java:142)
> at clojure.core$apply.invoke(core.clj:632)
> at clojure.core$load_libs.doInvoke(core.clj:5753)
> at clojure.lang.RestFn.applyTo(RestFn.java:137)
> at clojure.core$apply.invoke(core.clj:634)
>
[jira] [Commented] (STORM-2382) log4j and slf4j conflicting libraries issue
[
https://issues.apache.org/jira/browse/STORM-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363470#comment-16363470
]
Jungtaek Lim commented on STORM-2382:
-
Lower the priority since it is specific to the case and answer is provided. I'm
thinking about closing this if there's no further input here.
> log4j and slf4j conflicting libraries issue
> ---
>
> Key: STORM-2382
> URL: https://issues.apache.org/jira/browse/STORM-2382
> Project: Apache Storm
> Issue Type: Bug
>Affects Versions: 1.0.1
> Environment: centos 7.1, oracle jdk1.8
>Reporter: Sree Vaddi
>Priority: Minor
>
> my project storm 1.0.1 job's dependencies (log4j & slf4j) conflict with
> apache storm 1.0.1 default libraries (STORM_HOME/lib) and is preventing
> submitting new storm job.
> * 1. shadow my storm job jar *
> I get the following error:
> {code:language=java}
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/usr/hdp/2.5.0.0-1245/storm/lib/log4j-slf4j-impl-2.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/tmp/356865dafc1a11e69341ecb1d7ac1510.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.IllegalAccessError: tried to access
> method
> org.apache.logging.log4j.core.lookup.MapLookup.newMap(I)Ljava/util/HashMap;
> from class org.apache.logging.log4j.core.lookup.MainMapLookup
> at
> org.apache.logging.log4j.core.lookup.MainMapLookup.(MainMapLookup.java:37)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.logging.log4j.core.util.ReflectionUtil.instantiate(ReflectionUtil.java:185)
> at
> org.apache.logging.log4j.core.lookup.Interpolator.(Interpolator.java:65)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:346)
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:161)
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:359)
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:420)
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:138)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:147)
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
> at org.apache.logging.log4j.LogManager.getContext(LogManager.java:175)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:102)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:277)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
> at org.apache.storm.utils.LocalState.(LocalState.java:45)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at clojure.lang.RT.classForName(RT.java:2154)
> at clojure.lang.RT.classForName(RT.java:2163)
> at org.apache.storm.config__init.__init7(Unknown Source)
> at org.apache.storm.config__init.(Unknown Source)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at clojure.lang.RT.classForName(RT.java:2154)
> at clojure.lang.RT.classForName(RT.java:2163)
> at clojure.lang.RT.loadClassForName(RT.java:2182)
> at clojure.lang.RT.load(RT.java:436)
> at clojure.lang.RT.load(RT.java:412)
> at clojure.core$load$fn__5448.invoke(core.clj:5866)
> at clojure.core$load.doInvoke(core.clj:5865)
> at clojure.lang.RestFn.invoke(RestFn.java:408)
> at clojure.core$load_one.invoke(core.clj:5671)
> at clojure.core$load_lib$fn__5397.invoke(core.clj:5711)
> at clojure.core$load_lib.doInvoke(core.clj:5710)
> at clojure.lang.RestFn.applyTo(RestFn.java:142)
> at clojure.core$apply.invoke(core.clj:632)
> at
[jira] [Resolved] (STORM-1621) [Storm][DRPC] Request failed after 60 seconds
[ https://issues.apache.org/jira/browse/STORM-1621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-1621. - Resolution: Won't Fix Too old version which we don't track now, and also it is rather a kind of question which should be go through mailing list. Closing this. > [Storm][DRPC] Request failed after 60 seconds > - > > Key: STORM-1621 > URL: https://issues.apache.org/jira/browse/STORM-1621 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 0.10.0 > Environment: Linux-Ubuntu >Reporter: Raksmey PHAN >Priority: Blocker > Labels: newbie > > Hello. > We work with storm and use the DRPC to execute some algorithms on the cluster. > When the duration time on each node is less than 60 seconds, there is no > trouble: client receives correct result. However when we have to solve bigger > problem with the same algorithm (then duration time is more than 60 seconds) > we have the following message: > Exception in thread "main" DRPCExecutionException(msg:Request failed) at > backtype.storm.generated.DistributedRPC$execute_result$execute_resultStandardScheme.read(DistributedRPC.java:1222) > at > backtype.storm.generated.DistributedRPC$execute_result$execute_resultStandardScheme.read(DistributedRPC.java:1200) > at backtype.storm.generated... > It seems to be about the node, which sends a message to cluster with "Request > failed" and doesn't finish algorithm. Hope somebody have a hint to solve this > :( > Note that the Client-DRPC-Topology and the 10 test nodes work properly when > the duration is less than the minute. > Thank you. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-617) In Storm secure mode re-deploying trident topology causes zookeeper ACL issue
[ https://issues.apache.org/jira/browse/STORM-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-617. Resolution: Won't Fix Closing due to inactivity. It is also based on the version we don't track as of now. > In Storm secure mode re-deploying trident topology causes zookeeper ACL issue > - > > Key: STORM-617 > URL: https://issues.apache.org/jira/browse/STORM-617 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.10.0 >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani >Priority: Blocker > > This issue is caused by this line > https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/transactional/state/TransactionalState.java#L67 > If the storm cluster nimbus is running with a kerberos principal named > "nimbus" > and supervisors are running with principal "storm" . Storm puts the acl on > trident spout using principal "nimbus" and this won't be able to accessed or > modified by supervisor since they are logging into zookeeper as user "storm". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
Jungtaek Lim created STORM-2947: --- Summary: Review and fix/remove deprecated things in Storm 2.0.0 Key: STORM-2947 URL: https://issues.apache.org/jira/browse/STORM-2947 Project: Apache Storm Issue Type: Task Components: storm-client, storm-server Affects Versions: 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim We've been deprecating the things but haven't have time to replace/get rid of them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-1898) MAX_BATCH_SIZE_CONF not working in Trident storm Spout
[ https://issues.apache.org/jira/browse/STORM-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363490#comment-16363490 ] Jungtaek Lim commented on STORM-1898: - The issue doesn't look like blocker. Lowering priority. > MAX_BATCH_SIZE_CONF not working in Trident storm Spout > --- > > Key: STORM-1898 > URL: https://issues.apache.org/jira/browse/STORM-1898 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 1.0.0 >Reporter: Narendra Bidari >Priority: Major > > Ideally Trident process should process tuples in Batch. > ex > > https://github.com/apache/storm/blob/ab66003c18fe4f8c0926b3219408b735b2ce2adf/storm-core/src/jvm/org/apache/storm/trident/spout/RichSpoutBatchExecutor.java > there is a parameter called MAX_BATCH_SIZE_CONF which limits the size of the > batch. > This parameter is not present in TridentKafkaEmitter. > https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java > --- > Problem is that now everytime the topology restarts it just fetches all > messages from Kafka. > Could any one throw some idea on it, I certainly feel its a bug. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-1898) MAX_BATCH_SIZE_CONF not working in Trident storm Spout
[ https://issues.apache.org/jira/browse/STORM-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1898: Priority: Major (was: Blocker) > MAX_BATCH_SIZE_CONF not working in Trident storm Spout > --- > > Key: STORM-1898 > URL: https://issues.apache.org/jira/browse/STORM-1898 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka >Affects Versions: 1.0.0 >Reporter: Narendra Bidari >Priority: Major > > Ideally Trident process should process tuples in Batch. > ex > > https://github.com/apache/storm/blob/ab66003c18fe4f8c0926b3219408b735b2ce2adf/storm-core/src/jvm/org/apache/storm/trident/spout/RichSpoutBatchExecutor.java > there is a parameter called MAX_BATCH_SIZE_CONF which limits the size of the > batch. > This parameter is not present in TridentKafkaEmitter. > https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java > --- > Problem is that now everytime the topology restarts it just fetches all > messages from Kafka. > Could any one throw some idea on it, I certainly feel its a bug. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2306) Redesign Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2306. - Resolution: Fixed Fix Version/s: 2.0.0 Marking as resolved since [~revans2] merged this in. I've reported higher CPU in multiple workers in the patch. [~roshan_naik] If you can reproduce it, could you please file a follow-up issue (with setting priority to blocker) and assign to you? I'll label it as Storm 2.0.0 epic. > Redesign Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 60h > Remaining Estimate: 0h > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. > 3) *Backpressure Model* > https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing > Describes the Backpressure model integrated into the new messaging subsystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2933) Add a storm-perf topology that uses storm-kafka-client, so we can benchmark that module
[ https://issues.apache.org/jira/browse/STORM-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2933. - Resolution: Fixed Thanks [~Srdo], I merged into master. > Add a storm-perf topology that uses storm-kafka-client, so we can benchmark > that module > --- > > Key: STORM-2933 > URL: https://issues.apache.org/jira/browse/STORM-2933 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka-client >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2938) Share CuratorFramework instance in a process (daemons/worker)
[
https://issues.apache.org/jira/browse/STORM-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-2938:
Priority: Minor (was: Major)
> Share CuratorFramework instance in a process (daemons/worker)
> -
>
> Key: STORM-2938
> URL: https://issues.apache.org/jira/browse/STORM-2938
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client, storm-server
>Reporter: Jungtaek Lim
>Priority: Minor
>
> According to Curator web site, each process (nimbus, supervisor, worker,
> etc.) can share singleton curator framework instance per ZK cluster since
> it's thread-safe.
> [http://curator.apache.org/curator-framework/]
> {quote}CuratorFrameworks are allocated using the CuratorFrameworkFactory
> which provides both factory methods and a builder for creating instances.
> IMPORTANT: CuratorFramework instances are fully thread-safe. You should share
> one CuratorFramework per ZooKeeper cluster in your application.
> {quote}
> We are using Curator 4.0 both master and 1.x branch, hence we are good to
> apply this.
> We already applied this for Nimbus via STORM-2901. This issue is to track the
> efforts for other processes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-2938) Share CuratorFramework instance in a process (daemons/worker)
Jungtaek Lim created STORM-2938:
---
Summary: Share CuratorFramework instance in a process
(daemons/worker)
Key: STORM-2938
URL: https://issues.apache.org/jira/browse/STORM-2938
Project: Apache Storm
Issue Type: Improvement
Components: storm-client, storm-server
Reporter: Jungtaek Lim
According to Curator web site, each process (nimbus, supervisor, worker, etc.)
can share singleton curator framework instance per ZK cluster since it's
thread-safe.
[http://curator.apache.org/curator-framework/]
{quote}CuratorFrameworks are allocated using the CuratorFrameworkFactory which
provides both factory methods and a builder for creating instances. IMPORTANT:
CuratorFramework instances are fully thread-safe. You should share one
CuratorFramework per ZooKeeper cluster in your application.
{quote}
We are using Curator 4.0 both master and 1.x branch, hence we are good to apply
this.
We already applied this for Nimbus via STORM-2901. This issue is to track the
efforts for other processes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (STORM-2948) Add -XX:+HeapDumpOnOutOfMemoryError for maven-surefire-plugin in pom.xml
[ https://issues.apache.org/jira/browse/STORM-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2948. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. > Add -XX:+HeapDumpOnOutOfMemoryError for maven-surefire-plugin in pom.xml > - > > Key: STORM-2948 > URL: https://issues.apache.org/jira/browse/STORM-2948 > Project: Apache Storm > Issue Type: Improvement >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Trivial > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2950) ArrayIndexOutOfBoundsException in FieldsGrouper.chooseTasks function
[
https://issues.apache.org/jira/browse/STORM-2950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2950.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~ethanli], I merged into master.
> ArrayIndexOutOfBoundsException in FieldsGrouper.chooseTasks function
>
>
> Key: STORM-2950
> URL: https://issues.apache.org/jira/browse/STORM-2950
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Ethan Li
>Assignee: Ethan Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> TupleUtils.chooseTaskIndex:
> {code:java}
> public static int chooseTaskIndex(List keys, int numTasks) {
> return Math.abs(listHashCode(keys)) % numTasks;
> }
> {code}
>
> listHashCode(keys) might return Integer.MIN_VALUE (-2147483648).
> Math.abs(-2147483648) equals to -2147483648.
> In this case, chooseTaskIndex might return negative numbers and it will cause
> ArrayIndexOutOfBoundsException in FieldsGrouper.chooseTasks
> {code:java}
> @Override
> public List chooseTasks(int taskId, List values) {
> int targetTaskIndex =
> TupleUtils.chooseTaskIndex(outFields.select(groupFields, values), numTasks);
> return targetTasks.get(targetTaskIndex);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (STORM-2964) Deprecate old Spout wait strategy model in 1.x version line
Jungtaek Lim created STORM-2964: --- Summary: Deprecate old Spout wait strategy model in 1.x version line Key: STORM-2964 URL: https://issues.apache.org/jira/browse/STORM-2964 Project: Apache Storm Issue Type: Task Components: storm-client Reporter: Jungtaek Lim This is follow-up issue for STORM-2958, deprecating old Spout wait strategy in 1.x version line with properly noting that it will be removed at Storm 2.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2958) Use new wait strategies for Spout as well
[ https://issues.apache.org/jira/browse/STORM-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369735#comment-16369735 ] Jungtaek Lim commented on STORM-2958: - [~roshan_naik] I filed new issue (STORM-2964) about deprecating old spout model in 1.x version line. I'd be really appreciated if you could take up. Please let me know if you are busy with other tasks. I'll be happy to take up. > Use new wait strategies for Spout as well > - > > Key: STORM-2958 > URL: https://issues.apache.org/jira/browse/STORM-2958 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > STORM-2306 introduced a new configurable wait strategy system for these > situations > * BackPressure Wait (used by spout & bolt) > * No incoming data (used by bolt) > There is another wait situation in the spout when there are no emits > generated in a nextTuple() or if max.spout.pending has been reached. This > Jira is to transition the spout wait strategy from the old model to the new > model. Thereby we have a uniform model for dealing with wait strategies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2962) KeyValue State Resharding
Jungtaek Lim created STORM-2962: --- Summary: KeyValue State Resharding Key: STORM-2962 URL: https://issues.apache.org/jira/browse/STORM-2962 Project: Apache Storm Issue Type: Improvement Components: storm-client Reporter: Jungtaek Lim Storm's KeyValueState leverages namespace which is typically composed to component name + task id, which means the key-value pair is bound to the task. To allow rebalancing topology with different parallelism for stateful component, we should support resharding of the state. I wonder more thing about current State implementation but it should be filed to other issues after verification. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2958) Use new wait strategies for Spout as well
[ https://issues.apache.org/jira/browse/STORM-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2958. - Resolution: Fixed Thanks [~roshan_naik], I merged into master. > Use new wait strategies for Spout as well > - > > Key: STORM-2958 > URL: https://issues.apache.org/jira/browse/STORM-2958 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > STORM-2306 introduced a new configurable wait strategy system for these > situations > * BackPressure Wait (used by spout & bolt) > * No incoming data (used by bolt) > There is another wait situation in the spout when there are no emits > generated in a nextTuple() or if max.spout.pending has been reached. This > Jira is to transition the spout wait strategy from the old model to the new > model. Thereby we have a uniform model for dealing with wait strategies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2964) Deprecate old Spout wait strategy model in 1.x version line
[ https://issues.apache.org/jira/browse/STORM-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370885#comment-16370885 ] Jungtaek Lim commented on STORM-2964: - [~roshan_naik] OK thanks. It might be better to convert this issue to address all the configurations/interfaces/classes to mark as deprecated in 1.x version line, so STORM-2947 for removing deprecated things from 2.0.0, and this one for marking deprecation for 1.x version line. > Deprecate old Spout wait strategy model in 1.x version line > --- > > Key: STORM-2964 > URL: https://issues.apache.org/jira/browse/STORM-2964 > Project: Apache Storm > Issue Type: Task > Components: storm-client >Reporter: Jungtaek Lim >Priority: Major > > This is follow-up issue for STORM-2958, deprecating old Spout wait strategy > in 1.x version line with properly noting that it will be removed at Storm > 2.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2964) Deprecate removed configs/interfaces/classes from Storm 2.0.0 to 1.x version line
[ https://issues.apache.org/jira/browse/STORM-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2964: Summary: Deprecate removed configs/interfaces/classes from Storm 2.0.0 to 1.x version line (was: Deprecate old Spout wait strategy model in 1.x version line) > Deprecate removed configs/interfaces/classes from Storm 2.0.0 to 1.x version > line > - > > Key: STORM-2964 > URL: https://issues.apache.org/jira/browse/STORM-2964 > Project: Apache Storm > Issue Type: Task > Components: storm-client >Reporter: Jungtaek Lim >Priority: Major > > This is follow-up issue for STORM-2958, deprecating old Spout wait strategy > in 1.x version line with properly noting that it will be removed at Storm > 2.0.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2961) Refactoring duplicate code in Topology Builder classes
[
https://issues.apache.org/jira/browse/STORM-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2961.
-
Resolution: Fixed
Fix Version/s: 2.0.0
Thanks [~kishorvpatil], I merged into master.
> Refactoring duplicate code in Topology Builder classes
> --
>
> Key: STORM-2961
> URL: https://issues.apache.org/jira/browse/STORM-2961
> Project: Apache Storm
> Issue Type: Improvement
>Reporter: Kishor Patil
>Assignee: Kishor Patil
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Most subclasses for {{BaseConfigurationDeclarer}} class are duplicating code
> for {{addResource}} and {{addResources. }}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (STORM-2964) Deprecate removed configs/interfaces/classes from Storm 2.0.0 to 1.x version line
[ https://issues.apache.org/jira/browse/STORM-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2964: Description: This is to track the effort on deprecating removed configs/interfaces/classes from Storm 2.0.0 to 1.x version line. We may be late on the party but still be better to annotate rather than suddenly introducing the change. (was: This is follow-up issue for STORM-2958, deprecating old Spout wait strategy in 1.x version line with properly noting that it will be removed at Storm 2.0.0.) > Deprecate removed configs/interfaces/classes from Storm 2.0.0 to 1.x version > line > - > > Key: STORM-2964 > URL: https://issues.apache.org/jira/browse/STORM-2964 > Project: Apache Storm > Issue Type: Task > Components: storm-client >Reporter: Jungtaek Lim >Priority: Major > > This is to track the effort on deprecating removed configs/interfaces/classes > from Storm 2.0.0 to 1.x version line. We may be late on the party but still > be better to annotate rather than suddenly introducing the change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2947) Review and fix/remove deprecated things in Storm 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368706#comment-16368706 ] Jungtaek Lim commented on STORM-2947: - storm.local.mode.zmq is not annotated as deprecated, but it is not used from fairly long ago. It was effectively deprecated. > Review and fix/remove deprecated things in Storm 2.0.0 > -- > > Key: STORM-2947 > URL: https://issues.apache.org/jira/browse/STORM-2947 > Project: Apache Storm > Issue Type: Task > Components: storm-client, storm-hdfs, storm-kafka, storm-server, > storm-solr >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > We've been deprecating the things but haven't have time to replace/get rid of > them. It should be better if we have time to review and address them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2960) Better to stress importance of setting up proper OS account for Storm processes
Jungtaek Lim created STORM-2960: --- Summary: Better to stress importance of setting up proper OS account for Storm processes Key: STORM-2960 URL: https://issues.apache.org/jira/browse/STORM-2960 Project: Apache Storm Issue Type: Documentation Components: documentation Reporter: Jungtaek Lim Assignee: Jungtaek Lim We have SECURITY.md and also "Firewall/OS level Security" section, but the document doesn't explicitly mention the fact that Storm processes should use OS account(s) which should be properly restricted. We may also want to note that workers which could execute arbitrary code will be running with Supervisor OS account by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2939) Create interface for processing worker metrics
[ https://issues.apache.org/jira/browse/STORM-2939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2939. - Resolution: Fixed Marking as resolved as [~agresch] stated the work for issue is finished. > Create interface for processing worker metrics > -- > > Key: STORM-2939 > URL: https://issues.apache.org/jira/browse/STORM-2939 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Aaron Gresch >Assignee: Aaron Gresch >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > In Container.java, we send worker metrics to Nimbus to store to RocksDB. > Other implementations (HBase, etc) may want to process in different fashions. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2952) Deprecate storm-kafka in 1.x
[ https://issues.apache.org/jira/browse/STORM-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375488#comment-16375488 ] Jungtaek Lim commented on STORM-2952: - [~Srdo] Could you work on following up task: removing storm-kafka in 2.0.0? Thanks in advance! > Deprecate storm-kafka in 1.x > > > Key: STORM-2952 > URL: https://issues.apache.org/jira/browse/STORM-2952 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka >Affects Versions: 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > Labels: pull-request-available > Fix For: 1.1.3, 1.0.7, 1.2.2 > > Time Spent: 40m > Remaining Estimate: 0h > > Add deprecation notice to storm-kafka for the 1.x branch. See the thread at > http://mail-archives.apache.org/mod_mbox/storm-dev/201802.mbox/%3CCAF5108hZBRf7Y%2BOFPif9kSw9%2BuZuPx5iBgSt1TOX5zMvF0-NfQ%40mail.gmail.com%3E. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (STORM-2969) remove @isDerivedFrom validation for topology.spout/bolt/backpressure.wait.strategy
[ https://issues.apache.org/jira/browse/STORM-2969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2969. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~ethanli], I merged into master. > remove @isDerivedFrom validation for > topology.spout/bolt/backpressure.wait.strategy > --- > > Key: STORM-2969 > URL: https://issues.apache.org/jira/browse/STORM-2969 > Project: Apache Storm > Issue Type: Bug >Reporter: Ethan Li >Assignee: Ethan Li >Priority: Minor > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > It would be better to support users to create their own wait strategies (for > example, topology.spout.wait.strategy). To make it work, we need to get rid > of @isDerivedFrom validation for them. The topology submission will fail on > validation because the user defined class is not in the nimbus' classpath. > This applies to topology.bolt.wait.strategy and > topology.backpressure.wait.strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2953) Remove storm-kafka in 2.0.0
[ https://issues.apache.org/jira/browse/STORM-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2953: Priority: Blocker (was: Minor) > Remove storm-kafka in 2.0.0 > --- > > Key: STORM-2953 > URL: https://issues.apache.org/jira/browse/STORM-2953 > Project: Apache Storm > Issue Type: Task > Components: storm-kafka >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Blocker > > Remove storm-kafka from master. See the thread at > http://mail-archives.apache.org/mod_mbox/storm-dev/201802.mbox/%3CCAF5108hZBRf7Y%2BOFPif9kSw9%2BuZuPx5iBgSt1TOX5zMvF0-NfQ%40mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (STORM-2965) Interpret wildcard in classpath correctly when reading config from classpath
Jungtaek Lim created STORM-2965: --- Summary: Interpret wildcard in classpath correctly when reading config from classpath Key: STORM-2965 URL: https://issues.apache.org/jira/browse/STORM-2965 Project: Apache Storm Issue Type: Bug Components: storm-client Affects Versions: 2.0.0 Reporter: Jungtaek Lim Assignee: Jungtaek Lim While reading configuration file from classpath at Utils.getConfigFromClasspath(), it doesn't interpret wildcard (*) as Java classpath does. It should behave same as Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (STORM-2965) Interpret wildcard in classpath correctly when reading config from classpath
[ https://issues.apache.org/jira/browse/STORM-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2965: Description: While reading configuration file from classpath at Utils.getConfigFromClasspath(), it doesn't interpret wildcard (\*) as Java classpath does. It should behave same as Java. (was: While reading configuration file from classpath at Utils.getConfigFromClasspath(), it doesn't interpret wildcard (*) as Java classpath does. It should behave same as Java.) > Interpret wildcard in classpath correctly when reading config from classpath > > > Key: STORM-2965 > URL: https://issues.apache.org/jira/browse/STORM-2965 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > > While reading configuration file from classpath at > Utils.getConfigFromClasspath(), it doesn't interpret wildcard (\*) as Java > classpath does. It should behave same as Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (STORM-2878) Supervisor collapse continuously when there is a expired assignment for overdue storm
[
https://issues.apache.org/jira/browse/STORM-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim closed STORM-2878.
---
Resolution: Duplicate
Assignee: Yuzhao Chen
> Supervisor collapse continuously when there is a expired assignment for
> overdue storm
> -
>
> Key: STORM-2878
> URL: https://issues.apache.org/jira/browse/STORM-2878
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-server
>Affects Versions: 2.0.0, 1.x
>Reporter: Yuzhao Chen
>Assignee: Yuzhao Chen
>Priority: Critical
> Labels: patch
>
> For now, when a topology is reassigned or killed for a cluster, supervisor
> will delete 4 files for an overdue storm:
> - storm-code
> - storm-ser
> - storm-jar
> - LocalAssignment
> Slot.java
> static DynamicState cleanupCurrentContainer(DynamicState dynamicState,
> StaticState staticState, MachineState nextState) throws Exception {
> assert(dynamicState.container != null);
> assert(dynamicState.currentAssignment != null);
> assert(dynamicState.container.areAllProcessesDead());
>
> dynamicState.container.cleanUp();
> staticState.localizer.releaseSlotFor(dynamicState.currentAssignment,
> staticState.port);
> DynamicState ret = dynamicState.withCurrentAssignment(null, null);
> if (nextState != null) {
> ret = ret.withState(nextState);
> }
> return ret;
> }
> But we do not make a transaction to do this, if an exception occurred during
> deleting storm-code/ser/jar, an overdue local assignment will be left on disk.
> Then when supervisor restart from the exception above, the slots will be
> initial and container will be recovered from LocalAssignments, the blob store
> will fetch the files from Nimbus/Master, but will get a KeyNotFoundException,
> and supervisor collapses again.
> This will happens continuously and supervisor will never recover until we
> clean up all the local assignments manually.
> This is the stack:
> 2017-12-27 14:15:04.434 o.a.s.l.AsyncLocalizer [INFO] Cleaning up unused
> topologies in /opt/meituan/storm/data/supervisor/stormdist
> 2017-12-27 14:15:04.434 o.a.s.d.s.AdvancedFSOps [INFO] Deleting path
> /opt/meituan/storm/data/supervisor/stormdist/app_dpsr_realtime_shop_vane_allcates-14-1513685785
> 2017-12-27 14:15:04.445 o.a.s.d.s.Slot [INFO] STATE EMPTY msInState: 109 ->
> WAITING_FOR_BASIC_LOCALIZATION msInState: 1
> 2017-12-27 14:15:04.471 o.a.s.d.s.Supervisor [INFO] Starting supervisor with
> id 255d3fed-f3ee-4c7e-8a08-b693c9a6a072 at host gq-data-rt48.gq.sankuai.com.
> 2017-12-27 14:15:04.502 o.a.s.u.Utils [ERROR] An exception happened while
> downloading
> /opt/meituan/storm/data/supervisor/tmp/ca4f8174-59be-40a4-b431-dbc8b697f063/stormjar.jar
> from blob store.
> org.apache.storm.generated.KeyNotFoundException: null
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26656)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result$beginBlobDownload_resultStandardScheme.read(Nimbus.java:26624)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$beginBlobDownload_result.read(Nimbus.java:26555)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$Client.recv_beginBlobDownload(Nimbus.java:864)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.generated.Nimbus$Client.beginBlobDownload(Nimbus.java:851)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.blobstore.NimbusBlobStore.getBlob(NimbusBlobStore.java:357)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorAttempt(Utils.java:598)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisorImpl(Utils.java:582)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.utils.Utils.downloadResourcesAsSupervisor(Utils.java:574)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.downloadBaseBlobs(AsyncLocalizer.java:123)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:148)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
> org.apache.storm.localizer.AsyncLocalizer$DownloadBaseBlobsDistributed.call(AsyncLocalizer.java:101)
> ~[storm-core-1.1.2-mt001.jar:?]
> at
[jira] [Resolved] (STORM-2965) Interpret wildcard in classpath correctly when reading config from classpath
[ https://issues.apache.org/jira/browse/STORM-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2965. - Resolution: Fixed Fix Version/s: 2.0.0 Merged into master. > Interpret wildcard in classpath correctly when reading config from classpath > > > Key: STORM-2965 > URL: https://issues.apache.org/jira/browse/STORM-2965 > Project: Apache Storm > Issue Type: Bug > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Jungtaek Lim >Assignee: Jungtaek Lim >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 4h > Remaining Estimate: 0h > > While reading configuration file from classpath at > Utils.getConfigFromClasspath(), it doesn't interpret wildcard (\*) as Java > classpath does. It should behave same as Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (STORM-1433) StormSQL Phase II
[ https://issues.apache.org/jira/browse/STORM-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed STORM-1433. --- Resolution: Fixed All subtasks completed. > StormSQL Phase II > - > > Key: STORM-1433 > URL: https://issues.apache.org/jira/browse/STORM-1433 > Project: Apache Storm > Issue Type: Epic > Components: storm-sql >Reporter: Haohui Mai >Priority: Major > > This epic tracks the effort of the phase II development of StormSQL. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (STORM-2909) New Metrics Reporting API - for 2.0.0
[
https://issues.apache.org/jira/browse/STORM-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376376#comment-16376376
]
Jungtaek Lim commented on STORM-2909:
-
[~ptgoetz]
Could you spend time to work on this issue? This is inconsistency between 1.x
version line (after 1.2.0) and 2.0.0.
> New Metrics Reporting API - for 2.0.0
> -
>
> Key: STORM-2909
> URL: https://issues.apache.org/jira/browse/STORM-2909
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client
>Affects Versions: 2.0.0
>Reporter: P. Taylor Goetz
>Assignee: P. Taylor Goetz
>Priority: Blocker
>
> This is a proposal to provide a new metrics reporting API based on [Coda
> Hale's metrics library | http://metrics.dropwizard.io/3.1.0/] (AKA
> Dropwizard/Yammer metrics).
> h2. Background
> In a [discussion on the dev@ mailing list |
> http://mail-archives.apache.org/mod_mbox/storm-dev/201610.mbox/%3ccagx0urh85nfh0pbph11pmc1oof6htycjcxsxgwp2nnofukq...@mail.gmail.com%3e]
> a number of community and PMC members recommended replacing Storm’s metrics
> system with a new API as opposed to enhancing the existing metrics system.
> Some of the objections to the existing metrics API include:
> # Metrics are reported as an untyped Java object, making it very difficult to
> reason about how to report it (e.g. is it a gauge, a counter, etc.?)
> # It is difficult to determine if metrics coming into the consumer are
> pre-aggregated or not.
> # Storm’s metrics collection occurs through a specialized bolt, which in
> addition to potentially affecting system performance, complicates certain
> types of aggregation when the parallelism of that bolt is greater than one.
> In the discussion on the developer mailing list, there is growing consensus
> for replacing Storm’s metrics API with a new API based on Coda Hale’s metrics
> library. This approach has the following benefits:
> # Coda Hale’s metrics library is very stable, performant, well thought out,
> and widely adopted among open source projects (e.g. Kafka).
> # The metrics library provides many existing metric types: Meters, Gauges,
> Counters, Histograms, and more.
> # The library has a pluggable “reporter” API for publishing metrics to
> various systems, with existing implementations for: JMX, console, CSV, SLF4J,
> Graphite, Ganglia.
> # Reporters are straightforward to implement, and can be reused by any
> project that uses the metrics library (i.e. would have broader application
> outside of Storm)
> As noted earlier, the metrics library supports pluggable reporters for
> sending metrics data to other systems, and implementing a reporter is fairly
> straightforward (an example reporter implementation can be found here). For
> example if someone develops a reporter based on Coda Hale’s metrics, it could
> not only be used for pushing Storm metrics, but also for any system that used
> the metrics library, such as Kafka.
> h2. Scope of Effort
> The effort to implement a new metrics API for Storm can be broken down into
> the following development areas:
> # Implement API for Storms internal worker metrics: latencies, queue sizes,
> capacity, etc.
> # Implement API for user defined, topology-specific metrics (exposed via the
> {{org.apache.storm.task.TopologyContext}} class)
> # Implement API for storm daemons: nimbus, supervisor, etc.
> h2. Relationship to Existing Metrics
> This would be a new API that would not affect the existing metrics API. Upon
> completion, the old metrics API would presumably be deprecated, but kept in
> place for backward compatibility.
> Internally the current metrics API uses Storm bolts for the reporting
> mechanism. The proposed metrics API would not depend on any of Storm's
> messaging capabilities and instead use the [metrics library's built-in
> reporter mechanism |
> http://metrics.dropwizard.io/3.1.0/manual/core/#man-core-reporters]. This
> would allow users to use existing {{Reporter}} implementations which are not
> Storm-specific, and would simplify the process of collecting metrics.
> Compared to Storm's {{IMetricCollector}} interface, implementing a reporter
> for the metrics library is much more straightforward (an example can be found
> [here |
> https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/ConsoleReporter.java].
> The new metrics capability would not use or affect the ZooKeeper-based
> metrics used by Storm UI.
> h2. Relationship to JStorm Metrics
> [TBD]
> h2. Target Branches
> [TBD]
> h2. Performance Implications
> [TBD]
> h2. Metrics Namespaces
> [TBD]
> h2. Metrics Collected
> *Worker*
> || Namespace || Metric Type || Description ||
> *Nimbus*
> || Namespace || Metric Type || Description ||
> *Supervisor*
> || Namespace || Metric
[jira] [Resolved] (STORM-2863) Some ras tests fail because of static resource caching
[ https://issues.apache.org/jira/browse/STORM-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2863. - Resolution: Fixed Fix Version/s: 2.0.0 Thanks [~revans2], I merged into master. > Some ras tests fail because of static resource caching > -- > > Key: STORM-2863 > URL: https://issues.apache.org/jira/browse/STORM-2863 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > We noticed that depending on the ordering of tests, some of the RAS tests can > fail because of metrics being cached. We really should clear this static > cache before each test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-2690) resurrect invocation of ISupervisor.assigned() & make Supervisor.launchDaemon() accessible
[
https://issues.apache.org/jira/browse/STORM-2690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2690.
-
Resolution: Fixed
Fix Version/s: 1.0.6
1.1.2
1.2.0
2.0.0
Thanks [~erikdw], I merged into master, 1.x, 1.1.x, 1.0.x branches.
> resurrect invocation of ISupervisor.assigned() & make
> Supervisor.launchDaemon() accessible
> --
>
> Key: STORM-2690
> URL: https://issues.apache.org/jira/browse/STORM-2690
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 2.0.0, 1.1.0, 1.0.3, 1.0.4, 1.1.1, 1.1.2, 1.0.5
>Reporter: Erik Weathers
>Assignee: Erik Weathers
>Priority: Minor
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.0, 1.1.2, 1.0.6
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> As [discussed in
> STORM-2018|https://issues.apache.org/jira/browse/STORM-2018?focusedCommentId=16108307=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16108307],
> that change subtly broke the storm-mesos integration framework because of
> the removal of the invocation of
> [{{ISupervisor.assigned()}}|https://github.com/apache/storm/blob/v1.0.4/storm-core/src/jvm/org/apache/storm/scheduler/ISupervisor.java#L44].
> So this ticket is tracking the reinstatement of that invocation from the
> supervisor core code.
> Also, the
> [{{launchDaemon()}}|https://github.com/apache/storm/blob/v1.0.4/storm-core/src/jvm/org/apache/storm/daemon/supervisor/Supervisor.java#L248]
> method of the {{Supervisor}} is not public, so we had to use reflection to
> allow calling it from the storm-mesos integration. That should be changed
> too.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2843) Flux: properties file not found when loading resources from classpath
[
https://issues.apache.org/jira/browse/STORM-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2843.
-
Resolution: Fixed
Fix Version/s: 1.2.0
2.0.0
Thanks [~vesense], I merged into master, 1.x-branch.
> Flux: properties file not found when loading resources from classpath
> -
>
> Key: STORM-2843
> URL: https://issues.apache.org/jira/browse/STORM-2843
> Project: Apache Storm
> Issue Type: Bug
> Components: Flux
>Affects Versions: 2.0.0, 1.x
>Reporter: Xin Wang
>Assignee: Xin Wang
> Labels: pull-request-available
> Fix For: 2.0.0, 1.2.0
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Scenario: The CI auto-build a fat jar including filter properties file and
> topology yaml file. Filter properties file not found when load resources from
> classpath.
> example:
> {code}storm jar myTopology-0.1.0-SNAPSHOT.jar org.apache.storm.flux.Flux
> --remote --resource --filter dev.properties my_config.yaml{code}
> The dev.properties file cannot be found in classpath.
> After this patch, the FluxParser will load filter properties file as same as
> the way of topology yaml.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (STORM-2862) More flexible logging in multilang (Python, Ruby, JS)
[
https://issues.apache.org/jira/browse/STORM-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303483#comment-16303483
]
Jungtaek Lim commented on STORM-2862:
-
[~hmcc]
You can extract component name and task ID from thread name (only if it is
logged from executor threads) though I agree it is not that easy to build parse
format.
The format of executor thread is like
Thread---executor, and the format of
executorId is \[startTaskId endTaskId].
You could construct regex for {code} Thread-[0-9]+-(.+)-executor\[([0-9]+)
([0-9]+)\] {code}
Btw, I'm not sure we can modify the log format, which may be backward
incompatible for the cases which need to parse log messages.
> More flexible logging in multilang (Python, Ruby, JS)
> -
>
> Key: STORM-2862
> URL: https://issues.apache.org/jira/browse/STORM-2862
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-client, storm-multilang
>Affects Versions: 2.0.0, 1.1.1
>Reporter: Heather McCartney
>Assignee: Heather McCartney
>Priority: Trivial
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We're running a Storm topology written in Python, using storm.py from
> storm-multilang. As well as human-readable logs, the topology is also
> configured to write JSON logs which are sent to ELK.
> At the moment, when storm-core receives a "log" command, it outputs the pid,
> component name, and the message it received, like so:
> {{ShellLog pid:, name: }}
> The code that does this is (currently) in [ShellBolt line
> 254|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/task/ShellBolt.java#L254]
> and [ShellSpout line
> 227|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/spout/ShellSpout.java#L227].
> As well as the pid and component name, it would be great to have the task ID,
> tuple ID, and the ID of the originating tuple - but this would make parsing
> the string even more laborious than it is now, and would make the default log
> message too long.
> Would it be possible to put contextual information like this in the
> [ThreadContext|https://logging.apache.org/log4j/2.x/manual/thread-context.html]
> instead? Then our JSON layout could read from the context instead of parsing
> the string, and human-readable logs could use "%mdc" in the PatternLayout
> format string.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-2869) KafkaSpout discards all pending records when adjusting the consumer position after a commit
[ https://issues.apache.org/jira/browse/STORM-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2869. - Resolution: Fixed Fix Version/s: 1.2.0 2.0.0 Thanks [~Srdo], I merged into master and 1.x-branch. > KafkaSpout discards all pending records when adjusting the consumer position > after a commit > --- > > Key: STORM-2869 > URL: https://issues.apache.org/jira/browse/STORM-2869 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > As part of the STORM-2666 fix the spout clears out waitingToEmit when the > consumer position falls behind the committed offset during a commit. We only > need to do it for the affected partition, and then only for the records that > are behind the committed offset. > Also the validation check in emitOrRetryTuple is slightly too permissive, it > should check whether the current record is behind the committed offset, not > whether the consumer position is behind the committed offset. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2870) FileBasedEventLogger leaks non-daemon ExecutorService which prevents process to be finished
Jungtaek Lim created STORM-2870:
---
Summary: FileBasedEventLogger leaks non-daemon ExecutorService
which prevents process to be finished
Key: STORM-2870
URL: https://issues.apache.org/jira/browse/STORM-2870
Project: Apache Storm
Issue Type: Bug
Components: storm-core
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.0.6
Reporter: Jungtaek Lim
Assignee: Jungtaek Lim
{code}
private void setUpFlushTask() {
ScheduledExecutorService scheduler =
Executors.newSingleThreadScheduledExecutor();
Runnable task = new Runnable() {
@Override
public void run() {
try {
if(dirty) {
eventLogWriter.flush();
dirty = false;
}
} catch (IOException ex) {
LOG.error("Error flushing " + eventLogPath, ex);
throw new RuntimeException(ex);
}
}
};
scheduler.scheduleAtFixedRate(task, FLUSH_INTERVAL_MILLIS,
FLUSH_INTERVAL_MILLIS, TimeUnit.MILLISECONDS);
}
{code}
The code block initializes ExecutorService locally, which served threads are
not daemons so it can prevent JVM to be exit successfully.
Moreover it should be considered as bad case: not labeling thread name. I
observed the process hung and got jstack, but hard to know where is the root,
because leaked thread has default thread name.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (STORM-1188) PartialKeyGrouping missing from storm.thrift (and can't use it via custom_object)
[ https://issues.apache.org/jira/browse/STORM-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16304176#comment-16304176 ] Jungtaek Lim commented on STORM-1188: - [~dan.blanchard] How do you define the topology with accessing PartialKeyGrouping outside of JVM? I guess if PartialKeyGrouping is accessible and initializable, Fields is expected to be accessible too, hence asking the question. Btw, I'm not familiar with backward-compatibility manner of Thrift, so I'm sure I might be missing here, but as far as I googled it (sad) adding field into "union" doesn't look like backward compatible. For "struct" we are adding new things as "optional" but "union" doesn't provide that option. > PartialKeyGrouping missing from storm.thrift (and can't use it via > custom_object) > - > > Key: STORM-1188 > URL: https://issues.apache.org/jira/browse/STORM-1188 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.10.0, 1.0.0, 0.10.1, 1.0.1, 0.10.2, 1.0.2, 1.1.0, > 1.0.3, 1.x, 0.10.3, 1.0.4, 1.1.1, 1.0.5 >Reporter: Dan Blanchard > > I'm working on a Python DSL for Storm to add to streamparse, and as part of > it I realized that the new partial key grouping was never added to the > Grouping struct in storm.thrift, so it's not usable outside of JVM-based > topology definitions (at least not easily). My initial thought was to just > use Grouping.custom_object, but the PartialKeyGrouping constructor takes a > Fields object, which isn't a type defined in storm.thrift, so I can't use it. > The fields grouping explicitly takes a list of strings in storm.thrift, so it > would seem PartialKeyGrouping needs to be added in the same way. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-442) multilang ShellBolt/ShellSpout die() can be hang when Exception happened
[
https://issues.apache.org/jira/browse/STORM-442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-442:
---
Fix Version/s: 0.9.7
> multilang ShellBolt/ShellSpout die() can be hang when Exception happened
>
>
> Key: STORM-442
> URL: https://issues.apache.org/jira/browse/STORM-442
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-multilang
>Affects Versions: 0.9.3
>Reporter: DashengJu
>Assignee: DashengJu
> Fix For: 0.10.0, 0.9.7
>
>
> In ShellBolt, the _readerThread read command from python/shell process, and
> handle like this:
> try {
> ShellMsg shellMsg = _process.readShellMsg();
> ...
> } catch (InterruptedException e) {
> } catch (Throwable t) {
> die(t);
> }
> And in the die function, getProcessTerminationInfoString will read
> getErrorsString() from processErrorStream.
> private void die(Throwable exception) {
>
> String processInfo = _process.getProcessInfoString() +
> _process.getProcessTerminationInfoString();
>
> _exception = new RuntimeException(processInfo, exception);
>
> }
> so when ShellBolt got exception(for example, readShellMsg() throw NPE ) ,
> but it is not an error from sub process, then
> getProcessTerminationInfoString will be hang because processErrorStream have
> no data to read.
> On the other hand, as [~xiaokang] says ShellBolt should fail fast on
> exception ( https://github.com/apache/incubator-storm/pull/46 ) , I think it
> is not a good idea to read error info from stream.
> Because [~xiaokang] 's PR is based old version, so I will move his code to
> this PR, and modify some other place in ShellSpout.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2311) A browser which is outside the cluster cant's access storm ui when storm cluster in a security mode
[ https://issues.apache.org/jira/browse/STORM-2311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2311: Fix Version/s: (was: 1.0.2) > A browser which is outside the cluster cant's access storm ui when storm > cluster in a security mode > > > Key: STORM-2311 > URL: https://issues.apache.org/jira/browse/STORM-2311 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.2 > Environment: linux >Reporter: liuzhaokun > Time Spent: 0.5h > Remaining Estimate: 0h > > when storm cluster in a security mode such as kerberos,We can not use a > browser which is outside the cluster to access storm ui,even if we remove > the configuration item which named ui.filter.There is a mistake like "server > 500",because those methods to get cluster's info for ui can't access the > cluster which is in a security mode -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1838) [storm-kafka-client] the resumed OffsetEntry goes different
[ https://issues.apache.org/jira/browse/STORM-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1838: Fix Version/s: (was: 1.0.1) > [storm-kafka-client] the resumed OffsetEntry goes different > --- > > Key: STORM-1838 > URL: https://issues.apache.org/jira/browse/STORM-1838 > Project: Apache Storm > Issue Type: Bug >Reporter: Jifeng Yin > Time Spent: 10m > Remaining Estimate: 0h > > There are no more new messages after consumer rejoins the group. And it turns > out the old OffsetEntry instance kept commitOffsetsForAckedTuples failing. > the comment "leave the acked offsets as they were to resume where it left > off" doesn't work for me, but the ackedMsgs goes different. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1776) Error when processing event java.io.FileNotFoundException
[
https://issues.apache.org/jira/browse/STORM-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1776:
Fix Version/s: (was: 1.0.0)
> Error when processing event java.io.FileNotFoundException
> -
>
> Key: STORM-1776
> URL: https://issues.apache.org/jira/browse/STORM-1776
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
>Reporter: Mageshkumar
>
> when i am trying to start storm cluster , i got the below error.
> 24628 [Thread-10] INFO o.a.s.d.supervisor - Copying resources at
> jar:file:/D:/.m2/repository/org/apache/storm/flux-core/1.0.0/flux-core-1.0.0.jar!/resources
> to
> C:\Users\MAGESH~1\AppData\Local\Temp\1991ad4f-acbc-44ed-b07b-24643381f7a0\supervisor\stormdist\test-1-1462884218\resources
> 24631 [Thread-10] ERROR o.a.s.event - Error when processing event
> java.io.FileNotFoundException: Source
> 'file:\D:\.m2\repository\org\apache\storm\flux-core\1.0.0\flux-core-1.0.0.jar!\resources'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1368)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.daemon.supervisor$fn__9351.invoke(supervisor.clj:1194)
> ~[storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.MultiFn.invoke(MultiFn.java:243) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9070$fn__9088.invoke(supervisor.clj:582)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9070.invoke(supervisor.clj:581)
> ~[storm-core-1.0.0.jar:1.0.0]
> at org.apache.storm.event$event_manager$fn__8622.invoke(event.clj:40)
> [storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
> 24638 [Thread-8] INFO o.a.s.d.supervisor - Copying resources at
> jar:file:/D:/.m2/repository/org/apache/storm/flux-core/1.0.0/flux-core-1.0.0.jar!/resources
> to
> C:\Users\MAGESH~1\AppData\Local\Temp\f4009546-87fe-4885-a0b0-cb775e8b784f\supervisor\stormdist\test-1-1462884218\resources
> 24638 [Thread-8] ERROR o.a.s.event - Error when processing event
> java.io.FileNotFoundException: Source
> 'file:\D:\.m2\repository\org\apache\storm\flux-core\1.0.0\flux-core-1.0.0.jar!\resources'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1368)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1261)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1230)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.daemon.supervisor$fn__9351.invoke(supervisor.clj:1194)
> ~[storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.MultiFn.invoke(MultiFn.java:243) ~[clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9070$fn__9088.invoke(supervisor.clj:582)
> ~[storm-core-1.0.0.jar:1.0.0]
> at
> org.apache.storm.daemon.supervisor$mk_synchronize_supervisor$this__9070.invoke(supervisor.clj:581)
> ~[storm-core-1.0.0.jar:1.0.0]
> at org.apache.storm.event$event_manager$fn__8622.invoke(event.clj:40)
> [storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
> 24642 [Thread-10] ERROR o.a.s.util - Halting process: ("Error when processing
> an event")
> java.lang.RuntimeException: ("Error when processing an event")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.event$event_manager$fn__8622.invoke(event.clj:48)
> [storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
> 24643 [Thread-8] ERROR o.a.s.util - Halting process: ("Error when processing
> an event")
> java.lang.RuntimeException: ("Error when processing an event")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.0.jar:1.0.0]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
[jira] [Updated] (STORM-1689) set request headersize for logviewer
[ https://issues.apache.org/jira/browse/STORM-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1689: Fix Version/s: (was: 2.0.0) (was: 1.0.0) > set request headersize for logviewer > > > Key: STORM-1689 > URL: https://issues.apache.org/jira/browse/STORM-1689 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Sriharsha Chintalapani >Assignee: Priyank Shah > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1367) Issue with OpaqueTridentKafkaSpout - TridentKafkaConfig getting (java.lang.RuntimeException: kafka.common.OffsetOutOfRangeException)
[
https://issues.apache.org/jira/browse/STORM-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1367:
Fix Version/s: (was: 1.0.0)
> Issue with OpaqueTridentKafkaSpout - TridentKafkaConfig getting
> (java.lang.RuntimeException: kafka.common.OffsetOutOfRangeException)
>
>
> Key: STORM-1367
> URL: https://issues.apache.org/jira/browse/STORM-1367
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core, storm-kafka, storm-multilang
>Affects Versions: 0.10.0
>Reporter: Rakesh Surendra
>Assignee: Sriharsha Chintalapani
> Labels: patch, performance
>
> I'm using trident topology with OpaqueTridentKafkaSpout.
> Code snippet of TridentKafkaConfig i’m using :-
> OpaqueTridentKafkaSpout kafkaSpout = null;
> TridentKafkaConfig spoutConfig = new TridentKafkaConfig(new
> ZkHosts("xxx.x.x.9:2181,xxx.x.x.1:2181,xxx.x.x.2:2181"), "topic_name");
> spoutConfig.scheme = new SchemeAsMultiScheme(new
> StringScheme());
> spoutConfig.forceFromStart = true;
> spoutConfig.fetchSizeBytes = 147483600;
> kafkaSpout = new OpaqueTridentKafkaSpout(spoutConfig);
> I get this runtime exception from one of the workers :-
> java.lang.RuntimeException: storm.kafka.UpdateOffsetException at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:135)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106)
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> at
> backtype.storm.daemon.executor$fn__5694$fn__5707$fn__5758.invoke(executor.clj:819)
> at backtype.storm.util$async_loop$fn__545.invoke(util.clj:479) at
> clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)
> Caused by: storm.kafka.UpdateOffsetException at
> storm.kafka.KafkaUtils.fetchMessages(KafkaUtils.java:186) at
> storm.kafka.trident.TridentKafkaEmitter.fetchMessages(TridentKafkaEmitter.java:132)
> at
> storm.kafka.trident.TridentKafkaEmitter.doEmitNewPartitionBatch(TridentKafkaEmitter.java:113)
> at
> storm.kafka.trident.TridentKafkaEmitter.failFastEmitNewPartitionBatch(TridentKafkaEmitter.java:72)
> at
> storm.kafka.trident.TridentKafkaEmitter.emitNewPartitionBatch(TridentKafkaEmitter.java:79)
> at
> storm.kafka.trident.TridentKafkaEmitter.access$000(TridentKafkaEmitter.java:46)
> at
> storm.kafka.trident.TridentKafkaEmitter$1.emitPartitionBatch(TridentKafkaEmitter.java:204)
> at
> storm.kafka.trident.TridentKafkaEmitter$1.emitPartitionBatch(TridentKafkaEmitter.java:194)
> at
> storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Emitter.emitBatch(OpaquePartitionedTridentSpoutExecutor.java:127)
> at
> storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> at
> storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:370)
> at
> backtype.storm.daemon.executor$fn__5694$tuple_action_fn__5696.invoke(executor.clj:690)
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5615.invoke(executor.clj:436)
> at
> backtype.storm.disruptor$clojure_handler$reify__5189.onEvent(disruptor.clj:58)
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:127)
> ... 6 more
> But when i set spoutConfig.forceFromStart = true... It works fine for a while
> and then fails throwing this exception. I need a trident topology to give out
> accurate exactly-once processing even when the topology is restarted.
> As per some posts, I have tried setting spoutConfig :-
> spoutConfig.maxOffsetBehind = Long.MAX_VALUE;
> spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
> My Kafka retention time is default - 128 Hours i.e. 7 Days and kafka producer
> is sending 6800 messages/second to Storm/Trident topology. I have gone
> through most of the posts, but none of them seem to solve this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-1469) Unable to deploy large topologies on apache storm
[ https://issues.apache.org/jira/browse/STORM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1469: Fix Version/s: (was: 1.0.0) > Unable to deploy large topologies on apache storm > - > > Key: STORM-1469 > URL: https://issues.apache.org/jira/browse/STORM-1469 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 1.0.0, 2.0.0 >Reporter: Rudra Sharma >Assignee: Kishor Patil > Time Spent: 10m > Remaining Estimate: 0h > > When deploying to a nimbus a topology which is larger in size >17MB, we get > an exception. In storm 0.9.3 this could be mitigated by using the following > config on the storm.yaml to increse the buffer size to handle the topology > size. i.e. 50MB would be > nimbus.thrift.max_buffer_size: 5000 > This configuration does not resolve the issue in the master branch of storm > and we cannot deploy topologies which are large in size. > Here is the log on the client side when attempting to deploy to the nimbus > node: > java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException > at > backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:251) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:272) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:155) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > com.trustwave.siem.storm.topology.deployer.TopologyDeployer.deploy(TopologyDeployer.java:149) > [siem-ng-storm-deployer-cloud.jar:] > at > com.trustwave.siem.storm.topology.deployer.TopologyDeployer.main(TopologyDeployer.java:87) > [siem-ng-storm-deployer-cloud.jar:] > Caused by: org.apache.thrift7.transport.TTransportException > at > org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:77) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:238) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:222) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > at > backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:237) > ~[storm-core-0.11.0-SNAPSHOT.jar:0.11.0-SNAPSHOT] > ... 4 more > Here is the log on the server side (nimbus.log): > 2016-01-13 10:48:07.206 o.a.s.d.nimbus [INFO] Cleaning inbox ... deleted: > stormjar-c8666220-fa19-426b-a7e4-c62dfb57f1f0.jar > 2016-01-13 10:55:09.823 o.a.s.d.nimbus [INFO] Uploading file from client to > /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar > 2016-01-13 10:55:11.910 o.a.s.d.nimbus [INFO] Finished uploading file from > client: > /var/storm-data/nimbus/inbox/stormjar-80ecdf05-6a25-4281-8c78-10062ac5e396.jar > 2016-01-13 10:55:12.084 o.a.t.s.AbstractNonblockingServer$FrameBuffer [WARN] > Exception while invoking! > org.apache.thrift7.transport.TTransportException: Frame size (17435758) > larger than max length (16384000)! > at > org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:137) > at > org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101) > at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:86) > at > org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) > at > org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) > at
[jira] [Updated] (STORM-838) Upgrade Jackson
[ https://issues.apache.org/jira/browse/STORM-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-838: --- Fix Version/s: (was: 1.0.0) > Upgrade Jackson > --- > > Key: STORM-838 > URL: https://issues.apache.org/jira/browse/STORM-838 > Project: Apache Storm > Issue Type: Dependency upgrade > Components: storm-core >Reporter: Sergio Fernández >Priority: Minor > > Currently the project is using a very old Jackson version (2.3.1). There are > already many new (and better versions: > http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.fasterxml.jackson.core%22%20AND%20a%3A%22jackson-core%22 > So it'd be great to upgrade to 2.5.3. > I can provide a patch if you agree on the request. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1777) Backport KafkaBolt from Storm 1.0.0 to 0.10.0 to support properties configuration method
[ https://issues.apache.org/jira/browse/STORM-1777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-1777: Fix Version/s: (was: 0.10.0) > Backport KafkaBolt from Storm 1.0.0 to 0.10.0 to support properties > configuration method > > > Key: STORM-1777 > URL: https://issues.apache.org/jira/browse/STORM-1777 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 0.10.0 >Reporter: Ambud Sharma >Assignee: Ambud Sharma > > Storm KafkaBolt requires Map object supplied under the > kafka.broker.properties which makes the configuration values global i.e. > there can be only one set of KafkaBolts in a given topology. > This issue has already been resolved in Storm 1.0.0 by introducing the > withProducerProperties method. > This ticket is to back port those changes to 0.10.0 so that we can use the > community version rather than project specific back ports. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-586) Trident kafka spout fails instead of updating offset when kafka offset is out of range.
[
https://issues.apache.org/jira/browse/STORM-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-586:
---
Fix Version/s: (was: 0.10.0)
> Trident kafka spout fails instead of updating offset when kafka offset is out
> of range.
> ---
>
> Key: STORM-586
> URL: https://issues.apache.org/jira/browse/STORM-586
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Critical
>
> Trident KafkaEmitter does not catch the newly added UpdateOffsetException
> which results in the spout failing repeatedly instead of automatically
> updating the offset to earliest time.
> PROBLEM: Exception while using the Trident Kafka Spout.
> {code}
> 2014-12-04 18:38:03 b.s.util ERROR Async loop died!
> java.lang.RuntimeException: storm.kafka.UpdateOffsetException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$fn_4195$fn4207$fn_4254.invoke(executor.clj:745)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at backtype.storm.util$async_loop$fn__442.invoke(util.clj:436)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at clojure.lang.AFn.run(AFn.java:24) clojure-1.4.0.jar:na
> at java.lang.Thread.run(Thread.java:745) na:1.7.0_71
> Caused by: storm.kafka.UpdateOffsetException: null
> at storm.kafka.KafkaUtils.fetchMessages(KafkaUtils.java:186) ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.fetchMessages(TridentKafkaEmitter.java:132)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.doEmitNewPartitionBatch(TridentKafkaEmitter.java:113)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.failFastEmitNewPartitionBatch(TridentKafkaEmitter.java:72)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.access$400(TridentKafkaEmitter.java:46)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter$2.emitPartitionBatchNew(TridentKafkaEmitter.java:233)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter$2.emitPartitionBatchNew(TridentKafkaEmitter.java:225)
> ~stormjar.jar:na
> at
> storm.trident.spout.PartitionedTridentSpoutExecutor$Emitter$1.init(PartitionedTridentSpoutExecutor.java:125)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.state.RotatingTransactionalState.getState(RotatingTransactionalState.java:83)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.state.RotatingTransactionalState.getStateOrCreate(RotatingTransactionalState.java:110)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.spout.PartitionedTridentSpoutExecutor$Emitter.emitBatch(PartitionedTridentSpoutExecutor.java:121)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:369)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$fn_4195$tuple_action_fn_4197.invoke(executor.clj:630)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__4118.invoke(executor.clj:398)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.disruptor$clojure_handler$reify__723.onEvent(disruptor.clj:58)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:99)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> ... 6 common frames omitted
> 2014-12-04 18:38:03 b.s.d.executor ERROR
> java.lang.RuntimeException: storm.kafka.UpdateOffsetException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~[storm-core-0.9.1.2.1.7.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-2292) Kafka spout enhancement, for our of range edge cases
[ https://issues.apache.org/jira/browse/STORM-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-2292: Fix Version/s: (was: 0.10.0) > Kafka spout enhancement, for our of range edge cases > > > Key: STORM-2292 > URL: https://issues.apache.org/jira/browse/STORM-2292 > Project: Apache Storm > Issue Type: Improvement > Components: storm-kafka >Affects Versions: 0.10.0 >Reporter: WayneZhou >Assignee: Hugo Louro > Attachments: KafkaSpout.java.txt > > Original Estimate: 336h > Remaining Estimate: 336h > > @hmcl and all, we have communicated via email for a while and going forward > let's talk in this thread so everyone is in same page. > Base on the spout from the community(written by you), we have several fixes > and it worked quite stable in our production for about 6 months. > We want to share the latest spout to you and could you please kindly help > review and merge to the community version if any fix is reasonable? we want > to avoid diverging too much from the community version. > Below are our major fixes: > For failed message, in next tuple method, originally the spout seek back to > the non-continuous offset, so the failed message will be polled again for > retry, say we seek back to message 10 for retry, now if kafka log file was > purged, earliest offset is 1000, it means we will seek to 10 but reset to > 1000 as per the reset policy, and we cannot poll the message 10, so spout not > work. > Our fix is: we manually catch the out of range exception, commit the offset > to earliest offset first, then seek to the earliest offset > Currently the way to find next committed offset is very complex, under some > edge cases – a), if no message acked back because bolt has some issue or > cannot catch up with the spout emit; b) seek back is happened frequently and > it is much faster than the message be acked back > We give each message a status – None, emit, acked, failed(if failed number is > bigger than the maximum retry, set to acked) > One of our use cases need ordering in partition level, so after seek back for > retry, we re-emit all the follow messages again no matter they have emitted > or not, if possible, maybe you can give an option here to configure it – > either re-emit all the message from the failed one, or just emit the failed > one, same as current version. > We record the message count for acked, failed, emitted, just for statistics. > Could you please kindly help review and let us know if you can merge it into > the community version? Any comments/concern pls feel free to let us know. > Btw, our code is attached in this Jira. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-1021) HeartbeatExecutorService issue in Shell Spout\Bolt
[
https://issues.apache.org/jira/browse/STORM-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1021:
Fix Version/s: (was: 0.10.0)
> HeartbeatExecutorService issue in Shell Spout\Bolt
> --
>
> Key: STORM-1021
> URL: https://issues.apache.org/jira/browse/STORM-1021
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-multilang
>Affects Versions: 0.10.0, 0.9.5
> Environment: Alpine Linux 3.2, openjdk-7
>Reporter: Oleh Hordiichuk
> Labels: easyfix
>
> ShellSpout class (and it seems that this touches ShellBolt as well) doesn't
> restart when hearbeat timeout occurs. To reproduce this bug you should do the
> following:
> 1. Set supervisor.worker.timeout.secs property to e.g. 1;
> 2. Create a shell spout (as a standalone application, not Java class) that
> hangs for more than 1 second and doesn't respond on heartbeat messages, e.g.
> Thread.Sleep(5000);
> 3. After timeout Storm will try to kill the shell spout process with calling
> die function:
> https://github.com/apache/storm/blob/v0.10.0-beta/storm-core/src/jvm/backtype/storm/spout/ShellSpout.java#L237
> 4. The "die" function will call heartBeatExecutorService.shutdownNow()
> function that raises InterruptedException, which is not caughted by the
> calling thread. In a result topology stops working properly, however you may
> see it in ./storm list.
> I'm not Java developer and thus I'm not sure whether code below is valid,
> however it seems to fix the problem:
> private void die(Throwable exception) {
> heartBeatExecutorService.shutdownNow();
> try {
> heartBeatExecutorService.awaitTermination(5, TimeUnit.SECONDS);
> } catch (InterruptedException e) {
> LOG.error("await catch ", e);
> }
> _collector.reportError(exception);
> _process.destroy();
> System.exit(11);
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (STORM-487) Remove storm.cmd, no need to duplicate work python runs on windows too.
[ https://issues.apache.org/jira/browse/STORM-487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-487. Resolution: Fixed Assignee: Jungtaek Lim It was fixed via letting "storm" script be compatible with Windows. > Remove storm.cmd, no need to duplicate work python runs on windows too. > --- > > Key: STORM-487 > URL: https://issues.apache.org/jira/browse/STORM-487 > Project: Apache Storm > Issue Type: Improvement > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Jungtaek Lim > Labels: newbie > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (STORM-586) Trident kafka spout fails instead of updating offset when kafka offset is out of range.
[
https://issues.apache.org/jira/browse/STORM-586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-586.
Resolution: Fixed
Fix Version/s: 0.10.0
Marking as resolved since the patch got merged.
> Trident kafka spout fails instead of updating offset when kafka offset is out
> of range.
> ---
>
> Key: STORM-586
> URL: https://issues.apache.org/jira/browse/STORM-586
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-kafka
>Affects Versions: 0.9.3
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Critical
> Fix For: 0.10.0
>
>
> Trident KafkaEmitter does not catch the newly added UpdateOffsetException
> which results in the spout failing repeatedly instead of automatically
> updating the offset to earliest time.
> PROBLEM: Exception while using the Trident Kafka Spout.
> {code}
> 2014-12-04 18:38:03 b.s.util ERROR Async loop died!
> java.lang.RuntimeException: storm.kafka.UpdateOffsetException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$fn_4195$fn4207$fn_4254.invoke(executor.clj:745)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at backtype.storm.util$async_loop$fn__442.invoke(util.clj:436)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at clojure.lang.AFn.run(AFn.java:24) clojure-1.4.0.jar:na
> at java.lang.Thread.run(Thread.java:745) na:1.7.0_71
> Caused by: storm.kafka.UpdateOffsetException: null
> at storm.kafka.KafkaUtils.fetchMessages(KafkaUtils.java:186) ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.fetchMessages(TridentKafkaEmitter.java:132)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.doEmitNewPartitionBatch(TridentKafkaEmitter.java:113)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.failFastEmitNewPartitionBatch(TridentKafkaEmitter.java:72)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter.access$400(TridentKafkaEmitter.java:46)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter$2.emitPartitionBatchNew(TridentKafkaEmitter.java:233)
> ~stormjar.jar:na
> at
> storm.kafka.trident.TridentKafkaEmitter$2.emitPartitionBatchNew(TridentKafkaEmitter.java:225)
> ~stormjar.jar:na
> at
> storm.trident.spout.PartitionedTridentSpoutExecutor$Emitter$1.init(PartitionedTridentSpoutExecutor.java:125)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.state.RotatingTransactionalState.getState(RotatingTransactionalState.java:83)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.state.RotatingTransactionalState.getStateOrCreate(RotatingTransactionalState.java:110)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.spout.PartitionedTridentSpoutExecutor$Emitter.emitBatch(PartitionedTridentSpoutExecutor.java:121)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:369)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$fn_4195$tuple_action_fn_4197.invoke(executor.clj:630)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__4118.invoke(executor.clj:398)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.disruptor$clojure_handler$reify__723.onEvent(disruptor.clj:58)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:99)
> ~storm-core-0.9.1.2.1.7.0-784.jar:0.9.1.2.1.7.0-784
> ... 6 common frames omitted
> 2014-12-04 18:38:03 b.s.d.executor ERROR
> java.lang.RuntimeException: storm.kafka.UpdateOffsetException
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:107)
> ~[storm-core-0.9.1.2.1.7.0
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (STORM-755) An exception occured while executing the Java class. fyp-storm-try.src.jvm.Topology
[
https://issues.apache.org/jira/browse/STORM-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-755:
---
Fix Version/s: (was: 0.9.4)
> An exception occured while executing the Java class.
> fyp-storm-try.src.jvm.Topology
>
>
> Key: STORM-755
> URL: https://issues.apache.org/jira/browse/STORM-755
> Project: Apache Storm
> Issue Type: Question
> Components: storm-core
>Affects Versions: 0.9.4
> Environment: Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-44-generic x86_64)
> at Amazon Web Services EC2
>Reporter: Yuen
>Assignee: Yuen
> Labels: build, exec-maven-plugin, maven, newbie
>
> ubuntu@ip-10-0-0-101:~/storm/examples/fyp-storm-try$ sudo mvn exec:java -D
> storm.topology=fyp-storm-try.src.jvm.Topology
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.storm:fyp-storm-try:jar:0.9.4
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-javadoc-plugin is missing. @
> org.apache.storm:storm:0.9.4,
> /root/.m2/repository/org/apache/storm/storm/0.9.4/storm-0.9.4.pom, line 694,
> column 21
> [WARNING] 'reporting.plugins.plugin.version' for
> org.apache.maven.plugins:maven-surefire-report-plugin is missing. @
> org.apache.storm:storm:0.9.4,
> /root/.m2/repository/org/apache/storm/storm/0.9.4/storm-0.9.4.pom, line 660,
> column 21
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> [INFO]
> [INFO]
>
> [INFO] Building fyp-storm-try 0.9.4
> [INFO]
>
> [INFO]
> [INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try >>>
> [INFO]
> [INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try <<<
> [INFO]
> [INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ fyp-storm-try ---
> [WARNING]
> java.lang.ClassNotFoundException: fyp-storm-try.src.jvm.Topology
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:285)
> at java.lang.Thread.run(Thread.java:745)
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 3.838s
> [INFO] Finished at: Sun Apr 05 09:09:28 UTC 2015
> [INFO] Final Memory: 9M/22M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java
> (default-cli) on project fyp-storm-try: An exception occured while executing
> the Java class. fyp-storm-try.src.jvm.Topology -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> Below are my pom.xml:
> ?xml version="1.0" encoding="UTF-8"?>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
>
> storm
> org.apache.storm
> 0.9.4
> ../../pom.xml
>
> org.apache.storm
> fyp-storm-try
> jar
> fyp-storm-try
>
>
> junit
> junit
> test
>
>
> org.testng
> testng
> 6.8.5
> test
>
>
> org.mockito
> mockito-all
> test
>
>
> org.easytesting
> fest-assert-core
> 2.0M8
> test
>
>
> org.jmock
> jmock
> 2.6.0
> test
>
>
> org.twitter4j
> twitter4j-stream
> 3.0.3
>
>
> org.apache.storm
> storm-core
> ${project.version}
>
> provided
>
>
>
[jira] [Updated] (STORM-469) Storm UI Last Error Detail Insufficient for debugging
[ https://issues.apache.org/jira/browse/STORM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-469: --- Fix Version/s: (was: 0.10.0) > Storm UI Last Error Detail Insufficient for debugging > - > > Key: STORM-469 > URL: https://issues.apache.org/jira/browse/STORM-469 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.2-incubating > Environment: java 1.7 update 67, debian wheezy and squeeze >Reporter: Jason Kania >Assignee: Rekha Joshi > > The error text that is captured in the Storm UI is insufficient to debug the > issue as it gets cut off and does not link to the actual error to allow a > person to get more detail. This means that the debugging is unnecessarily > blind. This prevents diagnosis of the error. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-705) Add BigPetStore stream to examples/
[ https://issues.apache.org/jira/browse/STORM-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated STORM-705: --- Fix Version/s: (was: 0.9.4) > Add BigPetStore stream to examples/ > --- > > Key: STORM-705 > URL: https://issues.apache.org/jira/browse/STORM-705 > Project: Apache Storm > Issue Type: Task > Components: examples, storm-core >Reporter: jay vyas > > I work on the Apache BigTop project (bigpetstore-transaction-queue), and we > curate an internal data generator which is ideally suited for testing data > streams. > We could directly add this into the examples/ sections, as this will give > users a less abstract and more universal entry point to building storm > topologies. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
