Re: Is Java8 supported by Storm?

[Ambud Sharma]

Java 8 and Storm 1.x work just fine. I have used Java 8 and 0.10 is
production as well.

On Mon, Jul 10, 2017 at 7:06 AM, Bobby Evans  wrote:

> I filed https://issues.apache.org/jira/browse/STORM-2620 to update the
> docs and put up some pull requests to clarify things for the 1.x and 2.x
> branches (linked from the JIRA).
>
>
>
>
> - Bobby
>
>
>
> On Monday, July 10, 2017, 8:28:38 AM CDT, Bobby Evans 
> wrote:
>
>
> The integration tests for 0.10 versions of storm were run against java 6.
> The 1.x versions of storm are run against both java 7 and java 8.  The 2.x
> versions of storm are currently only run against java 8, but we expect to
> add in java 9 once that travis ci supports it.
>
> Internally my team runs storm only on java8 so it should work.   I will
> file a JIRA to update the docs to indicate what it correct.
>
>
> - Bobby
>
>
>
> On Monday, July 10, 2017, 8:18:49 AM CDT, Xin Wang 
> wrote:
>
>
> I'm not sure the release time of Storm 2.0, the first meta version maybe
> several weeks later. Storm 1.1.1 or 1.2 will be released ASAP.
>
> -Xin
>
> 2017-07-10 20:09 GMT+08:00 Hector Garzón 
> :
>
> Ok, thanks for the info.
> When is it planned to be released Storm v2.0? We need to decide if migrate
> all our code to java7 or wait to Storm 2.0 release.
>
> Thanks for your help.
>
> 2017-07-10 13:38 GMT+02:00 Xin Wang :
>
> Hi Hector,
>
> The Storm 2.0 needs JDK 1.8. Seems like the documents should be updated.
> And you can write a bolt using Java8 Lambda. Example:
> https://github.com/apache/stor m/blob/master/examples/storm-
> starter/src/jvm/org/apache/ storm/starter/LambdaTopology. java
> 
>
> Thanks,
> Xin
>
> 2017-07-10 19:13 GMT+08:00 Hector Garzón  >:
>
> Hello,
> As indicated on official documentation (http://storm.apache.org/relea
> ses/2.0.0-SNAPSHOT/Setting-up- a-Storm-cluster.html
> )
> the only java version tested with Storm is Java 7. And there is no guaranty
> of Storm working on another java version.
>
> So, my question is if there is any plan to support java8 (I can't see any
> project roadmap)?
>
> Has anybody already used Java8 on cluster mode on Production environment?
>
> Thanks
>
>
>
>
>

Re: Decreasing value of Complete Latency in Storm UI

[Ambud Sharma]

If I may add, it is also explained by the potential surge of tuples when
topology starts which will eventually reach an equilibrium which the normal
latency of your topology components.

On Jul 14, 2017 4:29 AM, "preethini v"  wrote:

> Hi,
>
> I am running WordCountTopology with 3 worker nodes. The parallelism of
> spout, split and count is 5, 8 and 12 respectively. I have enabled acking
> to measure the complete latency of the topology.
>
> I am considering  complete latency as a measure of end-to-end latency.
>
> The Complete latency is the time a tuple is emitted by a Spout until
> Spout.ack() is called.  Thus, it is the time from tuple being emitted,
> the tuple processing time, the time it spends in the internal input/output
> buffers and until the ack for the tuple is received by the Spout.
>
> The stats from storm UI show that the complete latency for a topology
> keeps decreasing with time.
>
> 1. Is this normal?
> 2. If yes, What explains the continuous decreasing complete latency value?
> 3. Is complete latency a good measure of end-to-end latency of a topology?
>
> Thanks,
> Preethini
>

Re: Storm topology freezes and does not process tuples from Kafka

[Ambud Sharma]

Please check if you have orphan workers. Orphan workers happen when a
topology is redeployed in a short period of time and the old workers
haven't yet been cleaned up.

Check this running ps aux|grep java or specific jar keyword if you have one.

On Jul 14, 2017 11:17 PM, "Sreeram"  wrote:

> Thank you Taylor for replying.
>
> I checked the worker logs and there were no messages that get printed
> once the worker goes into freeze( I had the worker logs set at ERROR
> level ).
>
> Regarding the components, I have kafka spout in the topology and bolts
> that write to HBase.
>
> I had been rebalancing the topology with wait time of 30seconds. Is it
> recommended to match it with topology.message.timeout.secs ?
>
> Please let me know if you need any specific info.
>
> Thanks,
> Sreeram
>
> On Sat, Jul 15, 2017 at 12:39 AM, P. Taylor Goetz 
> wrote:
> >> Supervisor log at the time of freeze looks like below
> >>
> >> 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >
> >
> > There are two situations where you would see those messages: When a
> topology is first deployed, and when a worker has died and is being
> restarted.
> >
> > I suspect the latter. Have you looked at the worker logs for any
> indication that the workers might be crashing and what might be causing it?
> >
> > What components are involved in you’re topology?
> >
> > -Taylor
> >
> >
> >> On Jul 12, 2017, at 5:26 AM, Sreeram  wrote:
> >>
> >> Hi,
> >>
> >> I am observing that my storm topology intermediately freezes and does
> >> not continue to process tuples from Kafka. This happens frequently and
> >> when it happens this freeze lasts for 5 to 15 minutes. No content is
> >> written to any of the worker log files during this time.
> >>
> >> The version of storm I use is 1.0.2 and Kafka version is 0.9.0.
> >>
> >> Any suggestions to solve the issue ?
> >>
> >> Thanks,
> >> Sreeram
> >>
> >> Supervisor log at the time of freeze looks like below
> >>
> >> 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:47.212 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:47.712 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:48.213 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:48.713 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:49.213 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:49.713 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:50.214 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >> 2017-07-12 14:38:50.714 o.a.s.d.supervisor [INFO]
> >> d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
> >>
> >>
> >> Thread stacks (sample)
> >> Most of worker threads during this freeze period look like one of the
> >> below two stack traces.
> >>
> >> Thread 104773: (state = BLOCKED)
> >> - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
> >> information may be imprecise)
> >> - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> >> long) @bci=20, line=215 (Compiled frame)
> >> - java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(
> java.util.concurrent.SynchronousQueue$TransferStack$SNode,
> >> boolean, long) @bci=160, line=460 (Compil
> >> ed frame)
> >> - java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.
> lang.Object,
> >> boolean, long) @bci=102, line=362 (Compiled frame)
> >> - java.util.concurrent.SynchronousQueue.poll(long,
> >> java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame)
> >> - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134,
> >> line=1066 (Compiled frame)
> >> - java.util.concurrent.ThreadPoolExecutor.runWorker(
> java.util.concurrent.ThreadPoolExecutor$Worker)
> >> @bci=26, line=1127 (Compiled frame)
> >> - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5,
> >> line=617 (Compiled frame)
> >> - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
> >>
> >> Thread 147495: (state = IN_NATIVE)
> >> - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
> >> (Compiled frame; information may be imprecise)
> >> - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled
> frame)
> >> - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93
> (Compiled frame)
> >> - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86
> >> (Compiled frame)
> >> - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled
> frame)
> >> - org.apache.kafka.common.network.Selector.select(long) 

Re: storm-hive not updating table record when table has a primary key

[Federico D'Ambrosio]

I realized that this issue is related to Hive, since as of Hive2, it
doesn't support yet uniqueness of keys. This feature is however listed as
being available in Hive 3 (
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310843=12340268).


So, being this feature not available, is there any way to check this
constraint in the storm topology? Using an ad-hoc HiveUpdater or HiveState?

2017-07-16 17:46 GMT+02:00 Federico D'Ambrosio <
federico.dambro...@smartlab.ws>:

> Hi,
>
> I have a Trident Storm topology which ingest data from a Kafka Spout to an
> Hive table. This table is set with a primary key, an id I generate. Now,
> shouldn't the presence of a primary key
> guarantee of an update transaction, rather than an insert?
>
> For example, in my table there's a row:
>
> |id|origin|timestamp|
> | 2| "LAX"| 15 |
>
> then, new data comes in and I get
>
> |id|origin|timestamp|
> ---
> | 2| "LAX"| 15 |
> | 2| "LAX"| 150150 |
>
> I'm wondering if I'm missing something about the ACID support of Hive, and
> its implementation in storm-hive, or rather I'm just getting the expected
> result.
>
> Thanks for your attention,
> Federico
>

storm-hive not updating table record when table has a primary key

[Federico D'Ambrosio]

Hi,

I have a Trident Storm topology which ingest data from a Kafka Spout to an
Hive table. This table is set with a primary key, an id I generate. Now,
shouldn't the presence of a primary key
guarantee of an update transaction, rather than an insert?

For example, in my table there's a row:

|id|origin|timestamp|
| 2| "LAX"| 15 |

then, new data comes in and I get

|id|origin|timestamp|
---
| 2| "LAX"| 15 |
| 2| "LAX"| 150150 |

I'm wondering if I'm missing something about the ACID support of Hive, and
its implementation in storm-hive, or rather I'm just getting the expected
result.

Thanks for your attention,
Federico