[GitHub] storm issue #2928: STORM-3270: Build Storm with Java 11, excluding some inco...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2928
  
The build failed because of a hiccup when downloading dependencies. See 
https://travis-ci.org/srdo/storm/builds/468124888 instead.


---

[GitHub] storm pull request #2928: STORM-3270: Build Storm with Java 11, excluding so...

[srdo]

GitHub user srdo opened a pull request:

https://github.com/apache/storm/pull/2928

STORM-3270: Build Storm with Java 11, excluding some incompatible mod…

…ules

https://issues.apache.org/jira/browse/STORM-3270

All Hadoop-related modules are excluded from tests, since Hadoop doesn't 
work with Java 11 yet. Same for Cassandra.

Libraries I'm aware of that need to be updated, other than Netty which is 
bumped in this PR:
* Clojure to 1.10 (not released yet, currently going through RCs)
* Hadoop/Hive/HBase need to be updated once 
https://issues.apache.org/jira/browse/HADOOP-15338 is resolved and released.
* Cassandra needs to be updated to 4.0.0. This update should be easy, since 
we only depend on it in tests, the production code only relies on the Datastax 
driver, which should be compatible with most Cassandra versions.
* Dropwizard Metrics should probably be updated to 4.x, but we're blocked 
by https://issues.apache.org/jira/browse/CASSANDRA-14667. I'm not sure how 
critical the fixes for Metrics are.
* Kafka should probably be upgraded to 2.1.0. I don't think it is an issue 
for storm-kafka-client, since users can just specify another client library 
version, but for storm-kafka-monitor it might be necessary for us to upgrade.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srdo/storm STORM-3270

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/storm/pull/2928.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2928


commit ec634bd65087d1b67a53c99ef3d5d3efeaea0afe
Author: Stig Rohde Døssing 
Date:   2018-09-25T16:06:43Z

STORM-3270: Build Storm with Java 11, excluding some incompatible modules




---

[GitHub] storm pull request #2907: STORM-2990, STORM-3279: Fix issue where Kafka Trid...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2907


---

[GitHub] storm issue #2907: STORM-2990, STORM-3279: Fix issue where Kafka Trident spo...

[srdo]

Github user srdo commented on the issue:

https://github.com/apache/storm/pull/2907
  
Thanks for the reviews, merged to master.


---

[GitHub] storm pull request #2907: STORM-2990, STORM-3279: Fix issue where Kafka Trid...

[revans2]

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2907#discussion_r241782960
  
--- Diff: 
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutEmitter.java
 ---
@@ -170,15 +170,25 @@ public void reEmitPartitionBatch(TransactionAttempt 
tx, TridentCollector collect
 
 seek(currBatchTp, lastBatchMeta);
 
-final ConsumerRecords records = 
consumer.poll(pollTimeoutMs);
-LOG.debug("Polled [{}] records from Kafka.", records.count());
+final List> records = 
consumer.poll(pollTimeoutMs).records(currBatchTp);
+LOG.debug("Polled [{}] records from Kafka.", records.size());
 
 if (!records.isEmpty()) {
 for (ConsumerRecord record : records) {
 emitTuple(collector, record);
 }
-// build new metadata
-currentBatch = new 
KafkaTridentSpoutBatchMetadata(records.records(currBatchTp), 
this.topologyContext.getStormId());
+// build new metadata based on emitted records
+currentBatch = new KafkaTridentSpoutBatchMetadata(
+records.get(0).offset(),
+records.get(records.size() - 1).offset(),
+topologyContext.getStormId());
+} else {
+//Build new metadata based on the consumer position.
+//We want the next emit to start at the current consumer 
position,
+//so make a meta that indicates that position - 1 is the 
last emitted offset
+//This helps us avoid cases like STORM-3279, and 
simplifies the seek logic.
+long lastEmittedOffset = consumer.position(currBatchTp) - 
1;
--- End diff --

okay sounds good.


---

[GitHub] storm issue #2907: STORM-2990, STORM-3279: Fix issue where Kafka Trident spo...

[revans2]

Github user revans2 commented on the issue:

https://github.com/apache/storm/pull/2907
  
+1


---

Re: Storm 2.0.0 release?

[Bobby Evans]

Sorry I was out at a conference for the past week, and have been heads down
on a different project for a while before that.  I'll respond to the JIRA.
I am happy to let it go in.

Thanks,

Bobby

On Thu, Dec 13, 2018 at 1:07 PM Stig Rohde Døssing 
wrote:

> I think STORM-2990/3279 is ready. Bobby had a question (
> https://github.com/apache/storm/pull/2907#discussion_r234329136) regarding
> whether Kafka offsets loop, but I wasn't sure where he was going with it,
> so I didn't want to merge prematurely.
>
> I agree that we can postpone STORM-2720. As far as I know it's waiting for
> STORM-2990 to go in, since it's going to be touching the same code.
>
> Den tor. 13. dec. 2018 kl. 19.54 skrev Roshan Naik
> :
>
> >  Sounds like -  https://github.com/apache/storm/pull/2913 (STORM-3290)
> is
> > merged -  https://github.com/apache/storm/pull/2908 (STORM-3276) is
> > nearly complete and may need some small tweaks. -
> > https://github.com/apache/storm/pull/2907  (STORM-2990, STORM-3279)
> > appears ready to be committed ?
> >
> > and- https://github.com/apache/storm/pull/2911   - (STORM-2720)  seems a
> > bit inactive and may not be critical enough to wait on.
> >
> > -roshan
> > On Monday, November 26, 2018, 9:49:30 AM PST, Stig Rohde Døssing <
> > stigdoess...@gmail.com> wrote:
> >
> >  I would like to get at least https://github.com/apache/storm/pull/2913
> > (breaking changes) and https://github.com/apache/storm/pull/2908
> > (regression) in.
> >
> > I think it would be nice to also get
> > https://github.com/apache/storm/pull/2907 and
> > https://github.com/apache/storm/pull/2911 in, but if we're in a hurry
> they
> > could go in the next release.
> >
> > Den man. 26. nov. 2018 kl. 17.37 skrev Julien Nioche <
> > lists.digitalpeb...@gmail.com>:
> >
> > > Hi devs,
> > >
> > > Is there anything blocking the release of Storm 2.0? Any idea of when
> it
> > > could happen?
> > >
> > > Thanks
> > >
> > > Julien
> > >
> > > --
> > >
> > > *Open Source Solutions for Text Engineering*
> > >
> > > http://www.digitalpebble.com
> > > http://digitalpebble.blogspot.com/
> > > #digitalpebble 
> > >
> >
>