[jira] [Assigned] (STORM-1635) Master branch broken! Exception thrown immediately after submitting topology
[ https://issues.apache.org/jira/browse/STORM-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fang reassigned STORM-1635: Assignee: John Fang > Master branch broken! Exception thrown immediately after submitting topology > > > Key: STORM-1635 > URL: https://issues.apache.org/jira/browse/STORM-1635 > Project: Apache Storm > Issue Type: Bug >Reporter: Boyang Jerry Peng >Assignee: John Fang >Priority: Critical > > Exception thrown: > 2016-03-16 22:20:14.940 o.a.s.t.s.AbstractNonblockingServer$FrameBuffer > [ERROR] Unexpected throwable while invoking! > java.lang.NullPointerException > at clojure.lang.RT.intCast(RT.java:1163) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191$iter__2267__2271$fn__2272.invoke(nimbus.clj:1837) > at clojure.lang.LazySeq.sval(LazySeq.java:40) > at clojure.lang.LazySeq.seq(LazySeq.java:49) > at clojure.lang.RT.seq(RT.java:507) > at clojure.core$seq__4128.invoke(core.clj:137) > at clojure.core$dorun.invoke(core.clj:3009) > at clojure.core$doall.invoke(core.clj:3025) > at > org.apache.storm.daemon.nimbus$fn__2162$exec_fn__589__auto__$reify__2191.getClusterInfo(nimbus.clj:1822) > at > org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3724) > at > org.apache.storm.generated.Nimbus$Processor$getClusterInfo.getResult(Nimbus.java:3708) > at > org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39) > at > org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:158) > at > org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518) > at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > This is also causing UI to not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1537) Upgrade to Kryo 3
[ https://issues.apache.org/jira/browse/STORM-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197226#comment-15197226 ] ASF GitHub Bot commented on STORM-1537: --- GitHub user abhishekagarwal87 opened a pull request: https://github.com/apache/storm/pull/1223 STORM-1537: Upgrade to kryo 3 You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhishekagarwal87/storm kryo3-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1223.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1223 commit 514ba4e2a3aa8b13b46b9426d3619922428c93db Author: Abhishek Agarwal Date: 2016-03-08T20:13:23Z STORM-1537: Upgrade to kryo 3 > Upgrade to Kryo 3 > - > > Key: STORM-1537 > URL: https://issues.apache.org/jira/browse/STORM-1537 > Project: Apache Storm > Issue Type: Improvement >Affects Versions: 1.0.0 >Reporter: Oscar Boykin >Assignee: P. Taylor Goetz > > In storm, Kryo (2.21) is used for serialization: > https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L231 > The user must use the same version storm does, or there will be a java class > error at runtime. > Storm depends on a quasi-abandoned library: carbonite: > https://github.com/apache/storm/blob/02a44c7fc1b7b3a1571b326fde7bcae13e1b5c8d/pom.xml#L210 > which depends on Kryo 2.21 and Twitter chill 0.3.6: > https://github.com/sritchie/carbonite/blob/master/project.clj#L8 > Chill, currently on 0.7.3, would like to upgrade to Kryo 3.0.3: > https://github.com/twitter/chill/pull/245 > because Spark, also depending on chill, would like to upgrade for performance > improvements and bugfixes. > https://issues.apache.org/jira/browse/SPARK-11416 > Unfortunately, summingbird depends on storm: > https://github.com/twitter/summingbird/blob/develop/build.sbt#L34 > so, if chill is upgraded, and that gets on the classpath, summingbird will > break at runtime. > I propose: > 1) copy the carbonite code into storm. It is likely the only consumer. > 2) bump the storm kryo dependency after chill upgrades: recall that storm > actually depends on chill-java. A dependency that could possibly be removed > after you pull carbonite in. > 3) once a new version of storm is published, summingbird (and scalding) can > upgrade to the latest chill. > Also, I hope for: > 4) we as a JVM community get better about classpath isolation and versioning. > Diamonds like this in one big classpath make large codebases very fragile. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200040#comment-15200040
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56548912
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a
[GitHub] storm pull request: STORM-1616: Add RAS API for Trident
Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-197539412 @arunmahadevan @satishd @ptgoetz, Would like your comments on the builder pattern. There are +1's but I believe there were some concerns. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199693#comment-15199693
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56521036
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaRecordTupleBuilder.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.List;
+
+public class KafkaRecordTupleBuilder implements
KafkaSpoutTupleBuilder {
+@Override
+public List buildTuple(final ConsumerRecord
consumerRecord, KafkaSpoutStreams kafkaSpoutStreams) {
--- End diff --
@abhishekagarwal87 So what is the use case you are trying to solve, that
this does not work for?
The current code has a many to one relationship from topics to output
stream. But the KafkaSpoutTupleBuilder is a singleton. If we keep this the
same moving the output fields to the KafkaSpoutTupleBuilder would result in all
output streams being required to have the same set of fields.
To me it feels like the issue is that the KafkSpoutTupleBuilder is a
singleton. If it had a one to one relationship with the output stream instead,
the fields would fit into it better.
> As a storm developer I’d like to use the new kafka consumer API (0.8.3) to
> reduce dependencies and use long term supported kafka apis
> --
>
> Key: STORM-822
> URL: https://issues.apache.org/jira/browse/STORM-822
> Project: Apache Storm
> Issue Type: Story
> Components: storm-kafka
>Reporter: Thomas Becker
>Assignee: Hugo Louro
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56387329
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy;
+import org.apache.storm.kafka.spout.KafkaSpoutConfig.PollStrategy;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseRichSpout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.EARLIEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_LATEST;
+
+public class KafkaSpout extends BaseRichSpout {
+private static final Logger LOG =
LoggerFactory.getLogger(KafkaSpout.class);
+private static final Comparator OFFSET_COMPARATOR
= new OffsetComparator();
+
+// Storm
+protected SpoutOutputCollector collector;
+
+// Kafka
+private final KafkaSpoutConfig kafkaSpoutConfig;
+private transient KafkaConsumer kafkaConsumer;
+private transient boolean consumerAutoCommitMode;
+
+
+// Bookkeeping
+private KafkaSpoutStreams kafkaSpoutStreams;
+private KafkaSpoutTupleBuilder tupleBuilder;
+private transient Timer commitTimer;
// timer == null for auto commit mode
+private transient Timer logTimer;
+private transient Map acked; //
emitted tuples that were successfully acked. These tuples will be committed
periodically when the timer expires, on consumer rebalance, or on
close/deactivate
+private transient int maxRetries; //
Max number of times a tuple is retried
+private transient boolean initialized; // Flag indicating
that the spout is still undergoing initialization process.
+// Initialization is only complete after the first call to
KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+private transient long numUncommittedOffsets; // Number of offsets
that have been polled and emitted but not yet been committed
+private transient FirstPollOffsetStrategy firstPollOffsetStrategy;
+private transient PollStrategy pollStrategy;
+
+
+public KafkaSpout(KafkaSpoutConfig kafkaSpoutConfig,
KafkaSpoutTupleBuilder tupleBuilder) {
+this.kafkaSpoutConfig = kafkaSpoutConfig; // Pass
in configuration
+this.kafkaSpoutStreams = kafkaSpoutConfig.getKafkaSpoutStreams();
+this.tupleBuilder = tupleBuilder;
+}
+
+@Override
+public void open(Map conf, TopologyContext context,
SpoutOutputCollector collector) {
+initialized = false;
+
+// Spout internals
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user hmcl commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56462751
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
@revans2 @abhishekagar
[jira] [Created] (STORM-1634) Minor Refactoring Resource Aware Scheduler
Boyang Jerry Peng created STORM-1634: Summary: Minor Refactoring Resource Aware Scheduler Key: STORM-1634 URL: https://issues.apache.org/jira/browse/STORM-1634 Project: Apache Storm Issue Type: Improvement Reporter: Boyang Jerry Peng Priority: Minor Refactored the following: 1. The API interface to define custom scheduling strategies. The API exposed from RAS to a per topology scheduling strategy is messy and needs to be cleaned up. 2. In RAS, the state of the scheduler is held by several member variables. Its cleaner if the scheduler state is represented by a single object. This will help we reduce the amount of code need when checkpointing the state of the scheduler and potentially restoring the state in the future when a bad scheduling happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56390890
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
Yes that
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56409148
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
@revans2 makes sen
[jira] [Commented] (STORM-1616) Add RAS API for Trident
[ https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200333#comment-15200333 ] ASF GitHub Bot commented on STORM-1616: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-198074007 @ptgoetz The clojure tests diverged, but it's just import statements. If you want me to create a PR for that, I will, or if you trust me to resolve the imports, I'll just fix them and merge. > Add RAS API for Trident > --- > > Key: STORM-1616 > URL: https://issues.apache.org/jira/browse/STORM-1616 > Project: Apache Storm > Issue Type: Bug >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1637: Correct the master version of stor...
Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1231#issuecomment-198389354 BTW, I think @revans2 's main concern is **why JDK7 builds pass while JDK8 builds fail with** in JIRA [STORM-1637](https://issues.apache.org/jira/browse/STORM-1637).  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user roshannaik commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-197551650 @abhishekagarwal87 This perf hit of 7-9% exists even if user does not enable logging from the UI. With logging enabled, it should be even higher. @arunmahadevan The perf improvement i described here is observed when going from the default value nil to 0 for topology.eventlogger.executors **Topology detail:** 1 spout, 1 bolt, 1 acker. Spout generates random numbers. Bolt does this math on the tuple: (value+1)*2.In effect this is just a speed of light topology. The perf hit that i noted is actually due to those very same checking of flag. This checking of flags in clojure turns into very expensive lookups in clojure internals. It internally invokes Java reflection! If you are thinking 'what the hell'.. then yes that was my reaction too. Specifically this is the problematic lookup for this code path : 'storm-component->debug-atom' I agree with @arunmahadevan 's concern that this will confuse the users when they don't see logs after enabling it on the UI. The alternative fix for this is to change the manner in which this flag is made available to the code. Basically make it more efficient. There are some other lookups in the critical path that are also are causing perf hits... which i plan to address in a separate jira. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1537: Upgrade to kryo 3
GitHub user abhishekagarwal87 opened a pull request: https://github.com/apache/storm/pull/1223 STORM-1537: Upgrade to kryo 3 You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhishekagarwal87/storm kryo3-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1223.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1223 commit 514ba4e2a3aa8b13b46b9426d3619922428c93db Author: Abhishek Agarwal Date: 2016-03-08T20:13:23Z STORM-1537: Upgrade to kryo 3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1483: add storm-mongodb connector
Github user harshach commented on the pull request: https://github.com/apache/storm/pull/1219#issuecomment-197625582 Thanks @vesense merged into 1.x-branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1300) port backtype.storm.scheduler.resource-aware-scheduler-test to java
[ https://issues.apache.org/jira/browse/STORM-1300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202168#comment-15202168 ] ASF GitHub Bot commented on STORM-1300: --- GitHub user zhuoliu opened a pull request: https://github.com/apache/storm/pull/1232 [STORM-1300] backtype.storm.scheduler.resource-aware-scheduler-test Port backtype.storm.scheduler.resource-aware-scheduler-test to java You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhuoliu/storm 1300 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1232.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1232 commit c1b93de1650d113df0e1d0493780d9915bf3dacc Author: zhuol Date: 2016-03-18T21:06:00Z [STORM-1300] port backtype.storm.scheduler.resource-aware-scheduler-test to java. > port backtype.storm.scheduler.resource-aware-scheduler-test to java > > > Key: STORM-1300 > URL: https://issues.apache.org/jira/browse/STORM-1300 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Zhuo Liu > Labels: java-migration, jstorm-merger > > Test RAS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (STORM-1234) port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java
[ https://issues.apache.org/jira/browse/STORM-1234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Agarwal reassigned STORM-1234: --- Assignee: Abhishek Agarwal > port backtype.storm.security.auth.DefaultHttpCredentialsPlugin-test to java > > > Key: STORM-1234 > URL: https://issues.apache.org/jira/browse/STORM-1234 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > > to junit test conversion -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-584) LoggingMetricsConsumer metrics.log file is shared by multiple topologies.
[ https://issues.apache.org/jira/browse/STORM-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197210#comment-15197210 ] ASF GitHub Bot commented on STORM-584: -- GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1221 Fix logging for LoggingMetricsConsumer STORM-584 Currently the metrics.log file is always empty and it declared in the cluster.xml file whereas it should be in the worker one. The metrics end up in the log file which is not a huge issue but it would be cleaner to have them in a separate file, as originally intended. This patch addresses this and makes so that there is one metrics file per worker, as per the other log files, which should address the issues mentioned in [https://issues.apache.org/jira/browse/STORM-584]. You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm metricsFile Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1221.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1221 commit 4d75ec8494eeda24d6510641259d4c469cff3ee2 Author: Julien Nioche Date: 2016-03-16T11:14:24Z Fix logging for LoggingMetricsConsumer STORM-584 > LoggingMetricsConsumer metrics.log file is shared by multiple topologies. > - > > Key: STORM-584 > URL: https://issues.apache.org/jira/browse/STORM-584 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.6 >Reporter: Robert Joseph Evans > > The current cluster.xml file has special configuration for the > LoggingMetricsConsumer to place it in a metrics.log file. There are several > issues with this. > The worker, where the LoggingMetricsConsumer is run, is configured using > worker.xml and not cluster.xml. > The metrics.log file is shared by all workers running on the same node. So > if by chance we have LoggingMetricsConsumer instances running at the same > time, the resulting log file could have a number of issues, including data > corruption. > This becomes much worse under security where only the first > LoggingMetricsConsumer to run on the node will own the file, and all others > will fail to log metrics. > Also there is no way to access metrics.log through the ui now. > Ideally the metrics.log file will follow a pattern similar to the worker log > file, and have the topology name and the worker port a part of it. We also > would want to update the logviewer daemon to allow serving up these files and > deleting them after a specific period of time. The it would be awesome if > the UI could detect that a logging metrics consumer is installed, and provide > links to the metrics.log file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1611) port org.apache.storm.pacemaker.pacemaker to java
[ https://issues.apache.org/jira/browse/STORM-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197776#comment-15197776 ] ASF GitHub Bot commented on STORM-1611: --- Github user redsanket commented on a diff in the pull request: https://github.com/apache/storm/pull/1195#discussion_r56381761 --- Diff: storm-core/src/jvm/org/apache/storm/pacemaker/Pacemaker.java --- @@ -0,0 +1,246 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.storm.pacemaker; + +import org.apache.storm.generated.*; --- End diff -- single class imports > port org.apache.storm.pacemaker.pacemaker to java > - > > Key: STORM-1611 > URL: https://issues.apache.org/jira/browse/STORM-1611 > Project: Apache Storm > Issue Type: New Feature >Reporter: John Fang >Assignee: John Fang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1549) Add support for extending tuple tree timeout
[ https://issues.apache.org/jira/browse/STORM-1549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197404#comment-15197404 ] ASF GitHub Bot commented on STORM-1549: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1174 > Add support for extending tuple tree timeout > > > Key: STORM-1549 > URL: https://issues.apache.org/jira/browse/STORM-1549 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing >Priority: Minor > > During the discussion of https://github.com/apache/storm/pull/700 the issue > of allowing timeout extension in case of unavailable external components > (such as a web service) came up. > The current implementation makes tuples fail at a set interval, regardless of > whether or not replaying them is necessary. This can be irritating in > topologies that emit to multiple services, since one hanging service will > cause replays to hit all the working services as well. > I suggest adding a resetTimeout function to IOutputCollector, which will make > the relevant ackers and spouts reinsert the tuple tree information in their > pending maps. > The intended usage is that a bolt can call this function on an interval if it > needs to delay expiration, for example if it needs to retry calling a web > server a few times. It may also be useful for slow topologies that want Storm > to detect hanging/dropped tuples faster than the max expected complete > latency of the topology. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1602) Blobstore UTs are failing on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-1602.
-
Resolution: Fixed
Fix Version/s: 1.0.0
Merged into master and 1.x-branch
> Blobstore UTs are failing on Windows
>
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
> Fix For: 1.0.0
>
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.124 sec <<< ERROR!
> java.lang.RuntimeException: j
[GitHub] storm pull request: STORM-1629 (For 1.x) Files/move doesn't work p...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1215 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1602 Blobstore UTs are failed on Windows
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1229 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1602) Blobstore UTs are failing on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203051#comment-15203051
]
ASF GitHub Bot commented on STORM-1602:
---
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1229
> Blobstore UTs are failing on Windows
>
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.124 sec <<<
[jira] [Commented] (STORM-1602) Blobstore UTs are failing on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203050#comment-15203050
]
ASF GitHub Bot commented on STORM-1602:
---
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1227
> Blobstore UTs are failing on Windows
>
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.124 sec <<<
[GitHub] storm pull request: STORM-1602 Blobstore UTs are failed on Windows
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1227 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (STORM-1629) Files/move doesn't work properly with non-empty directory in Windows
[
https://issues.apache.org/jira/browse/STORM-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim closed STORM-1629.
---
Resolution: Fixed
Fix Version/s: 1.0.0
Merged into master and 1.x-branch.
> Files/move doesn't work properly with non-empty directory in Windows
>
>
> Key: STORM-1629
> URL: https://issues.apache.org/jira/browse/STORM-1629
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Blocker
> Fix For: 1.0.0
>
>
> Distributed version of download-storm-code uses Files#move().
> It runs well on *Nix (including OSX) but fails on Windows.
> Javadoc describes this behavior, please refer below link.
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...)
> {quote}
> When invoked to move a directory that is not empty then the directory is
> moved if it does not require moving the entries in the directory. For
> example, renaming a directory on the same FileStore will usually not require
> moving the entries in the directory. When moving a directory requires that
> its entries be moved then this method fails (by throwing an IOException). To
> move a file tree may involve copying rather than moving directories and this
> can be done using the copy method in conjunction with the Files.walkFileTree
> utility method.
> {quote}
> If directory is not empty, file system should treat "move directory" as
> "rename".
> Unfortunately, file system on Windows 8 doesn't.
> We should change the way to be compatible with both kinds of OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1629) Files/move doesn't work properly with non-empty directory in Windows
[
https://issues.apache.org/jira/browse/STORM-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203048#comment-15203048
]
ASF GitHub Bot commented on STORM-1629:
---
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1215
> Files/move doesn't work properly with non-empty directory in Windows
>
>
> Key: STORM-1629
> URL: https://issues.apache.org/jira/browse/STORM-1629
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Blocker
>
> Distributed version of download-storm-code uses Files#move().
> It runs well on *Nix (including OSX) but fails on Windows.
> Javadoc describes this behavior, please refer below link.
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...)
> {quote}
> When invoked to move a directory that is not empty then the directory is
> moved if it does not require moving the entries in the directory. For
> example, renaming a directory on the same FileStore will usually not require
> moving the entries in the directory. When moving a directory requires that
> its entries be moved then this method fails (by throwing an IOException). To
> move a file tree may involve copying rather than moving directories and this
> can be done using the copy method in conjunction with the Files.walkFileTree
> utility method.
> {quote}
> If directory is not empty, file system should treat "move directory" as
> "rename".
> Unfortunately, file system on Windows 8 doesn't.
> We should change the way to be compatible with both kinds of OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: [STORM-1573] Add batch support for MongoInsert...
GitHub user vesense opened a pull request: https://github.com/apache/storm/pull/1230 [STORM-1573] Add batch support for MongoInsertBolt https://issues.apache.org/jira/browse/STORM-1573 You can merge this pull request into a Git repository by running: $ git pull https://github.com/vesense/storm STORM-1573 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1230.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1230 commit 70f76a357a10d9061e0e605c6eb7ebcc6e310992 Author: vesense Date: 2016-03-17T12:51:51Z STORM-1573: Add batch support for MongoInsertBolt --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1057) Add throughput metric to spout/bolt and display them on web ui
[ https://issues.apache.org/jira/browse/STORM-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197446#comment-15197446 ] ASF GitHub Bot commented on STORM-1057: --- Github user d2r commented on the pull request: https://github.com/apache/storm/pull/753#issuecomment-197367675 > @revans2 @d2r @redsanket can you comment. thanks. I remember having the concern that the way the throughput stats are presented (e.g., extrapolating when the time window was too short) would result in inconsistency with the other stats presented. My concern is not serious enough concern to block this from being merged. We can handle issues that appear later as they come up. +0 > Add throughput metric to spout/bolt and display them on web ui > -- > > Key: STORM-1057 > URL: https://issues.apache.org/jira/browse/STORM-1057 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Li Wang >Assignee: Li Wang > Original Estimate: 168h > Remaining Estimate: 168h > > Throughput is a fundamental metric to reasoning about the performance > bottleneck of a topology. Displaying the throughputs of components and tasks > on the web ui could greatly facilitate the user identifying the performance > bottleneck and checking whether the the workload among components and tasks > are balanced. > What to do: > 1. Measure the throughput of each spout/bolt. > 2. Display the throughput metrics on web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1629) Files/move doesn't work properly with non-empty directory in Windows
[
https://issues.apache.org/jira/browse/STORM-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203046#comment-15203046
]
ASF GitHub Bot commented on STORM-1629:
---
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1214
> Files/move doesn't work properly with non-empty directory in Windows
>
>
> Key: STORM-1629
> URL: https://issues.apache.org/jira/browse/STORM-1629
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Blocker
>
> Distributed version of download-storm-code uses Files#move().
> It runs well on *Nix (including OSX) but fails on Windows.
> Javadoc describes this behavior, please refer below link.
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...)
> {quote}
> When invoked to move a directory that is not empty then the directory is
> moved if it does not require moving the entries in the directory. For
> example, renaming a directory on the same FileStore will usually not require
> moving the entries in the directory. When moving a directory requires that
> its entries be moved then this method fails (by throwing an IOException). To
> move a file tree may involve copying rather than moving directories and this
> can be done using the copy method in conjunction with the Files.walkFileTree
> utility method.
> {quote}
> If directory is not empty, file system should treat "move directory" as
> "rename".
> Unfortunately, file system on Windows 8 doesn't.
> We should change the way to be compatible with both kinds of OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-1629 Files/move doesn't work properly wi...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1214 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Fix logging for LoggingMetricsConsumer on bran...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1222#issuecomment-198042109 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1637: Correct the master version of stor...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1231#issuecomment-197915314 This build error is definitely unrelated (intermittent failure in storm-kafka) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198035#comment-15198035
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56403733
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a
[jira] [Commented] (STORM-584) LoggingMetricsConsumer metrics.log file is shared by multiple topologies.
[ https://issues.apache.org/jira/browse/STORM-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197215#comment-15197215 ] ASF GitHub Bot commented on STORM-584: -- GitHub user jnioche opened a pull request: https://github.com/apache/storm/pull/1222 Fix logging for LoggingMetricsConsumer on branch 0.10.x STORM-584 Port of #1221 to branch 0.10 You can merge this pull request into a Git repository by running: $ git pull https://github.com/DigitalPebble/storm 0.10.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1222.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1222 commit db5fbbde84e456de4594edf0a28e8982b5d00f12 Author: Julien Nioche Date: 2016-03-16T11:22:52Z Fix logging for LoggingMetricsConsumer on branch 0.10.x STORM-584 > LoggingMetricsConsumer metrics.log file is shared by multiple topologies. > - > > Key: STORM-584 > URL: https://issues.apache.org/jira/browse/STORM-584 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.6 >Reporter: Robert Joseph Evans > > The current cluster.xml file has special configuration for the > LoggingMetricsConsumer to place it in a metrics.log file. There are several > issues with this. > The worker, where the LoggingMetricsConsumer is run, is configured using > worker.xml and not cluster.xml. > The metrics.log file is shared by all workers running on the same node. So > if by chance we have LoggingMetricsConsumer instances running at the same > time, the resulting log file could have a number of issues, including data > corruption. > This becomes much worse under security where only the first > LoggingMetricsConsumer to run on the node will own the file, and all others > will fail to log metrics. > Also there is no way to access metrics.log through the ui now. > Ideally the metrics.log file will follow a pattern similar to the worker log > file, and have the topology name and the worker port a part of it. We also > would want to update the logviewer daemon to allow serving up these files and > deleting them after a specific period of time. The it would be awesome if > the UI could detect that a logging metrics consumer is installed, and provide > links to the metrics.log file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56396492
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
What do you mean by
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
Github user vesense commented on the pull request: https://github.com/apache/storm/pull/1228#issuecomment-197857053 +1 Good catch! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1602 Blobstore UTs are failed on Windows
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1229#issuecomment-197760319 #1227 is based on 1.x-branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56408115
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
@revans2
[GitHub] storm pull request: STORM-1637: Correct the master version of stor...
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1231#issuecomment-198401568 Yes. It is weird. I am taking a look at it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[ https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199960#comment-15199960 ] ASF GitHub Bot commented on STORM-822: -- Github user hmcl commented on the pull request: https://github.com/apache/storm/pull/1131#issuecomment-197979792 @revans2 @abhishekagarwal87 I am still going over all your comments and I will think through them thoroughly and I am welcome to incorporate everything that makes this better. However, one thing I would like to suggest is that we try to gather requirements and identify the best possible solution for all of those requirements, rather than simply trying to fit some of the existing KafkaSpout's solutions into this KafkaSpout. There are a lot of design decisions I don't agree with in the previous KafkaSpout that among other things made the code very complex and verbose. We should use the previous code to identify requirements, but we shouldn't simply follow it. > As a storm developer I’d like to use the new kafka consumer API (0.8.3) to > reduce dependencies and use long term supported kafka apis > -- > > Key: STORM-822 > URL: https://issues.apache.org/jira/browse/STORM-822 > Project: Apache Storm > Issue Type: Story > Components: storm-kafka >Reporter: Thomas Becker >Assignee: Hugo Louro > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1602 Blobstore UTs are failed on Windows
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1227#issuecomment-198153364 @revans2 Very odd indeed... It should give us error because of storm-mongo module. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1602) Blobstore UTs are failing on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200772#comment-15200772
]
ASF GitHub Bot commented on STORM-1602:
---
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/1227#issuecomment-198153364
@revans2 Very odd indeed... It should give us error because of storm-mongo
module.
> Blobstore UTs are failing on Windows
>
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cl
[GitHub] storm pull request: [STORM-1611] port org.apache.storm.pacemaker.p...
Github user knusbaum commented on a diff in the pull request:
https://github.com/apache/storm/pull/1195#discussion_r56557159
--- Diff: storm-core/src/clj/org/apache/storm/pacemaker/pacemaker.clj ---
@@ -1,242 +0,0 @@
-;; Licensed to the Apache Software Foundation (ASF) under one
-;; or more contributor license agreements. See the NOTICE file
-;; distributed with this work for additional information
-;; regarding copyright ownership. The ASF licenses this file
-;; to you under the Apache License, Version 2.0 (the
-;; "License"); you may not use this file except in compliance
-;; with the License. You may obtain a copy of the License at
-;;
-;; http://www.apache.org/licenses/LICENSE-2.0
-;;
-;; Unless required by applicable law or agreed to in writing, software
-;; distributed under the License is distributed on an "AS IS" BASIS,
-;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-;; See the License for the specific language governing permissions and
-;; limitations under the License.
-
-(ns org.apache.storm.pacemaker.pacemaker
- (:import [org.apache.storm.pacemaker PacemakerServer
IServerMessageHandler]
- [java.util.concurrent ConcurrentHashMap]
- [java.util.concurrent.atomic AtomicInteger]
- [org.apache.storm.generated HBNodes
-HBServerMessageType HBMessage HBMessageData HBPulse]
- [org.apache.storm.utils VersionInfo ConfigUtils]
- [uk.org.lidalia.sysoutslf4j.context SysOutOverSLF4J])
- (:use [clojure.string :only [replace-first split]]
-[org.apache.storm log config util])
- (:require [clojure.java.jmx :as jmx])
- (:gen-class))
-
-(def STORM-VERSION (VersionInfo/getVersion))
-
-;; Stats Functions
-
-(def sleep-seconds 60)
-
-
-(defn- check-and-set-loop [stats key new & {:keys [compare new-fn]
-:or {compare (fn [new old]
true)
- new-fn (fn [new old]
new)}}]
- (loop []
-(let [old (.get (key stats))
- new (new-fn new old)]
- (if (compare new old)
-(if (.compareAndSet (key stats) old new)
- nil
- (recur))
-nil
-
-(defn- set-average [stats size]
- (check-and-set-loop
- stats
- :average-heartbeat-size
- size
- :new-fn (fn [new old]
-(let [count (.get (:send-pulse-count stats))]
-; Weighted average
- (/ (+ new (* count old)) (+ count 1))
-
-(defn- set-largest [stats size]
- (check-and-set-loop
- stats
- :largest-heartbeat-size
- size
- :compare #'>))
-
-(defn- report-stats [heartbeats stats last-five-s]
- (loop []
-(let [send-count (.getAndSet (:send-pulse-count stats) 0)
- received-size (.getAndSet (:total-received-size stats) 0)
- get-count (.getAndSet (:get-pulse-count stats) 0)
- sent-size (.getAndSet (:total-sent-size stats) 0)
- largest (.getAndSet (:largest-heartbeat-size stats) 0)
- average (.getAndSet (:average-heartbeat-size stats) 0)
- total-keys (.size heartbeats)]
- (log-debug "\nReceived " send-count " heartbeats totaling "
received-size " bytes,\n"
- "Sent " get-count " heartbeats totaling " sent-size "
bytes,\n"
- "The largest heartbeat was " largest " bytes,\n"
- "The average heartbeat was " average " bytes,\n"
- "Pacemaker contained " total-keys " total keys\n"
- "in the last " sleep-seconds " second(s)")
- (dosync (ref-set last-five-s
- {:send-pulse-count send-count
-:total-received-size received-size
-:get-pulse-count get-count
-:total-sent-size sent-size
-:largest-heartbeat-size largest
-:average-heartbeat-size average
-:total-keys total-keys})))
-(Thread/sleep (* 1000 sleep-seconds))
-(recur)))
-
-;; JMX stuff
-(defn register [last-five-s]
- (jmx/register-mbean
-(jmx/create-bean
- last-five-s)
-"org.apache.storm.pacemaker.pacemaker:stats=Stats_Last_5_Seconds"))
--- End diff --
`(:require [clojure.java.jmx :as jmx])`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA tick
[jira] [Commented] (STORM-1636) Supervisor shutdown with worker id pass in being nil
[ https://issues.apache.org/jira/browse/STORM-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198730#comment-15198730 ] ASF GitHub Bot commented on STORM-1636: --- GitHub user jerrypeng opened a pull request: https://github.com/apache/storm/pull/1226 [STORM-1636] - Supervisor shutdown with worker id pass in being nil In function kill-existing-workers-with-change-in-components in supervisor.clj: The function tries to detect whether there is a change in assignment. The bug in this function is that the ordering of the assignment matters but it shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerrypeng/storm STORM-1636 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1226.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1226 commit 580ed9d2c4b012701fb6c1a307afe86eaf32863e Author: Boyang Jerry Peng Date: 2016-03-17T04:45:42Z [STORM-1636] - Supervisor shutdown with worker id pass in being nil > Supervisor shutdown with worker id pass in being nil > -- > > Key: STORM-1636 > URL: https://issues.apache.org/jira/browse/STORM-1636 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In function kill-existing-workers-with-change-in-components in supervisor.clj: > The function tries to detect whether there is a change in assignment. The bug > in this function is that the ordering of the assignment matters but it > shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it > changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1279) port backtype.storm.daemon.supervisor to java
[ https://issues.apache.org/jira/browse/STORM-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198734#comment-15198734 ] ASF GitHub Bot commented on STORM-1279: --- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/1184#issuecomment-197701084 Found another bug in supervisor: https://github.com/apache/storm/pull/1226 just another note here to make sure we remember to get in the fix whether in clojure or java > port backtype.storm.daemon.supervisor to java > - > > Key: STORM-1279 > URL: https://issues.apache.org/jira/browse/STORM-1279 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: John Fang > Labels: java-migration, jstorm-merger > Attachments: Discussion about supervisor.pdf > > > https://github.com/apache/storm/tree/jstorm-import/jstorm-core/src/main/java/com/alibaba/jstorm/daemon/supervisor > as an example > backtype.storm.event usage should be replaced with built-in java threadpools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (STORM-1637) MongoDB pom.xml fails on JDK8 Travis
Robert Joseph Evans created STORM-1637:
--
Summary: MongoDB pom.xml fails on JDK8 Travis
Key: STORM-1637
URL: https://issues.apache.org/jira/browse/STORM-1637
Project: Apache Storm
Issue Type: Bug
Components: storm-mongodb
Reporter: Robert Joseph Evans
Priority: Critical
https://travis-ci.org/apache/storm/builds/116573238
JDK7 builds pass JDK8 builds fail with.
{code}
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.storm:storm-mongodb:[unknown-version]
(/home/travis/build/apache/storm/external/storm-mongodb/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM: Could not transfer artifact
org.apache.storm:storm:pom:2.0.0-SNAPSHOT from/to codehaus-snapshots
(https://nexus.codehaus.org/snapshots/): nexus.codehaus.org and
'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown
host nexus.codehaus.org -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2]
http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56469003
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaRecordTupleBuilder.java
---
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.List;
+
+public class KafkaRecordTupleBuilder implements
KafkaSpoutTupleBuilder {
+@Override
+public List buildTuple(final ConsumerRecord
consumerRecord, KafkaSpoutStreams kafkaSpoutStreams) {
--- End diff --
There are three entities being constructed
1. Tuple - This is a list of objects being emitted by the spout e.g.
["electronics", "abhishek", "router"]
2. OutputFields - this indicates the position of fields by their name e.g.
[CATEGORY_NAME, USER_NAME, ITEM_NAME]
The bolt uses the field name in OutputFields to access the value in tuple.
e.g. one might access category using `String category =
tuple.getStringByField(CATEGORY_NAME)`
Person who is constructing the tuple, knows which positions of different
fields in tuple. That's why I suggested to have tuple building and output
fields in single interface because these two are tied and need to be consistent
with each other.
3. Schema - *a subset of the output fields* - I am not sure what it is. can
you elaborate more on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1228#issuecomment-198152909 +1 good --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-584) LoggingMetricsConsumer metrics.log file is shared by multiple topologies.
[ https://issues.apache.org/jira/browse/STORM-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198508#comment-15198508 ] ASF GitHub Bot commented on STORM-584: -- Github user harshach commented on the pull request: https://github.com/apache/storm/pull/1221#issuecomment-197625614 +1 > LoggingMetricsConsumer metrics.log file is shared by multiple topologies. > - > > Key: STORM-584 > URL: https://issues.apache.org/jira/browse/STORM-584 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.6 >Reporter: Robert Joseph Evans > > The current cluster.xml file has special configuration for the > LoggingMetricsConsumer to place it in a metrics.log file. There are several > issues with this. > The worker, where the LoggingMetricsConsumer is run, is configured using > worker.xml and not cluster.xml. > The metrics.log file is shared by all workers running on the same node. So > if by chance we have LoggingMetricsConsumer instances running at the same > time, the resulting log file could have a number of issues, including data > corruption. > This becomes much worse under security where only the first > LoggingMetricsConsumer to run on the node will own the file, and all others > will fail to log metrics. > Also there is no way to access metrics.log through the ui now. > Ideally the metrics.log file will follow a pattern similar to the worker log > file, and have the topology name and the worker port a part of it. We also > would want to update the logviewer daemon to allow serving up these files and > deleting them after a specific period of time. The it would be awesome if > the UI could detect that a logging metrics consumer is installed, and provide > links to the metrics.log file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1611] port org.apache.storm.pacemaker.p...
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/storm/pull/1195#discussion_r56659070
--- Diff: storm-core/src/jvm/org/apache/storm/pacemaker/Pacemaker.java ---
@@ -0,0 +1,372 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.pacemaker;
+
+import org.apache.storm.generated.HBMessage;
+import org.apache.storm.generated.HBMessageData;
+import org.apache.storm.generated.HBPulse;
+import org.apache.storm.generated.HBNodes;
+import org.apache.storm.generated.HBServerMessageType;
+import org.apache.storm.utils.ConfigUtils;
+import org.apache.storm.utils.Utils;
+import org.apache.storm.utils.VersionInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import uk.org.lidalia.sysoutslf4j.context.SysOutOverSLF4J;
+
+
+import javax.management.*;
+import java.lang.management.ManagementFactory;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class Pacemaker implements IServerMessageHandler {
+
+private static final Logger LOG =
LoggerFactory.getLogger(Pacemaker.class);
+
+private Map heartbeats;
+private PacemakerStats lastOneMinStats;
+private PacemakerStats pacemakerStats;
+private Map conf;
+private final long sleepSeconds = 60;
+
+private boolean isDaemon = true;
+private boolean startImmediately = true;
+
+private static class PacemakerStats {
+public AtomicInteger sendPulseCount = new AtomicInteger();
+public AtomicInteger totalReceivedSize = new AtomicInteger();
+public AtomicInteger getPulseCount = new AtomicInteger();
+public AtomicInteger totalSentSize = new AtomicInteger();
+public AtomicInteger largestHeartbeatSize = new AtomicInteger();
+public AtomicInteger averageHeartbeatSize = new AtomicInteger();
+private AtomicInteger totalKeys = new AtomicInteger();
+}
+private static class PaceMakerDynamicMBean implements DynamicMBean{
+
+private final MBeanInfo mBeanInfo;
+private final static String [] attributeNames = new String []{
+"send-pulse-count",
+"total-received-size",
+"get-pulse-count",
+"total-sent-size",
+"largest-heartbeat-size",
+"average-heartbeat-size",
+"total-keys"
+};
+private static String attributeType =
"java.util.concurrent.atomic.AtomicInteger";
+
+private static final MBeanAttributeInfo[] attributeInfos = new
MBeanAttributeInfo[] {
+new MBeanAttributeInfo("send-pulse-count",
attributeType, "send-pulse-count", true, false, false),
+new MBeanAttributeInfo("total-received-size",
attributeType, "total-received-size", true, false, false),
+new MBeanAttributeInfo("get-pulse-count",
attributeType, "get-pulse-count", true, false, false),
+new MBeanAttributeInfo("total-sent-size",
attributeType, "total-sent-size", true, false, false),
+new MBeanAttributeInfo("largest-heartbeat-size",
attributeType, "largest-heartbeat-size", true, false, false),
+new MBeanAttributeInfo("average-heartbeat-size",
attributeType, "average-heartbeat-size", true, false, false),
+new MBeanAttributeInfo("total-keys",
attributeType, "total-keys", true, false, false)
+};
+private PacemakerStats stats;
+
+public PaceMakerDynamicMBean(PacemakerStats stats) {
--- End diff --
minor nit "Pacemaker" instead of "PaceMaker"
---
If your project is set up for it, you can r
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197978#comment-15197978
]
ASF GitHub Bot commented on STORM-822:
--
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56399639
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH On
[jira] [Commented] (STORM-1629) Files/move doesn't work properly with non-empty directory in Windows
[
https://issues.apache.org/jira/browse/STORM-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198713#comment-15198713
]
ASF GitHub Bot commented on STORM-1629:
---
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/1215#issuecomment-197690330
Please review #1214 as well. Thanks!
> Files/move doesn't work properly with non-empty directory in Windows
>
>
> Key: STORM-1629
> URL: https://issues.apache.org/jira/browse/STORM-1629
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Blocker
>
> Distributed version of download-storm-code uses Files#move().
> It runs well on *Nix (including OSX) but fails on Windows.
> Javadoc describes this behavior, please refer below link.
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...)
> {quote}
> When invoked to move a directory that is not empty then the directory is
> moved if it does not require moving the entries in the directory. For
> example, renaming a directory on the same FileStore will usually not require
> moving the entries in the directory. When moving a directory requires that
> its entries be moved then this method fails (by throwing an IOException). To
> move a file tree may involve copying rather than moving directories and this
> can be done using the copy method in conjunction with the Files.walkFileTree
> utility method.
> {quote}
> If directory is not empty, file system should treat "move directory" as
> "rename".
> Unfortunately, file system on Windows 8 doesn't.
> We should change the way to be compatible with both kinds of OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1624) add maven central status in README
[ https://issues.apache.org/jira/browse/STORM-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198514#comment-15198514 ] ASF GitHub Bot commented on STORM-1624: --- Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1212 > add maven central status in README > -- > > Key: STORM-1624 > URL: https://issues.apache.org/jira/browse/STORM-1624 > Project: Apache Storm > Issue Type: Improvement >Reporter: Xin Wang >Assignee: Xin Wang > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1616) Add RAS API for Trident
[ https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200278#comment-15200278 ] ASF GitHub Bot commented on STORM-1616: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-198063862 @knusbaum Technically this should have waited another ~24 hrs. after the last code change, but I'm not worried because the last change was minor. Are you planning to pull this into 1.x-branch as well? > Add RAS API for Trident > --- > > Key: STORM-1616 > URL: https://issues.apache.org/jira/browse/STORM-1616 > Project: Apache Storm > Issue Type: Bug >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1616: Add RAS API for Trident
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1199 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: Hbasemapstate
GitHub user lujinhong opened a pull request: https://github.com/apache/storm/pull/1225 Hbasemapstate Add HBaseMapState demo You can merge this pull request into a Git repository by running: $ git pull https://github.com/lujinhong/storm hbasemapstate Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1225.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1225 commit 1a2e2e34a29185e3ae87dfd7caf82c034b3fddeb Author: jinhong-lu Date: 2016-03-17T02:57:28Z git add HBaseMapState Demo commit a29d0116ceda1d3fd2f26b6916e65f7a5472d89a Author: jinhong-lu Date: 2016-03-17T02:58:17Z git add HBaseMapState Demo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1623] fix bug about nimbus.clj
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1211 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1602 Blobstore UTs are failed on Windows
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1227 STORM-1602 Blobstore UTs are failed on Windows * ensures objects of InputStream / OutputStream are closed after used * clojure: with-open * java: try-with-resource * skip checking symbolic link in LocalizerTest when on Windows * Windows seems not handle symbolic link in compressed file properly Some integration tests are still failing on Windows, but it should be handled with another issues. I'll also address this based on master immediately. Please have a look. Thanks! You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm STORM-1602-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1227.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1227 commit 8965a6c2459841c607fb9587897502581f1e3cdf Author: Jungtaek Lim Date: 2016-03-17T06:53:15Z STORM-1602 Blobstore UTs are failed on Windows * ensures objects of InputStream / OutputStream are closed after using * clojure: with-open * java: try-with-resource * skip checking symbolic link in LocalizerTest when on Windows * Windows seems not handle symbolic link in compressed file properly --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
Github user dossett commented on the pull request: https://github.com/apache/storm/pull/1228#issuecomment-197833762 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (STORM-1602) Blobstore UTs are failing on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated STORM-1602:
Summary: Blobstore UTs are failing on Windows (was: Blobstore UTs are
failed on Windows)
> Blobstore UTs are failing on Windows
>
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testBasicLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.124 sec <<< ERROR!
> java.lang.RuntimeException: java.nio.file.DirectoryN
[GitHub] storm pull request: STORM-1616: Add RAS API for Trident
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-198063862 @knusbaum Technically this should have waited another ~24 hrs. after the last code change, but I'm not worried because the last change was minor. Are you planning to pull this into 1.x-branch as well? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1228 hotfix: parent version for pom.xml in storm-mongo is not properly set You can merge this pull request into a Git repository by running: $ git pull https://github.com/HeartSaVioR/storm fix-mongodb-parent-version-1.x-branch Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1228.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1228 commit 0c90fec1475c67dfb487d9e8b884855ee7ed1574 Author: Jungtaek Lim Date: 2016-03-17T07:29:24Z hotfix: parent version for pom.xml in storm-mongo is not properly set --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1636] - Supervisor shutdown with worker...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1226#issuecomment-197706290 nice catch. I have considered this when I port supervisor.clj to java. Than you @jerrypeng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1549: Add support for resetting tuple ti...
Github user revans2 commented on the pull request: https://github.com/apache/storm/pull/1107#issuecomment-197358536 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: [STORM-1611] port org.apache.storm.pacemaker.p...
Github user hustfxj commented on the pull request: https://github.com/apache/storm/pull/1195#issuecomment-197645359 @redsanket @abhishekagarwal87 I have addressed your comments, thank you. @redsanket I also am curious where is the jmx related code implemented. But I don't found, so I removed pacemaker.register () about jmx. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198097#comment-15198097
]
ASF GitHub Bot commented on STORM-822:
--
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56409148
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls
[jira] [Commented] (STORM-1602) Blobstore UTs are failed on Windows
[
https://issues.apache.org/jira/browse/STORM-1602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199155#comment-15199155
]
ASF GitHub Bot commented on STORM-1602:
---
Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/1229#issuecomment-197760319
#1227 is based on 1.x-branch.
> Blobstore UTs are failed on Windows
> ---
>
> Key: STORM-1602
> URL: https://issues.apache.org/jira/browse/STORM-1602
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Critical
>
> Blobstore related UTs are failed on Windows.
> {code}
> ---
> Test set: org.apache.storm.blobstore.BlobStoreTest
> ---
> Tests run: 7, Failures: 0, Errors: 7, Skipped: 0, Time elapsed: 2.306 sec <<<
> FAILURE! - in org.apache.storm.blobstore.BlobStoreTest
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.798 sec <<< ERROR!
> java.nio.file.AccessDeniedException:
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\1457050287771.tmp
> ->
> D:\storm\storm-core\target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at
> sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
> at
> sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
> at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
> at
> sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
> at java.nio.file.Files.move(Files.java:1347)
> at
> org.apache.storm.blobstore.LocalFsBlobStoreFile.commit(LocalFsBlobStoreFile.java:127)
> at
> org.apache.storm.blobstore.BlobStore$BlobStoreFileOutputStream.close(BlobStore.java:324)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultiple(BlobStoreTest.java:397)
> at
> org.apache.storm.blobstore.BlobStoreTest.testMultipleLocalFs(BlobStoreTest.java:168)
> testMultipleLocalFs(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 1.8 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-19f8e973-7c1b-4638-8679-2eb1adcac396\blobs\571\data_other\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testGetFileLength(org.apache.storm.blobstore.BlobStoreTest) Time elapsed:
> 0.067 sec <<< ERROR!
> java.io.IOException: Unable to delete file:
> target\blob-store-test-b6d39145-11ea-4aa7-ae30-28bda603fb3a\blobs\1017\data_test\data
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2279)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2270)
> at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1653)
> at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1535)
> at
> org.apache.storm.blobstore.BlobStoreTest.cleanup(BlobStoreTest.java:74)
> testBasicLocalFs(org.apac
[jira] [Commented] (STORM-1616) Add RAS API for Trident
[ https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198742#comment-15198742 ] ASF GitHub Bot commented on STORM-1616: --- Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-197702133 @knusbaum I guess the main concern against builder is that we want to maintain compatibility between core and trident. If so I am fine to change both the apis together and/or explore some other way to pass the config (load, parallelism etc) in trident as a followup. In my opinion, separating the config and the streaming operations would make the trident stream api less cluttered. > Add RAS API for Trident > --- > > Key: STORM-1616 > URL: https://issues.apache.org/jira/browse/STORM-1616 > Project: Apache Storm > Issue Type: Bug >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1616) Add RAS API for Trident
[
https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198136#comment-15198136
]
ASF GitHub Bot commented on STORM-1616:
---
Github user d2r commented on a diff in the pull request:
https://github.com/apache/storm/pull/1199#discussion_r56413444
--- Diff: storm-core/src/jvm/org/apache/storm/trident/TridentTopology.java
---
@@ -431,6 +454,64 @@ public StormTopology build() {
return builder.buildTopology();
}
+
+private static Map mergeDefaultResources(Map res, Map defaultConfig) {
+Map ret = new HashMap();
+
+Number onHeapDefault =
(Number)defaultConfig.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB);
+Number offHeapDefault =
(Number)defaultConfig.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB);
+Number cpuLoadDefault =
(Number)defaultConfig.get(Config.TOPOLOGY_COMPONENT_CPU_PCORE_PERCENT);
+
+if(res == null) {
+ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB,
onHeapDefault);
+ret.put(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB,
offHeapDefault);
+ret.put(Config.TOPOLOGY_COMPONENT_CPU_PCORE_PERCENT,
cpuLoadDefault);
+return ret;
+}
+
+Number onHeap =
res.get(Config.TOPOLOGY_COMPONENT_RESOURCES_ONHEAP_MEMORY_MB);
+Number offHeap =
res.get(Config.TOPOLOGY_COMPONENT_RESOURCES_OFFHEAP_MEMORY_MB);
+Number cpuLoad =
res.get(Config.TOPOLOGY_COMPONENT_CPU_PCORE_PERCENT);
+
+/* We take the max of the default and whatever the user put in
here.
+ Each node's resources can be the sum of several operations, so
the simplest
+ thing to do is get the max.
+
+ The situation we want to avoid is that the user sets low
resources on one
+ node, and when that node is combined with a bunch of others,
the sum is still
+ that low resource count. If any component isn't set, we want to
use the default.
+
+ Right now, this code does not check that. It just takes the max
of the summed
+ up resource counts for simplicity's sake. We could perform some
more complicated
+ logic to be more accurate, but the benefits are very small, and
only apply to some
+ very odd corner cases. */g
--- End diff --
Comment is OK, but there is a dangling `g` after comment.
> Add RAS API for Trident
> ---
>
> Key: STORM-1616
> URL: https://issues.apache.org/jira/browse/STORM-1616
> Project: Apache Storm
> Issue Type: Bug
>Reporter: Kyle Nusbaum
>Assignee: Kyle Nusbaum
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-584) LoggingMetricsConsumer metrics.log file is shared by multiple topologies.
[ https://issues.apache.org/jira/browse/STORM-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200191#comment-15200191 ] ASF GitHub Bot commented on STORM-584: -- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1221#issuecomment-198042279 +1 > LoggingMetricsConsumer metrics.log file is shared by multiple topologies. > - > > Key: STORM-584 > URL: https://issues.apache.org/jira/browse/STORM-584 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Affects Versions: 0.9.6 >Reporter: Robert Joseph Evans > > The current cluster.xml file has special configuration for the > LoggingMetricsConsumer to place it in a metrics.log file. There are several > issues with this. > The worker, where the LoggingMetricsConsumer is run, is configured using > worker.xml and not cluster.xml. > The metrics.log file is shared by all workers running on the same node. So > if by chance we have LoggingMetricsConsumer instances running at the same > time, the resulting log file could have a number of issues, including data > corruption. > This becomes much worse under security where only the first > LoggingMetricsConsumer to run on the node will own the file, and all others > will fail to log metrics. > Also there is no way to access metrics.log through the ui now. > Ideally the metrics.log file will follow a pattern similar to the worker log > file, and have the topology name and the worker port a part of it. We also > would want to update the logviewer daemon to allow serving up these files and > deleting them after a specific period of time. The it would be awesome if > the UI could detect that a logging metrics consumer is installed, and provide > links to the metrics.log file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1632 Disable event logging by default
Github user abhishekagarwal87 commented on the pull request: https://github.com/apache/storm/pull/1217#issuecomment-197534417 I agree with @arunmahadevan. You can also show an alert message in UI, when user turns on the event logging for a topology e.g. "Event logging may degrade the performance" or something similar. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1616: Add RAS API for Trident
Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-197702133 @knusbaum I guess the main concern against builder is that we want to maintain compatibility between core and trident. If so I am fine to change both the apis together and/or explore some other way to pass the config (load, parallelism etc) in trident as a followup. In my opinion, separating the config and the streaming operations would make the trident stream api less cluttered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1634) Minor Refactoring of Resource Aware Scheduler
[ https://issues.apache.org/jira/browse/STORM-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198672#comment-15198672 ] ASF GitHub Bot commented on STORM-1634: --- Github user jerrypeng commented on the pull request: https://github.com/apache/storm/pull/1224#issuecomment-197671679 Unrelated travis-ci failure > Minor Refactoring of Resource Aware Scheduler > - > > Key: STORM-1634 > URL: https://issues.apache.org/jira/browse/STORM-1634 > Project: Apache Storm > Issue Type: Improvement >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng >Priority: Minor > > Refactored the following: > 1. The API interface to define custom scheduling strategies. The API exposed > from RAS to a per topology scheduling strategy is messy and needs to be > cleaned up. > 2. In RAS, the state of the scheduler is held by several member variables. > Its cleaner if the scheduler state is represented by a single object. This > will help we reduce the amount of code need when checkpointing the state of > the scheduler and potentially restoring the state in the future when a bad > scheduling happens -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1611] port org.apache.storm.pacemaker.p...
Github user redsanket commented on a diff in the pull request: https://github.com/apache/storm/pull/1195#discussion_r56658785 --- Diff: storm-core/src/jvm/org/apache/storm/pacemaker/Pacemaker.java --- @@ -0,0 +1,372 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.storm.pacemaker; + +import org.apache.storm.generated.HBMessage; +import org.apache.storm.generated.HBMessageData; +import org.apache.storm.generated.HBPulse; +import org.apache.storm.generated.HBNodes; +import org.apache.storm.generated.HBServerMessageType; +import org.apache.storm.utils.ConfigUtils; +import org.apache.storm.utils.Utils; +import org.apache.storm.utils.VersionInfo; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import uk.org.lidalia.sysoutslf4j.context.SysOutOverSLF4J; + + +import javax.management.*; --- End diff -- single imports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197976#comment-15197976
]
ASF GitHub Bot commented on STORM-822:
--
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56399307
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197947#comment-15197947
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56396492
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a
[jira] [Commented] (STORM-1464) storm-hdfs should support writing to multiple files
[ https://issues.apache.org/jira/browse/STORM-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201176#comment-15201176 ] ASF GitHub Bot commented on STORM-1464: --- Github user arunmahadevan commented on the pull request: https://github.com/apache/storm/pull/1044#issuecomment-198255363 @dossett I went through the patch again and I have one question regarding the writer key. I see that you maintain a separate writer per writer key. In the docs you mention "The avro bolt will write records to separate files based on the schema of the record being processed. In other words, if the bolt receives records with two different schemas, it will write to two separate files." Is each writer expected to write to a separate file ? If so I dont see that happening because in `getBasePathForNextFile` the file name is based on the partition path and rotation id alone and then this path is passed to `makeNewWriter`. So there could be multiple writers writing to the same file and infact at overlapping offsets based on that writer's offset and may be corrupt the file. Can you help me understand if each writer can write to the same file or is it always supposed to write to different files? > storm-hdfs should support writing to multiple files > --- > > Key: STORM-1464 > URL: https://issues.apache.org/jira/browse/STORM-1464 > Project: Apache Storm > Issue Type: Improvement > Components: storm-hdfs >Reporter: Aaron Dossett >Assignee: Aaron Dossett > Labels: avro > > Examples of when this is needed include: > - One avro bolt writing multiple schemas, each of which require a different > file. Schema evolution is a common use of avro and the avro bolt should > support that seamlessly. > - Partitioning output to different directories based on the tuple contents. > For example, if the tuple contains a "USER" field, it should be possible to > partition based on that value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56386686
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -0,0 +1,517 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.clients.consumer.ConsumerRecords;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.consumer.OffsetAndMetadata;
+import org.apache.kafka.common.TopicPartition;
+import
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy;
+import org.apache.storm.kafka.spout.KafkaSpoutConfig.PollStrategy;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.task.TopologyContext;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.base.BaseRichSpout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableSet;
+import java.util.TreeSet;
+import java.util.concurrent.TimeUnit;
+
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.EARLIEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST;
+import static
org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_LATEST;
+
+public class KafkaSpout extends BaseRichSpout {
+private static final Logger LOG =
LoggerFactory.getLogger(KafkaSpout.class);
+private static final Comparator OFFSET_COMPARATOR
= new OffsetComparator();
+
+// Storm
+protected SpoutOutputCollector collector;
+
+// Kafka
+private final KafkaSpoutConfig kafkaSpoutConfig;
+private transient KafkaConsumer kafkaConsumer;
+private transient boolean consumerAutoCommitMode;
+
+
+// Bookkeeping
+private KafkaSpoutStreams kafkaSpoutStreams;
+private KafkaSpoutTupleBuilder tupleBuilder;
+private transient Timer commitTimer;
// timer == null for auto commit mode
+private transient Timer logTimer;
+private transient Map acked; //
emitted tuples that were successfully acked. These tuples will be committed
periodically when the timer expires, on consumer rebalance, or on
close/deactivate
+private transient int maxRetries; //
Max number of times a tuple is retried
+private transient boolean initialized; // Flag indicating
that the spout is still undergoing initialization process.
+// Initialization is only complete after the first call to
KafkaSpoutConsumerRebalanceListener.onPartitionsAssigned()
+private transient long numUncommittedOffsets; // Number of offsets
that have been polled and emitted but not yet been committed
+private transient FirstPollOffsetStrategy firstPollOffsetStrategy;
+private transient PollStrategy pollStrategy;
+
+
+public KafkaSpout(KafkaSpoutConfig kafkaSpoutConfig,
KafkaSpoutTupleBuilder tupleBuilder) {
+this.kafkaSpoutConfig = kafkaSpoutConfig; // Pass
in configuration
+this.kafkaSpoutStreams = kafkaSpoutConfig.getKafkaSpoutStreams();
+this.tupleBuilder = tupleBuilder;
+}
+
+@Override
+public void open(Map conf, TopologyContext context,
SpoutOutputCollector collector) {
+initialized = false;
+
+// Spout in
[jira] [Commented] (STORM-1637) MongoDB pom.xml fails on JDK8 Travis
[
https://issues.apache.org/jira/browse/STORM-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201596#comment-15201596
]
ASF GitHub Bot commented on STORM-1637:
---
Github user abhishekagarwal87 commented on the pull request:
https://github.com/apache/storm/pull/1231#issuecomment-198401568
Yes. It is weird. I am taking a look at it.
> MongoDB pom.xml fails on JDK8 Travis
>
>
> Key: STORM-1637
> URL: https://issues.apache.org/jira/browse/STORM-1637
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-mongodb
>Reporter: Robert Joseph Evans
>Assignee: Abhishek Agarwal
>Priority: Critical
>
> https://travis-ci.org/apache/storm/builds/116573238
> JDK7 builds pass JDK8 builds fail with.
> {code}
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR] The project org.apache.storm:storm-mongodb:[unknown-version]
> (/home/travis/build/apache/storm/external/storm-mongodb/pom.xml) has 1 error
> [ERROR] Non-resolvable parent POM: Could not transfer artifact
> org.apache.storm:storm:pom:2.0.0-SNAPSHOT from/to codehaus-snapshots
> (https://nexus.codehaus.org/snapshots/): nexus.codehaus.org and
> 'parent.relativePath' points at wrong local POM @ line 21, column 13: Unknown
> host nexus.codehaus.org -> [Help 2]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> [ERROR] [Help 2]
> http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-1629) Files/move doesn't work properly with non-empty directory in Windows
[
https://issues.apache.org/jira/browse/STORM-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200201#comment-15200201
]
ASF GitHub Bot commented on STORM-1629:
---
Github user ptgoetz commented on the pull request:
https://github.com/apache/storm/pull/1215#issuecomment-198044924
+1
> Files/move doesn't work properly with non-empty directory in Windows
>
>
> Key: STORM-1629
> URL: https://issues.apache.org/jira/browse/STORM-1629
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
>Affects Versions: 1.0.0, 2.0.0
> Environment: Windows
>Reporter: Jungtaek Lim
>Assignee: Jungtaek Lim
>Priority: Blocker
>
> Distributed version of download-storm-code uses Files#move().
> It runs well on *Nix (including OSX) but fails on Windows.
> Javadoc describes this behavior, please refer below link.
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path,%20java.nio.file.Path,%20java.nio.file.CopyOption...)
> {quote}
> When invoked to move a directory that is not empty then the directory is
> moved if it does not require moving the entries in the directory. For
> example, renaming a directory on the same FileStore will usually not require
> moving the entries in the directory. When moving a directory requires that
> its entries be moved then this method fails (by throwing an IOException). To
> move a file tree may involve copying rather than moving directories and this
> can be done using the copy method in conjunction with the Files.walkFileTree
> utility method.
> {quote}
> If directory is not empty, file system should treat "move directory" as
> "rename".
> Unfortunately, file system on Windows 8 doesn't.
> We should change the way to be compatible with both kinds of OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198165#comment-15198165
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56414932
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
Github user HeartSaVioR commented on the pull request: https://github.com/apache/storm/pull/1228#issuecomment-198368906 @vesense Thanks for noticing. Closing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1229) port backtype.storm.metric.testing to java
[ https://issues.apache.org/jira/browse/STORM-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202935#comment-15202935 ] ASF GitHub Bot commented on STORM-1229: --- GitHub user abhishekagarwal87 opened a pull request: https://github.com/apache/storm/pull/1238 STORM-1229: port backtype.storm.metric.testing to java You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhishekagarwal87/storm metric-testing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1238 commit 07e1b231e37217705add955d76b57fa530b99955 Author: Abhishek Agarwal Date: 2016-03-19T19:47:33Z STORM-1229: port backtype.storm.metric.testing to java > port backtype.storm.metric.testing to java > -- > > Key: STORM-1229 > URL: https://issues.apache.org/jira/browse/STORM-1229 > Project: Apache Storm > Issue Type: New Feature > Components: storm-core >Reporter: Robert Joseph Evans >Assignee: Abhishek Agarwal > Labels: java-migration, jstorm-merger > > This provides a way to capture and inspect the datapoints sent to a > FakeMetricConsumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1229: port backtype.storm.metric.testing...
GitHub user abhishekagarwal87 opened a pull request: https://github.com/apache/storm/pull/1238 STORM-1229: port backtype.storm.metric.testing to java You can merge this pull request into a Git repository by running: $ git pull https://github.com/abhishekagarwal87/storm metric-testing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1238.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1238 commit 07e1b231e37217705add955d76b57fa530b99955 Author: Abhishek Agarwal Date: 2016-03-19T19:47:33Z STORM-1229: port backtype.storm.metric.testing to java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: hotfix: parent version for pom.xml in storm-mo...
Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1228#issuecomment-198039817 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1614) Clean backpressure zk node in do-cleanup
[ https://issues.apache.org/jira/browse/STORM-1614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200224#comment-15200224 ] ASF GitHub Bot commented on STORM-1614: --- Github user ptgoetz commented on the pull request: https://github.com/apache/storm/pull/1206#issuecomment-198048775 Merged to 1.x-branch. > Clean backpressure zk node in do-cleanup > > > Key: STORM-1614 > URL: https://issues.apache.org/jira/browse/STORM-1614 > Project: Apache Storm > Issue Type: Bug > Components: storm-core >Reporter: Alessandro Bellina >Assignee: Alessandro Bellina >Priority: Minor > > Currently the backpressure node is being removed in killTopologyWithOpts in > nimbus. Remove instead like the other ZK nodes in do-cleanup for inactive > topos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (STORM-1636) Supervisor shutdown with worker id pass in being nil
[ https://issues.apache.org/jira/browse/STORM-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200093#comment-15200093 ] ASF GitHub Bot commented on STORM-1636: --- Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/1226#issuecomment-198013271 +1 > Supervisor shutdown with worker id pass in being nil > -- > > Key: STORM-1636 > URL: https://issues.apache.org/jira/browse/STORM-1636 > Project: Apache Storm > Issue Type: Bug >Affects Versions: 1.0.0, 2.0.0 >Reporter: Boyang Jerry Peng >Assignee: Boyang Jerry Peng > > In function kill-existing-workers-with-change-in-components in supervisor.clj: > The function tries to detect whether there is a change in assignment. The bug > in this function is that the ordering of the assignment matters but it > shouldn't. For example, if a worker assignment is [[1 1] [2 2]] and it > changed to [[2 2] [1 1]] it will cause the supervisor to restart the worker -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (STORM-1615) Update state checkpointing doc with bolt's acking contract
[ https://issues.apache.org/jira/browse/STORM-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] P. Taylor Goetz resolved STORM-1615. Resolution: Fixed Merged to asf-site. > Update state checkpointing doc with bolt's acking contract > -- > > Key: STORM-1615 > URL: https://issues.apache.org/jira/browse/STORM-1615 > Project: Apache Storm > Issue Type: Bug >Reporter: Arun Mahadevan >Assignee: Arun Mahadevan > Fix For: 1.0.0, 2.0.0 > > > Update > https://github.com/apache/storm/blob/asf-site/documentation/State-checkpointing.md -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-822: Kafka Spout New Consumer API
Github user abhishekagarwal87 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56399639
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a new batch of records from kafka once all
the records that came in the previous poll have been acked.
+ *
+ */
+public enum PollStrategy {
+STREAM,
+BATCH
--- End diff --
Throughpu
[jira] [Commented] (STORM-1616) Add RAS API for Trident
[ https://issues.apache.org/jira/browse/STORM-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199939#comment-15199939 ] ASF GitHub Bot commented on STORM-1616: --- Github user knusbaum commented on the pull request: https://github.com/apache/storm/pull/1199#issuecomment-197977093 @arunmahadevan Thanks, that sounds good to me. > Add RAS API for Trident > --- > > Key: STORM-1616 > URL: https://issues.apache.org/jira/browse/STORM-1616 > Project: Apache Storm > Issue Type: Bug >Reporter: Kyle Nusbaum >Assignee: Kyle Nusbaum > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: [STORM-1611] port org.apache.storm.pacemaker.p...
Github user redsanket commented on the pull request: https://github.com/apache/storm/pull/1195#issuecomment-197508158 Most of the changes look good, just a question about jmx reporter implementation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-822) As a storm developer I’d like to use the new kafka consumer API (0.8.3) to reduce dependencies and use long term supported kafka apis
[
https://issues.apache.org/jira/browse/STORM-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15199563#comment-15199563
]
ASF GitHub Bot commented on STORM-822:
--
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1131#discussion_r56507684
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.kafka.spout;
+
+import org.apache.kafka.common.serialization.Deserializer;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * KafkaSpoutConfig defines the required configuration to connect a
consumer to a consumer group, as well as the subscribing topics
+ */
+public class KafkaSpoutConfig implements Serializable {
+public static final long DEFAULT_POLL_TIMEOUT_MS = 2_000;
// 2s
+public static final long DEFAULT_OFFSET_COMMIT_PERIOD_MS = 15_000;
// 15s
+public static final int DEFAULT_MAX_RETRIES = Integer.MAX_VALUE;
// Retry forever
+public static final int DEFAULT_MAX_UNCOMMITTED_OFFSETS = 10_000;
// 10,000 records
+
+// Kafka property names
+public interface Consumer {
+String GROUP_ID = "group.id";
+String BOOTSTRAP_SERVERS = "bootstrap.servers";
+String ENABLE_AUTO_COMMIT = "enable.auto.commit";
+String AUTO_COMMIT_INTERVAL_MS = "auto.commit.interval.ms";
+String KEY_DESERIALIZER = "key.deserializer";
+String VALUE_DESERIALIZER = "value.deserializer";
+}
+
+/**
+ * The offset used by the Kafka spout in the first poll to Kafka
broker. The choice of this parameter will
+ * affect the number of consumer records returned in the first poll.
By default this parameter is set to UNCOMMITTED_EARLIEST.
+ * The allowed values are EARLIEST, LATEST, UNCOMMITTED_EARLIEST,
UNCOMMITTED_LATEST.
+ *
+ * EARLIEST means that the kafka spout polls records starting in
the first offset of the partition, regardless of previous commits
+ * LATEST means that the kafka spout polls records with offsets
greater than the last offset in the partition, regardless of previous
commits
+ * UNCOMMITTED_EARLIEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as EARLIEST.
+ * UNCOMMITTED_LATEST means that the kafka spout polls records
from the last committed offset, if any.
+ * If no offset has been committed, it behaves as LATEST.
+ *
+ * */
+public enum FirstPollOffsetStrategy {
+EARLIEST,
+LATEST,
+UNCOMMITTED_EARLIEST,
+UNCOMMITTED_LATEST }
+
+/**
+ * Defines when to poll the next batch of records from Kafka. The
choice of this parameter will affect throughput and the memory
+ * footprint of the Kafka spout. The allowed values are STREAM and
BATCH. STREAM will likely have higher throughput and use more memory
+ * (it stores in memory the entire KafkaRecord,including data). BATCH
will likely have less throughput but also use less memory.
+ * The BATCH behavior is similar to the behavior of the previous Kafka
Spout. De default value is STREAM.
+ *
+ * STREAM Every periodic call to nextTuple polls a new batch
of records from Kafka as long as the maxUncommittedOffsets
+ * threshold has not yet been reached. When the threshold his
reached, no more records are polled until enough offsets have been
+ * committed, such that the number of pending offsets is less than
maxUncommittedOffsets. See {@link Builder#setMaxUncommittedOffsets(int)}
+ *
+ * BATCH Only polls a
[jira] [Commented] (STORM-1617) storm.apache.org has no release specific documentation
[ https://issues.apache.org/jira/browse/STORM-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202895#comment-15202895 ] Robert Joseph Evans commented on STORM-1617: I know this is a lot but I think I have things worked out. I have created a branch for the new web page at `https://svn.apache.org/repos/asf/storm/branches/bobby-versioned-site` Most of the changes were not about rewriting anything, it was just moving things around and dividing them up by release, and what was relevant to a release. I did modify the 1.x and 2.x snapshot docs to point to the correct things in git and javadocs as namespaces have changed and then some clojure code has moved to java. Beyond that it is not that changed. I would like to get most of it in quickly so if there are minor things that need fixed/tweaked I am happy to do them on a follow on JIRA. > storm.apache.org has no release specific documentation > -- > > Key: STORM-1617 > URL: https://issues.apache.org/jira/browse/STORM-1617 > Project: Apache Storm > Issue Type: Bug > Components: asf-site >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > Our current documentation on http://storm.apache.org/ has no place to put > release specific documentation. Ever other project I know of has a little > bit of generic information, but most of the site is tied to a specific > release. I would like to copy that model and I propose that the following > files be maintained only on the asf-site branch. All other files will be > moved to a documentation directory on the individual branches, and a copy of > each, along with the generated javadocs, will be copied to a release specific > directory for each release. > ./index.html > ./releases.html (Combined downloads + documentation for each release) > ./contribute/BYLAWS.md > ./contribute/Contributing-to-Storm.md > ./contribute/People.md (Perhaps just PMC for this) > ./_posts > ./news.html > ./feed.xml > ./getting-help.html > ./LICENSE.html (Apache License) > ./talksAndVideos.md > Possibly also: > ./about/deployment.md > ./about/fault-tolerant.md > ./about/free-and-open-source.md > ./about/guarantees-data-processing.md > ./about/integrates.md > ./about/multi-language.md > ./about/scalable.md > ./about/simple-api.md > ./about.md -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1614: backpressure init and cleanup chan...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1206 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] storm pull request: STORM-1464: Support multiple file outputs
Github user dossett commented on the pull request: https://github.com/apache/storm/pull/1044#issuecomment-198393099 @arunmahadevan Writers should never be writing to the same file. `getBasePathForNextFile` will increment the rotation value before opening the file. File name also includes `System.currentTimeMillis()` which should further prevent conflict. There's also a unit test that attempts to validate this by writing two different schemas and confirming that two files are created: https://github.com/dossett/storm/blob/STORM-1494/external/storm-hdfs/src/test/java/org/apache/storm/hdfs/bolt/AvroGenericRecordBoltTest.java#L156-L168 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1617) storm.apache.org has no release specific documentation
[ https://issues.apache.org/jira/browse/STORM-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202886#comment-15202886 ] ASF GitHub Bot commented on STORM-1617: --- GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/1234 STORM-1617: Versioned Docs This is intended to be merged and used with https://svn.apache.org/repos/asf/storm/branches/bobby-versioned-site You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm versioned-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1234.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1234 commit abbfb9737e4c99eb27b96aa85335294b025c348e Author: Robert (Bobby) Evans Date: 2016-03-19T17:15:21Z STORM-1617: Release Specific Documentation 0.9.x Conflicts: .gitignore docs/README.md docs/_config.yml docs/_includes/footer.html docs/_includes/head.html docs/_includes/header.html docs/_layouts/about.html docs/_layouts/default.html docs/_layouts/documentation.html docs/_layouts/page.html docs/_layouts/post.html docs/assets/css/bootstrap.css docs/assets/css/bootstrap.css.map docs/assets/js/bootstrap.min.js docs/images/logos/alibaba.jpg docs/images/logos/groupon.jpg docs/images/logos/parc.png docs/images/logos/webmd.jpg docs/images/topology.png Conflicts: .gitignore docs/README.md docs/STORM-UI-REST-API.md docs/_config.yml docs/_includes/footer.html docs/_includes/head.html docs/_includes/header.html docs/_layouts/about.html docs/_layouts/default.html docs/_layouts/documentation.html docs/_layouts/page.html docs/_layouts/post.html docs/assets/css/bootstrap.css docs/assets/css/bootstrap.css.map docs/assets/js/bootstrap.min.js docs/images/logos/alibaba.jpg docs/images/logos/groupon.jpg docs/images/logos/parc.png docs/images/logos/webmd.jpg commit d909db8fe2aee7633b6f3b95ae71c415efea495f Author: Robert (Bobby) Evans Date: 2016-03-19T17:33:31Z STORM-1617: 0.10.x release docs Conflicts: docs/images/topology.png commit 69dfb5322f896057c21cf6bf758938cb14bc3984 Author: Robert (Bobby) Evans Date: 2016-03-19T17:56:04Z STORM-1617: added in 1.x specific doc changes commit 96b1cac47aac751f3a6bf8efa3df032e1759abf0 Author: Robert (Bobby) Evans Date: 2016-03-19T18:04:58Z STORM-1617: Added in 2.x specific doc changes > storm.apache.org has no release specific documentation > -- > > Key: STORM-1617 > URL: https://issues.apache.org/jira/browse/STORM-1617 > Project: Apache Storm > Issue Type: Bug > Components: asf-site >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > Our current documentation on http://storm.apache.org/ has no place to put > release specific documentation. Ever other project I know of has a little > bit of generic information, but most of the site is tied to a specific > release. I would like to copy that model and I propose that the following > files be maintained only on the asf-site branch. All other files will be > moved to a documentation directory on the individual branches, and a copy of > each, along with the generated javadocs, will be copied to a release specific > directory for each release. > ./index.html > ./releases.html (Combined downloads + documentation for each release) > ./contribute/BYLAWS.md > ./contribute/Contributing-to-Storm.md > ./contribute/People.md (Perhaps just PMC for this) > ./_posts > ./news.html > ./feed.xml > ./getting-help.html > ./LICENSE.html (Apache License) > ./talksAndVideos.md > Possibly also: > ./about/deployment.md > ./about/fault-tolerant.md > ./about/free-and-open-source.md > ./about/guarantees-data-processing.md > ./about/integrates.md > ./about/multi-language.md > ./about/scalable.md > ./about/simple-api.md > ./about.md -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] storm pull request: STORM-1617: Versioned Docs 0.9.x
GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/1237 STORM-1617: Versioned Docs 0.9.x This is intended to be merged and used with https://svn.apache.org/repos/asf/storm/branches/bobby-versioned-site You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm versioned-docs-0.9.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1237.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1237 commit 8af05748bc590fc0048630a8bbd893852a277e1e Author: Robert (Bobby) Evans Date: 2016-03-19T17:15:21Z STORM-1617: Release Specific Documentation 0.9.x --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (STORM-1617) storm.apache.org has no release specific documentation
[ https://issues.apache.org/jira/browse/STORM-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202894#comment-15202894 ] ASF GitHub Bot commented on STORM-1617: --- GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/1237 STORM-1617: Versioned Docs 0.9.x This is intended to be merged and used with https://svn.apache.org/repos/asf/storm/branches/bobby-versioned-site You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm versioned-docs-0.9.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/1237.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1237 commit 8af05748bc590fc0048630a8bbd893852a277e1e Author: Robert (Bobby) Evans Date: 2016-03-19T17:15:21Z STORM-1617: Release Specific Documentation 0.9.x > storm.apache.org has no release specific documentation > -- > > Key: STORM-1617 > URL: https://issues.apache.org/jira/browse/STORM-1617 > Project: Apache Storm > Issue Type: Bug > Components: asf-site >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > Our current documentation on http://storm.apache.org/ has no place to put > release specific documentation. Ever other project I know of has a little > bit of generic information, but most of the site is tied to a specific > release. I would like to copy that model and I propose that the following > files be maintained only on the asf-site branch. All other files will be > moved to a documentation directory on the individual branches, and a copy of > each, along with the generated javadocs, will be copied to a release specific > directory for each release. > ./index.html > ./releases.html (Combined downloads + documentation for each release) > ./contribute/BYLAWS.md > ./contribute/Contributing-to-Storm.md > ./contribute/People.md (Perhaps just PMC for this) > ./_posts > ./news.html > ./feed.xml > ./getting-help.html > ./LICENSE.html (Apache License) > ./talksAndVideos.md > Possibly also: > ./about/deployment.md > ./about/fault-tolerant.md > ./about/free-and-open-source.md > ./about/guarantees-data-processing.md > ./about/integrates.md > ./about/multi-language.md > ./about/scalable.md > ./about/simple-api.md > ./about.md -- This message was sent by Atlassian JIRA (v6.3.4#6332)
