[GitHub] storm pull request #2490: STORM-2840: Getting resource availability from pre...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2490 ---
[GitHub] storm issue #2494: STORM-2880 Minor optimisation about kafka spout.
Github user OuYangLiang commented on the issue: https://github.com/apache/storm/pull/2494 @HeartSaVioR thanks, you can't see previous patch because I've squashed to one commit. ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2498 Maybe we could file a blocker issue for Storm 2.0.0 and allow this patch. Filed https://issues.apache.org/jira/browse/STORM-2882 for that. ---
[GitHub] storm pull request #2491: STORM-2872: TestRebalance can be flaky - fixing
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2491 ---
[GitHub] storm pull request #2312: YSTORM-2725: Generic Resource Scheduling - initial...
Github user govind-menon closed the pull request at: https://github.com/apache/storm/pull/2312 ---
[GitHub] storm pull request #2500: STORM-2876: Work around memory leak, and try to sp...
GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2500 STORM-2876: Work around memory leak, and try to speed up tests The actual fix is just the changes in storm-hdfs/pom.xml But this added 1 min to the build time on a decently powerful box. So to combat that I made as many of the test run in parallel. Some of the tests didn't work well in this mode, so I disabled the parallelism for storm-hive, storm-kafka-client, and integration-test. Storm-hdfs and storm-hive have some issues with where data is stored, and we could probably make it work with the right settings. storm-kafka-client mostly works, but occasionally I would see a test hang and I didn't have time to debug it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/revans2/incubator-storm STORM-2876 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2500.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2500 commit 19878bdb05e5eca4bf47012039243bbf9d14ec81 Author: Robert (Bobby) EvansDate: 2018-01-03T20:35:04Z STORM-2876: Work around memory leak, and try to speed up tests ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2498 It doesn't represent we decided to not relocate. We just didn't decide it, and it should be done within releasing 2.0.0. ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2498 https://mail-archives.apache.org/mod_mbox/incubator-storm-dev/201703.mbox/%3ccaf5108gjjqzsyywcp99bgpgec7tufe-tbt9fi0m78ah8rkm...@mail.gmail.com%3E You can follow up the discussion to see why I removed relocation for now. We are splitting storm-core module to several parts, and if we relocate something and refer it within the project, it should be much painful. ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2498 @HeartSaVioR , any reason why we dont relocate in 2.0.0 vs 1.x ? If not users would hit issues with conflicting versions (like curator). ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2498 In master branch we don't relocate any dependencies so it will make conflict anyway. Looks like this case represents that we should have a solution for 2.0.0 sooner. ---
[GitHub] storm pull request #2499: STORM-2881: Explicitly specify the curator depende...
GitHub user arunmahadevan opened a pull request: https://github.com/apache/storm/pull/2499 STORM-2881: Explicitly specify the curator dependencies in storm-druid pom https://github.com/apache/storm/pull/2498 applied to 1.x branch. You can merge this pull request into a Git repository by running: $ git pull https://github.com/arunmahadevan/storm STORM-2881-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2499.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2499 commit 5e11867cfa39baa9058e4e81a400752954c678c3 Author: Arun MahadevanDate: 2018-01-03T20:52:47Z STORM-2881: Explicitly specify the curator dependencies in storm-druid pom ---
[GitHub] storm issue #2499: STORM-2881: Explicitly specify the curator dependencies i...
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2499 ping @omkreddy , @HeartSaVioR ---
[GitHub] storm issue #2498: STORM-2881: Explicitly specify the curator dependencies i...
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2498 ping @omkreddy , @HeartSaVioR ---
[GitHub] storm pull request #2498: STORM-2881: Explicitly specify the curator depende...
GitHub user arunmahadevan opened a pull request: https://github.com/apache/storm/pull/2498 STORM-2881: Explicitly specify the curator dependencies in storm-druid pom You can merge this pull request into a Git repository by running: $ git pull https://github.com/arunmahadevan/storm STORM-2881 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2498.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2498 commit 08b962730774482844551ffd622c3e4a960c5c6a Author: Arun MahadevanDate: 2018-01-03T20:52:47Z STORM-2881: Explicitly specify the curator dependencies in storm-druid pom ---
[GitHub] storm pull request #2203: STORM-2153: New Metrics Reporting API
Github user ptgoetz commented on a diff in the pull request: https://github.com/apache/storm/pull/2203#discussion_r159519245 --- Diff: storm-core/src/jvm/org/apache/storm/utils/DisruptorQueue.java --- @@ -65,7 +66,7 @@ private static final Object INTERRUPT = new Object(); private static final String PREFIX = "disruptor-"; private static final FlusherPool FLUSHER = new FlusherPool(); -private static final Timer METRICS_TIMER = new Timer("disruptor-metrics-timer", true); +private static final ScheduledThreadPoolExecutor METRICS_REPORTER_EXECUTOR = new ScheduledThreadPoolExecutor(1); --- End diff -- You were right. Fixed. ---
[GitHub] storm pull request #2203: STORM-2153: New Metrics Reporting API
Github user ptgoetz commented on a diff in the pull request: https://github.com/apache/storm/pull/2203#discussion_r159515622 --- Diff: storm-core/src/jvm/org/apache/storm/utils/DisruptorQueue.java --- @@ -65,7 +66,7 @@ private static final Object INTERRUPT = new Object(); private static final String PREFIX = "disruptor-"; private static final FlusherPool FLUSHER = new FlusherPool(); -private static final Timer METRICS_TIMER = new Timer("disruptor-metrics-timer", true); +private static final ScheduledThreadPoolExecutor METRICS_REPORTER_EXECUTOR = new ScheduledThreadPoolExecutor(1); --- End diff -- Good catch. I'll check and update as necessary. ---
[GitHub] storm pull request #2203: STORM-2153: New Metrics Reporting API
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2203#discussion_r159514108 --- Diff: storm-core/src/jvm/org/apache/storm/utils/DisruptorQueue.java --- @@ -65,7 +66,7 @@ private static final Object INTERRUPT = new Object(); private static final String PREFIX = "disruptor-"; private static final FlusherPool FLUSHER = new FlusherPool(); -private static final Timer METRICS_TIMER = new Timer("disruptor-metrics-timer", true); +private static final ScheduledThreadPoolExecutor METRICS_REPORTER_EXECUTOR = new ScheduledThreadPoolExecutor(1); --- End diff -- Thanks for replacing the Timer. If the threads for this still need to be daemon threads, you should use the constructor that takes a ThreadFactory, I believe the default factory produces non-daemon threads. ---
[GitHub] storm pull request #2480: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2480 ---
[GitHub] storm pull request #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2495 ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2495 I believe we don't need to wait 24 hours since this is just port work. Will merge shortly. ---
[GitHub] storm issue #2480: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2480 +1 ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user arunmahadevan commented on the issue: https://github.com/apache/storm/pull/2495 @omkreddy , thanks for the patch. +1 ---
[GitHub] storm issue #2467: STORM-2860: Add Kerberos support to Solr bolt
Github user omkreddy commented on the issue: https://github.com/apache/storm/pull/2467 PR for master: https://github.com/apache/storm/pull/2497 ---
[GitHub] storm pull request #2497: STORM-2860: Add Kerberos support to Solr bolt
GitHub user omkreddy opened a pull request: https://github.com/apache/storm/pull/2497 STORM-2860: Add Kerberos support to Solr bolt You can merge this pull request into a Git repository by running: $ git pull https://github.com/omkreddy/storm kerberos-solr-bolt-master1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2497.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2497 commit 8e256a532bc76435c3a292db814df57579200cd0 Author: Manikumar ReddyDate: 2017-12-18T07:05:19Z STORM-2860: Add Kerberos support to Solr bolt ---
[GitHub] storm pull request #2496: STORM-2858 1.x
GitHub user srdo opened a pull request: https://github.com/apache/storm/pull/2496 STORM-2858 1.x 1.x version of https://github.com/apache/storm/pull/2462 I didn't think we needed this initially since asprintf isn't used in 1.x, but it looks like the build is failing on 1.x after upgrading to the newest Travis image (e.g. https://travis-ci.org/apache/storm/builds/324620269). This only contains the replacement of make-maven-plugin. Make-maven-plugin fails to run autoreconf on Travis. The error log is ``` [DEBUG] Executing: /bin/sh -l -c cd /home/travis/build/srdo/storm/storm-core/target/native/worker-launcher && autoreconf -i [DEBUG] /bin/sh: 14: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 21: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 28: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 35: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 42: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 49: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 56: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 62: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 62: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 62: /opt/jdk_switcher/jdk_switcher.sh: [[: not found [DEBUG] /bin/sh: 109: /opt/jdk_switcher/jdk_switcher.sh: Syntax error: "(" unexpected (expecting "fi") ``` Googling it leads to https://github.com/travis-ci/travis-cookbooks/issues/964. I'd guess jdk_switcher isn't expecting to be called with a non-bash shell? You can merge this pull request into a Git repository by running: $ git pull https://github.com/srdo/storm STORM-2858-1.x Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2496.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2496 commit 5753913436b72fb3f1844c25ea559d4bc02cff30 Author: Stig Rohde DøssingDate: 2017-12-13T15:47:31Z STORM-2858: Fix worker-launcher build by erroring out if asprintf fails to allocate memory. Replace make-maven-plugin with a build shell script. ---
[GitHub] storm pull request #2489: STORM-2874 1.x: Minor refactoring of backpressure ...
Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2489 ---
[GitHub] storm issue #2494: STORM-2880 Minor optimisation about kafka spout.
Github user OuYangLiang commented on the issue: https://github.com/apache/storm/pull/2494 @srdo I still think it makes senses, and made a few changes, please review it again, thanks. ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user omkreddy commented on the issue: https://github.com/apache/storm/pull/2495 @srdo squashed the commits. ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2495 Thanks, this looks good. +1, if you squash to one commit we should be able to merge soon. Please also squash the 1.x version. ---
[GitHub] storm pull request #2480: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user omkreddy commented on a diff in the pull request: https://github.com/apache/storm/pull/2480#discussion_r159413588 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -144,10 +147,37 @@ public void open(Map conf, TopologyContext context, SpoutOutputCollector collect setCommitMetadata(context); tupleListener.open(conf, context); +if (canRegisterMetrics()) registerMetric(); --- End diff -- Updated the PR. ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user omkreddy commented on the issue: https://github.com/apache/storm/pull/2495 @srdo Thanks for the review. Fixed the checkstyle issues. ---
[GitHub] storm issue #2467: STORM-2860: Add Kerberos support to Solr bolt
Github user omkreddy commented on the issue: https://github.com/apache/storm/pull/2467 Updated the PR for 1.x. Will raise PR for master ---
[GitHub] storm pull request #2467: STORM-2860: Add Kerberos support to Solr bolt
Github user omkreddy commented on a diff in the pull request: https://github.com/apache/storm/pull/2467#discussion_r159409982 --- Diff: external/storm-solr/src/main/java/org/apache/storm/solr/schema/builder/RestJsonSchemaBuilderV2.java --- @@ -0,0 +1,127 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.storm.solr.schema.builder; + +import org.apache.solr.client.solrj.SolrClient; +import org.apache.solr.client.solrj.SolrServerException; +import org.apache.solr.client.solrj.impl.CloudSolrClient; +import org.apache.solr.client.solrj.impl.HttpClientUtil; +import org.apache.solr.client.solrj.impl.Krb5HttpClientConfigurer; +import org.apache.solr.client.solrj.request.schema.FieldTypeDefinition; +import org.apache.solr.client.solrj.request.schema.SchemaRequest; +import org.apache.solr.client.solrj.response.schema.SchemaRepresentation; +import org.apache.solr.client.solrj.response.schema.SchemaResponse; +import org.apache.storm.solr.config.SolrConfig; +import org.apache.storm.solr.schema.CopyField; +import org.apache.storm.solr.schema.Field; +import org.apache.storm.solr.schema.FieldType; +import org.apache.storm.solr.schema.Schema; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.LinkedList; +import java.util.List; +import java.util.Map; + +/** + * Class that builds the {@link Schema} object from the schema returned by the SchemaRequest + */ +public class RestJsonSchemaBuilderV2 implements SchemaBuilder { +private static final Logger logger = LoggerFactory.getLogger(RestJsonSchemaBuilderV2.class); +private Schema schema = new Schema(); +private SolrConfig solrConfig; +private String collection; + +public RestJsonSchemaBuilderV2(SolrConfig solrConfig, String collection) { +this.solrConfig = solrConfig; +this.collection = collection; +} + +@Override +public void buildSchema() throws IOException { +SolrClient solrClient = null; +try { +if (solrConfig.enableKerberos()) +HttpClientUtil.setConfigurer(new Krb5HttpClientConfigurer()); + +solrClient = new CloudSolrClient(solrConfig.getZkHostString()); --- End diff -- RestJsonSchemaBuilderV2.java is also based JSON. it just that instead parsing the url the result, we are using CloudSolrClient to build schema. This internally uses http calls. Functionality wise both are same. ---
[GitHub] storm pull request #2467: STORM-2860: Add Kerberos support to Solr bolt
Github user omkreddy commented on a diff in the pull request: https://github.com/apache/storm/pull/2467#discussion_r159409508 --- Diff: external/storm-solr/README.md --- @@ -171,7 +194,7 @@ Querying Solr for these patterns, you will see the values that have been indexe curl -X GET -H "Content-type:application/json" -H "Accept:application/json" http://localhost:8983/solr/gettingstarted_shard1_replica2/select?q=*id_fields_test_val*=json=true -curl -X GET -H "Content-type: application/json" -H "Accept: application/json" http://localhost:8983/solr/gettingstarted_shard1_replica2/select?q=*id_fields_test_val*=json=true +curl -X GET -H "Content-type:application/json" -H "Accept:application/json" http://localhost:8983/solr/gettingstarted_shard1_replica2/select?q=*json_test_val*=json=true --- End diff -- yes, looks like a type. same link pasted twice ---
[GitHub] storm issue #2494: STORM-2880 Minor optimisation about kafka spout.
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2494 Bikeshedding incoming: I'm not sure this is better. The two methods in ProcessingGuarantee are just named variants of equals. I think we might as well just replace the checks in KafkaSpout with == checks. I don't think e.g. `kafkaSpoutConfig.getProcessingGuarantee() == KafkaSpoutConfig.ProcessingGuarantee.AT_MOST_ONCE` is following the SRP any less than `kafkaSpoutConfig.getProcessingGuarantee().isAtMostOnceProcessing()` (the second one also violates the law of Demeter). ---
[GitHub] storm issue #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user srdo commented on the issue: https://github.com/apache/storm/pull/2495 There are checkstyle violations, you can find the list in storm-kafka-client/target/checkstyle-violations.xml. Other than that this looks good. ---
[GitHub] storm pull request #2480: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2480#discussion_r159407800 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -144,10 +147,37 @@ public void open(Map conf, TopologyContext context, SpoutOutputCollector collect setCommitMetadata(context); tupleListener.open(conf, context); +if (canRegisterMetrics()) registerMetric(); --- End diff -- Nit: Use braces. ---
[GitHub] storm issue #2480: STORM-2867: Add consumer lag metrics to KafkaSpout
Github user omkreddy commented on the issue: https://github.com/apache/storm/pull/2480 @srdo Raised PR for master : https://github.com/apache/storm/pull/2495 ---
[GitHub] storm pull request #2495: STORM-2867: Add consumer lag metrics to KafkaSpout
GitHub user omkreddy opened a pull request: https://github.com/apache/storm/pull/2495 STORM-2867: Add consumer lag metrics to KafkaSpout You can merge this pull request into a Git repository by running: $ git pull https://github.com/omkreddy/storm kafka-metrics-apache-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2495.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2495 commit ca2c2f7440db3c9efcd0200c0cda4504b6a8d779 Author: Manikumar Reddy ODate: 2017-12-20T15:18:51Z STORM-2867: Add consumer lag metrics to KafkaSpout commit 216f867fbfca59d8a0b2d67584feb5313e9123e1 Author: Manikumar Reddy Date: 2018-01-03T08:42:33Z use java supplier ---
[GitHub] storm pull request #2494: STORM-2880 Minor optimisation about kafka spout.
GitHub user OuYangLiang opened a pull request: https://github.com/apache/storm/pull/2494 STORM-2880 Minor optimisation about kafka spout. Based on the **single responsibility principle**, method `isAtLeastOnceProcessing()` should reside in `KafkaSpoutConfig` rather than `KafkaSpout`. This patch removes the dependency of `KafkaSpoutConfig.ProcessingGuarantee` from `KafkaSpout`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/OuYangLiang/storm master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/2494.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2494 commit 9a983167523a26d90266c1d689f36f464e062e66 Author: ouyangliangDate: 2018-01-03T08:52:42Z STORM-2880 Minor optimisation about kafka spout. Based on the single responsibility principle, method isAtLeastOnceProcessing() should reside in KafkaSpoutConfig rather than KafkaSpout. This patch removes the dependency of KafkaSpoutConfig.ProcessingGuarantee from KafkaSpout. ---