[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2310


---

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r161298748
  
--- Diff: nifi-assembly/LICENSE ---
@@ -2073,3 +2073,93 @@ style license.
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
+
+  The binary distribution of this product bundles 'ANTLR 4' which is 
available
+under a "3-clause BSD" license.  For details see 
http://www.antlr.org/license.html
+
+  Copyright (c) 2012 Terence Parr and Sam Harwell
+  All rights reserved.
+  Redistribution and use in source and binary forms, with or without 
modification, are permitted
+  provided that the following conditions are met:
+
+  Redistributions of source code must retain the above copyright 
notice, this list of
+  conditions and the following disclaimer.
+  Redistributions in binary form must reproduce the above copyright 
notice, this list of
+  conditions and the following disclaimer in the documentation and/or 
other materials
+  provided with the distribution.
+
+  Neither the name of the author nor the names of its contributors may 
be used to endorse
+  or promote products derived from this software without specific 
prior written permission.
+
+  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
"AS IS" AND ANY
+  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
IN NO EVENT SHALL
+  THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
INDIRECT, INCIDENTAL,
+  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
LIMITED TO,
+  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
PROFITS; OR BUSINESS
+  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 
IN CONTRACT,
+  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
ARISING IN ANY WAY OUT OF
+  THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 
DAMAGE.
+
+  The binary distribution of this product bundles 'icu4j'
+which is available under a X-style license.
--- End diff --

Done, and squashed to 2 commits (original from Vadim and one of mine)


---

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r161298116
  
--- Diff: nifi-assembly/LICENSE ---
@@ -2073,3 +2073,93 @@ style license.
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
+
+  The binary distribution of this product bundles 'ANTLR 4' which is 
available
+under a "3-clause BSD" license.  For details see 
http://www.antlr.org/license.html
+
+  Copyright (c) 2012 Terence Parr and Sam Harwell
+  All rights reserved.
+  Redistribution and use in source and binary forms, with or without 
modification, are permitted
+  provided that the following conditions are met:
+
+  Redistributions of source code must retain the above copyright 
notice, this list of
+  conditions and the following disclaimer.
+  Redistributions in binary form must reproduce the above copyright 
notice, this list of
+  conditions and the following disclaimer in the documentation and/or 
other materials
+  provided with the distribution.
+
+  Neither the name of the author nor the names of its contributors may 
be used to endorse
+  or promote products derived from this software without specific 
prior written permission.
+
+  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
"AS IS" AND ANY
+  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
IN NO EVENT SHALL
+  THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
INDIRECT, INCIDENTAL,
+  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
LIMITED TO,
+  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
PROFITS; OR BUSINESS
+  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 
IN CONTRACT,
+  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
ARISING IN ANY WAY OUT OF
+  THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 
DAMAGE.
+
+  The binary distribution of this product bundles 'icu4j'
+which is available under a X-style license.
--- End diff --

Please Refer to it as the ICU License as noted here 
http://source.icu-project.org/repos/icu/icu4j/tags/release-4-8-1/main/shared/licenses/license.html.

Fortunately that is in the Apache Category A so we're good here.  Just call 
it out as the ICU License as in the LIcense text instead of 'X-Style'

thanks


---

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r161297870
  
--- Diff: nifi-assembly/LICENSE ---
@@ -2073,3 +2073,93 @@ style license.
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
+
+  The binary distribution of this product bundles 'ANTLR 4' which is 
available
+under a "3-clause BSD" license.  For details see 
http://www.antlr.org/license.html
+
+  Copyright (c) 2012 Terence Parr and Sam Harwell
+  All rights reserved.
+  Redistribution and use in source and binary forms, with or without 
modification, are permitted
+  provided that the following conditions are met:
+
+  Redistributions of source code must retain the above copyright 
notice, this list of
+  conditions and the following disclaimer.
+  Redistributions in binary form must reproduce the above copyright 
notice, this list of
+  conditions and the following disclaimer in the documentation and/or 
other materials
+  provided with the distribution.
+
+  Neither the name of the author nor the names of its contributors may 
be used to endorse
+  or promote products derived from this software without specific 
prior written permission.
+
+  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
"AS IS" AND ANY
+  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
IN NO EVENT SHALL
+  THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
INDIRECT, INCIDENTAL,
+  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
LIMITED TO,
+  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
PROFITS; OR BUSINESS
+  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 
IN CONTRACT,
+  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
ARISING IN ANY WAY OUT OF
+  THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 
DAMAGE.
+
+  The binary distribution of this product bundles 'icu4j'
+which is available under a X-style license.
--- End diff --

The reference to "X-Style" comes from [this 
page](http://source.icu-project.org/repos/icu/icu4j/tags/release-4-8-1/readme.html#license).
 However the 
[license](http://source.icu-project.org/repos/icu/icu4j/tags/release-4-8-1/main/shared/licenses/license.html)
 does look MIT-style.


---

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r161297207
  
--- Diff: nifi-assembly/LICENSE ---
@@ -2073,3 +2073,93 @@ style license.
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
+
+  The binary distribution of this product bundles 'ANTLR 4' which is 
available
+under a "3-clause BSD" license.  For details see 
http://www.antlr.org/license.html
+
+  Copyright (c) 2012 Terence Parr and Sam Harwell
+  All rights reserved.
+  Redistribution and use in source and binary forms, with or without 
modification, are permitted
+  provided that the following conditions are met:
+
+  Redistributions of source code must retain the above copyright 
notice, this list of
+  conditions and the following disclaimer.
+  Redistributions in binary form must reproduce the above copyright 
notice, this list of
+  conditions and the following disclaimer in the documentation and/or 
other materials
+  provided with the distribution.
+
+  Neither the name of the author nor the names of its contributors may 
be used to endorse
+  or promote products derived from this software without specific 
prior written permission.
+
+  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
"AS IS" AND ANY
+  EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
IMPLIED WARRANTIES OF
+  MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
IN NO EVENT SHALL
+  THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, 
INDIRECT, INCIDENTAL,
+  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
LIMITED TO,
+  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 
PROFITS; OR BUSINESS
+  INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER 
IN CONTRACT,
+  STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
ARISING IN ANY WAY OUT OF
+  THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH 
DAMAGE.
+
+  The binary distribution of this product bundles 'icu4j'
+which is available under a X-style license.
--- End diff --

Cannot have 'X-Style'.  Is this MIT?  Can you share the link to the LICENSE 
this is coming from?


---

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2310#discussion_r157704512
  
--- Diff: 
nifi-nar-bundles/nifi-druid-bundle/nifi-druid-controller-service/src/main/java/org/apache/nifi/controller/druid/DruidTranquilityController.java
 ---
@@ -0,0 +1,614 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.druid;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationContext;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.controller.api.druid.DruidTranquilityService;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.codehaus.jackson.map.ObjectMapper;
+
+import com.metamx.common.Granularity;
+import com.metamx.tranquility.beam.Beam;
+import com.metamx.tranquility.beam.ClusteredBeamTuning;
+import com.metamx.tranquility.druid.DruidBeamConfig;
+import com.metamx.tranquility.druid.DruidBeams;
+import com.metamx.tranquility.druid.DruidDimensions;
+import com.metamx.tranquility.druid.DruidEnvironment;
+import com.metamx.tranquility.druid.DruidLocation;
+import com.metamx.tranquility.druid.DruidRollup;
+import com.metamx.tranquility.tranquilizer.Tranquilizer;
+import com.metamx.tranquility.typeclass.Timestamper;
+
+import io.druid.data.input.impl.TimestampSpec;
+import io.druid.granularity.QueryGranularity;
+import io.druid.query.aggregation.AggregatorFactory;
+import io.druid.query.aggregation.CountAggregatorFactory;
+import io.druid.query.aggregation.DoubleMaxAggregatorFactory;
+import io.druid.query.aggregation.DoubleMinAggregatorFactory;
+import io.druid.query.aggregation.DoubleSumAggregatorFactory;
+import io.druid.query.aggregation.LongMaxAggregatorFactory;
+import io.druid.query.aggregation.LongMinAggregatorFactory;
+import io.druid.query.aggregation.LongSumAggregatorFactory;
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+
+@Tags({"Druid", "Timeseries", "OLAP", "ingest"})
+@CapabilityDescription("Asynchronously sends flowfiles to Druid Indexing 
Task using Tranquility API. "
++ "If aggregation and roll-up of data is required, an Aggregator 
JSON descriptor needs to be provided."
++ "Details on how describe aggregation using JSON can be found at: 
http://druid.io/docs/latest/querying/aggregations.html;)
+public class DruidTranquilityController extends AbstractControllerService 
implements DruidTranquilityService {
+private final static String FIREHOSE_PATTERN = "druid:firehose:%s";
+
+private final static AllowableValue PT1M = new AllowableValue("PT1M", 
"1 minute", "1 minute");
+private final static AllowableValue PT10M = new 
AllowableValue("PT10M", "10 minutes", "10 minutes");
+private final static AllowableValue PT60M = new 
AllowableValue("PT60M", "60 minutes", "1 hour");
+
+private final static 

[GitHub] nifi pull request #2310: NIFI-4428: Add PutDruidRecord processor and DruidTr...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2310

NIFI-4428: Add PutDruidRecord processor and DruidTranquilityController 
service

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [x] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-4428

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2310.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2310


commit e502364ffb18683747c93af1deb85b0185b165a6
Author: vvaks 
Date:   2017-09-27T17:23:23Z

NIFI-4428: - Implement PutDruid Processor and Controller

update

added provenance report

added parameters for batch control

WIP

commit d7db6ed526a8e2a06625a2a0cd1e46ea2b923ff3
Author: Matthew Burgess 
Date:   2017-11-22T16:36:30Z

NIFI-4428: Replaced JSON input with RecordReader
using per-record flowfiles
NIFI-4428: Added initial L, fixed dependency hierarchy

NIFI-4428: Exposed additional ZK properties




---