[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/502 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/502#discussion_r71220449
--- Diff:
nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-processors/src/main/java/org/apache/nifi/processors/ignite/cache/PutIgniteCache.java
---
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.ignite.cache;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.ignite.IgniteDataStreamer;
+import org.apache.ignite.lang.IgniteFuture;
+import org.apache.nifi.annotation.behavior.EventDriven;
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.SupportsBatching;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.expression.AttributeExpression.ResultType;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.io.InputStreamCallback;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.stream.io.StreamUtils;
+
+/**
+ * Put cache processors which pushes the flow file content into Ignite
Cache using
+ * DataStreamer interface
+ */
+@EventDriven
+@SupportsBatching
+@Tags({ "Ignite", "insert", "update", "stream", "write", "put", "cache",
"key" })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@CapabilityDescription("Stream the contents of a FlowFile to Ignite Cache
using DataStreamer. " +
+"The processor uses the value of FlowFile attribute (Ignite cache
entry key) as the " +
+"cache key and the byte array of the FlowFile as the value of the
cache entry value. Both the string key and a " +
+" non-empty byte array value are required otherwise the FlowFile is
transfered to the failure relation. " +
+"Note - The Ignite Kernel periodically outputs node perforance
statistics to the logs. This message " +
+" can be turned off by setting the log level for logger
'org.apache.ignite' to WARN.")
--- End diff --
typo: "performance"
also I'd say "... to WARN in logback.xml configuration file."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user mans2singh commented on a diff in the pull request: https://github.com/apache/nifi/pull/502#discussion_r71093727 --- Diff: nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,26 @@ +nifi-ignite-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +=== +Apache Software License v2 +=== + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) Apache Ignite --- End diff -- Updated NOTICE. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/502#discussion_r71086795 --- Diff: nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,26 @@ +nifi-ignite-nar --- End diff -- Any NOTICE information in this Notice file that is not in the nifi-assembly NOTICE, should be added to the assembly NOTICE as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/502#discussion_r71086770 --- Diff: nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,26 @@ +nifi-ignite-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +=== +Apache Software License v2 +=== + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) Apache Ignite --- End diff -- This notice should be this: https://github.com/apache/ignite/blob/1.6.0/NOTICE --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/502#discussion_r71086749 --- Diff: pom.xml --- @@ -227,6 +227,29 @@ language governing permissions and limitations under the License. --> 1.5.3-M1 +org.apache.ignite +ignite-core +1.6.0 + + +org.apache.ignite +ignite-spring +1.6.0 + + +org.apached.ignite +ignite-core +1.6.0 +test-jar +test --- End diff -- I'd suggest against providing the scope here and instead putting it in the pom where it's used. Adding the scope test here will propagate to any package that brings in this dependency (even transitively). Comment also applies to ignite-log4j2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user JPercivall commented on a diff in the pull request: https://github.com/apache/nifi/pull/502#discussion_r71086694 --- Diff: nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-nar/src/main/resources/META-INF/NOTICE --- @@ -0,0 +1,26 @@ +nifi-ignite-nar +Copyright 2016 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +=== +Apache Software License v2 +=== + +The following binary components are provided under the Apache Software License v2 + + (ASLv2) Apache Ignite +The following NOTICE information applies: + Apache Ignite + Copyright 2015 The Apache Software Foundation + + (ASLv2) Apache Commons IO +The following NOTICE information applies: + Apache Commons IO + Copyright 2002-2012 The Apache Software Foundation + + (ASLv2) Guava --- End diff -- Since this is only a test scope dependency and not bundled into the nar, this does not need to be a part of the notice --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/502#discussion_r70520699
--- Diff:
nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-processors/src/main/java/org/apache/nifi/processors/ignite/cache/AbstractIgniteCacheProcessor.java
---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.ignite.cache;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.ignite.IgniteCache;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.ignite.AbstractIgniteProcessor;
+
+/**
+ * Base class of Ignite cache based processor
+ */
+public abstract class AbstractIgniteCacheProcessor extends
AbstractIgniteProcessor {
+
+/**
+ * Flow File attribute for cache entry key
+ */
+public static final String IGNITE_CACHE_ENTRY_KEY =
"ignite.cache.entry.key";
+
+/**
+ * Ignite cache name
+ */
+protected static final PropertyDescriptor CACHE_NAME = new
PropertyDescriptor.Builder()
+.displayName("Ignite Cache Name")
+.name("ignite-cache-name")
+.description("The name of the ignite cache where e")
--- End diff --
"where e"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #502: Nifi-1972 Apache Ignite Put Cache Processor
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/502#discussion_r70520573
--- Diff:
nifi-nar-bundles/nifi-ignite-bundle/nifi-ignite-processors/src/main/java/org/apache/nifi/processors/ignite/AbstractIgniteProcessor.java
---
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.nifi.processors.ignite;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.Ignition;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.util.StandardValidators;
+
+/**
+ * Base class for Ignite processors
+ */
+public abstract class AbstractIgniteProcessor extends AbstractProcessor {
+
+/**
+ * Ignite spring configuration file
+ */
+public static final PropertyDescriptor IGNITE_CONFIGURATION_FILE = new
PropertyDescriptor.Builder()
+.displayName("Ignite Spring Properties Xml File")
+.name("ignite-spring-properties-xml-file")
+.description("Ignite spring configuration file,
/.xml. If the " +
+"configuration file is not provided, default Ignite
configuration is used which default " +
+"configuration is used which binds to 127.0.0.1:47500..47509")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+/**
+ * Success relation
+ */
+public static final Relationship REL_SUCCESS = new
Relationship.Builder().name("success")
+.description("All FlowFiles that are written to Ignite cache
are routed to this relationship").build();
+
+/**
+ * Failure relation
+ */
+public static final Relationship REL_FAILURE = new
Relationship.Builder().name("failure")
+.description("All FlowFiles that cannot be written to Ignite
cache are routed to this relationship").build();
+
+/**
+ * The ignite instance
+ */
+private transient Ignite ignite;
+
+/**
+ * Get ignite instance
+ * @return iginte instance
+ */
+protected Ignite getIgnite() {
+return ignite;
+}
+
+/**
+ * Close ignite instance
+ */
+public void closeIgnite() {
+if (ignite != null) {
+getLogger().info("Closing ignite client");
+ignite.close();
+ignite = null;
+}
+}
+
+/**
+ * Initialize ignite instance
+ * @param context process context
+ */
+public void initializeIgnite(ProcessContext context) {
+
+if ( getIgnite() != null ) {
+getLogger().info("Ignite already initilaized");
--- End diff --
Typo. Also, do we want this with info level instead of debug? just asking
in case we could have this message a lot...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
