tpalfy commented on code in PR #6303: URL: https://github.com/apache/nifi/pull/6303#discussion_r977978856
########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/resources/docs/org.apache.nifi.shopify.processors.GetShopify/additionalDetails.html: ########## @@ -0,0 +1,148 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8"/> + <title>GetShopify</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/> +</head> + +<body> +<h2>Setting Up a Custom App</h2> +<p> + Follow the <a href="https://help.shopify.com/en/manual/apps/custom-apps";>Shopify tutorial</a> to enable and create + private apps, set API Scopes and generate API tokens. +</p> +<h2>Incremental Loading</h2> +<p> + Some resources can be processed incrementally by NiFi. This means that only resources created or modified after the + last run time of the processor are displayed. The processor state can be reset in the context menu. The following + list shows which date-time fields are incremented for which resources. +<ul> + <li>Access + <ul> + <li>Access Scope: none</li> + <li>StoreFront Access Token: none</li> + </ul> + </li> + <li>Analytics + <ul> + <li>Reports: updated_at_min</li> + </ul> + </li> + <li>Billing + <ul> + <li>Application Charge: none</li> + <li>Application Credit: none</li> + <li>Recurring Application Charge: none</li> + </ul> + </li> + <li>Customers + <ul> + <li>Customers: updated_at_min</li> + <li>Customer Saved Searches: none</li> + </ul> + </li> + <li>Discounts + <ul> + <li>Price Rules: updated_at_min</li> + </ul> + </li> + <li>Events + <ul> + <li>Events: created_at_min</li> + </ul> + </li> + <li>Inventory + <ul> + <li>Inventory Levels: updated_at_min</li> + <li>Locations: none</li> + </ul> + </li> + <li>Marketing Event + <ul> + <li>Marketing Events: none</li> + </ul> + </li> + <li>Metafields + <ul> + <li>Metafields: updated_at_min</li> + </ul> + </li> + <li>Online Store + <ul> + <li>Blogs: none</li> + <li>Comment: none</li> + <li>Pages: none</li> + <li>Redirects: none</li> + <li>Script Tags: updated_at_min</li> + <li>Themes: none</li> + </ul> + </li> + <li>Orders + <ul> + <li>Abandoned Checkouts: updated_at_min</li> + <li>Draft Orders: updated_at_min</li> + <li>Orders: updated_at_min</li> + </ul> + </li> + <li>Plus + <ul> + <li>Gift Cards: none</li> + <li>Users: none</li> + </ul> + </li> + <li>Product + <ul> + <li>Collects: none</li> + <li>Custom Collections: updated_at_min</li> + <li>Products: updated_at_min</li> + <li>Smart Collections: updated_at_min</li> + </ul> + </li> + <li>Sales Channels + <ul> + <li>Collection Listings: none</li> + <li>Mobile Platform Applications: none</li> + <li>Product Listings: updated_at_min</li> + <li>Resource Feedbacks: none</li> + </ul> + </li> + <li>Shipping and Fulfillments + <ul> + <li>Carrier Services: none</li> + </ul> + </li> + <li>Store Properties + <ul> + <li>Countries: none</li> + <li>Currencies: none</li> + <li>Policies: none</li> + <li>Shipping Zones: updated_at_min</li> + <li>Shop: none</li> + </ul> + </li> + <li>Tender Transactions + <ul> + <li>Tender Transactions: processed_at_min</li> + </ul> + </li> +</ul> + +Specific trap type can set in case of Enterprise Specific generic trap type is chosen. Review Comment: I don't think we need trap-related information here. ########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/resources/docs/org.apache.nifi.shopify.processors.GetShopify/additionalDetails.html: ########## @@ -0,0 +1,148 @@ +<!DOCTYPE html> Review Comment: Should be under `org.apache.nifi.processors.shopify.GetShopify` (`shopify` is missing). ########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/java/org/apache/nifi/processors/shopify/model/ResourceDirectory.java: ########## @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.shopify.model; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +public class ResourceDirectory { + + private static final boolean PAGINATION_SUPPORTED = true; + private static final boolean PAGINATION_NOT_SUPPORTED = false; + + public static final List<ShopifyResource> CUSTOMER_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "customers", + "Customers", + "The Customer resource stores information about a shop's customers, such as their contact details," + + " their order history, and whether they've agreed to receive email marketing.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "customer_saved_searches", + "Customer Saved Searches", + "A customer saved search is a search query that represents a group of customers defined by the shop owner.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ))); + public static final List<ShopifyResource> DISCOUNT_RESOURCES = + Collections.singletonList(ShopifyResource.newInstance( + "price_rules", + "Price Rules", + "The PriceRule resource can be used to get discounts using conditions", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + )); + public static final List<ShopifyResource> INVENTORY_RESOURCES = + Collections.singletonList(ShopifyResource.newInstance( + "locations", + "Locations", + "A location represents a geographical location where your stores, pop-up stores, headquarters and warehouses exist.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + )); + public static final List<ShopifyResource> ONLINE_STORE_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "blogs", + "Blogs", + "Shopify shops come with a built-in blogging engine, allowing a shop to have one or more blogs.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "comments", + "Comments", + "A comment is a reader's response to an article in a blog.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "pages", + "Pages", + "Shopify stores come with a tool for creating basic HTML web pages.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "redirects", + "Redirects", + "A redirect causes a visitor on a specific path on the shop's site to be automatically sent to a different location.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "script_tags", + "Script Tags", + "The ScriptTag resource represents remote JavaScript code that is loaded into the pages of a shop's storefront or the order status page of checkout.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "themes", + "Themes", + "A theme controls the look and feel of a Shopify online store.", + IncrementalLoadingParameter.NONE, + PAGINATION_NOT_SUPPORTED + ))); + public static final List<ShopifyResource> ORDER_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "checkouts", + "Abandoned Checkouts", + "A checkout is considered abandoned after the customer has added contact information, but before the customer has completed their purchase.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "draft_orders", + "Draft Orders", + "Merchants can use draft orders to create orders on behalf of their customers.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "orders", + "Orders", + "An order is a customer's request to purchase one or more products from a shop.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ))); + public static final List<ShopifyResource> PRODUCT_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "collects", + "Collects", + "Collects are meant for managing the relationship between products and custom collections.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "custom_collections", + "Custom Collections", + "A custom collection is a grouping of products that a merchant can create to make their store easier to browse. ", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "products", + "Products", + "Get products in a merchant's store ", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "smart_collections", + "Smart Collections", + "A smart collection is a grouping of products defined by rules that are set by the merchant.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ))); + public static final List<ShopifyResource> SALES_CHANNEL_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "collection_listings", + "Collection Listings", + "A CollectionListing resource represents a product collection that a merchant has made available to your sales channel.", + IncrementalLoadingParameter.NONE, + PAGINATION_SUPPORTED + ), ShopifyResource.newInstance( + "product_listings", + "Product Listings", + "A ProductListing resource represents a Product which is available to your sales channel.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_SUPPORTED + ))); + public static final List<ShopifyResource> STORE_PROPERTY_RESOURCES = + Collections.unmodifiableList(Arrays.asList(ShopifyResource.newInstance( + "countries", + "Countries", + "The Country resource represents the tax rates applied to orders from the different countries where a shop sells its products.", + IncrementalLoadingParameter.NONE, + PAGINATION_NOT_SUPPORTED + ), ShopifyResource.newInstance( + "currencies", + "Currencies", + "Merchants who use Shopify Payments can allow customers to pay in their local currency on the online store.", + IncrementalLoadingParameter.NONE, + PAGINATION_NOT_SUPPORTED + ), ShopifyResource.newInstance( + "policies", + "Policies", + "Policy resource can be used to access the policies that a merchant has configured for their shop, such as their refund and privacy policies.", + IncrementalLoadingParameter.NONE, + PAGINATION_NOT_SUPPORTED + ), ShopifyResource.newInstance( + "shipping_zones", + "Shipping Zones", + "ShippingZone resource can be used to view shipping zones and their countries, provinces, and shipping rates.", + IncrementalLoadingParameter.UPDATED_AT, + PAGINATION_NOT_SUPPORTED + ), ShopifyResource.newInstance( Review Comment: When I tested, `Shop` returned a zip and the current implementation can only handle JSON responses. ########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/java/org/apache/nifi/processors/shopify/model/IncrementalLoadingParameter.java: ########## @@ -0,0 +1,25 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.shopify.model; + +public enum IncrementalLoadingParameter { + CREATED_AT, + UPDATED_AT, + PUBLISHED_AT, + PROCESSED_AT, Review Comment: ```suggestion ``` They are not used. ########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/resources/docs/org.apache.nifi.shopify.processors.GetShopify/additionalDetails.html: ########## @@ -0,0 +1,148 @@ +<!DOCTYPE html> +<html lang="en"> +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +<head> + <meta charset="utf-8"/> + <title>GetShopify</title> + <link rel="stylesheet" href="../../../../../css/component-usage.css" type="text/css"/> +</head> + +<body> +<h2>Setting Up a Custom App</h2> +<p> + Follow the <a href="https://help.shopify.com/en/manual/apps/custom-apps";>Shopify tutorial</a> to enable and create + private apps, set API Scopes and generate API tokens. +</p> +<h2>Incremental Loading</h2> +<p> + Some resources can be processed incrementally by NiFi. This means that only resources created or modified after the + last run time of the processor are displayed. The processor state can be reset in the context menu. The following + list shows which date-time fields are incremented for which resources. +<ul> + <li>Access + <ul> + <li>Access Scope: none</li> Review Comment: Wouldn't it be better to just leave out the ones that don't support incremental loading? ########## nifi-nar-bundles/nifi-shopify-bundle/nifi-shopify-processors/src/main/java/org/apache/nifi/processors/shopify/GetShopify.java: ########## @@ -0,0 +1,377 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.shopify; + +import com.fasterxml.jackson.core.JsonEncoding; +import com.fasterxml.jackson.core.JsonFactory; +import com.fasterxml.jackson.core.JsonGenerator; +import com.fasterxml.jackson.core.JsonParser; +import com.fasterxml.jackson.core.JsonToken; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.PrimaryNodeOnly; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.components.state.StateMap; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.flowfile.FlowFile; +import org.apache.nifi.processor.AbstractProcessor; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.ProcessSession; +import org.apache.nifi.processor.Relationship; +import org.apache.nifi.processor.exception.ProcessException; +import org.apache.nifi.processor.io.OutputStreamCallback; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processors.shopify.model.IncrementalLoadingParameter; +import org.apache.nifi.processors.shopify.model.ResourceType; +import org.apache.nifi.processors.shopify.model.ShopifyResource; +import org.apache.nifi.processors.shopify.rest.ShopifyRestService; +import org.apache.nifi.web.client.api.HttpResponseEntity; +import org.apache.nifi.web.client.api.HttpResponseStatus; +import org.apache.nifi.web.client.api.HttpUriBuilder; +import org.apache.nifi.web.client.api.WebClientService; +import org.apache.nifi.web.client.provider.api.WebClientServiceProvider; + +import java.io.IOException; +import java.net.URISyntaxException; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.EnumMap; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import java.util.stream.Collectors; + +@PrimaryNodeOnly +@TriggerSerially +@InputRequirement(Requirement.INPUT_FORBIDDEN) +@Tags({"shopify"}) +@Stateful(scopes = Scope.CLUSTER, description = + "For a few resources the processors support incremental loading. The list of the resources with the supported parameters" + + " can be found in additional details.") Review Comment: ```suggestion "For a few resources the processor supports incremental loading. The list of the resources with the supported parameters" + " can be found in the additional details.") ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
