[
https://issues.apache.org/jira/browse/NIFI-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15279858#comment-15279858
]
ASF GitHub Bot commented on NIFI-1280:
--------------------------------------
Github user zinking commented on a diff in the pull request:
https://github.com/apache/nifi/pull/420#discussion_r62818694
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestFilterCSVColumns.java
---
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.standard;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.List;
+
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.MockFlowFile;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestFilterCSVColumns {
+
+ private static final Logger LOGGER;
+
+ static {
+ System.setProperty("org.slf4j.simpleLogger.defaultLogLevel",
"info");
+ System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
+ System.setProperty("org.slf4j.simpleLogger.log.nifi.io.nio",
"debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.FilterCSVColumns",
"debug");
+
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.standard.TestFilterCSVColumns",
"debug");
+ LOGGER = LoggerFactory.getLogger(TestFilterCSVColumns.class);
+ }
+
+ @Test
+ public void testTransformSimple() throws InitializationException,
IOException, SQLException {
+ String sql = "select first_name, last_name, company_name, address,
city from CSV.A where city='New York'";
+
+ Path inpath =
Paths.get("src/test/resources/TestFilterCSVColumns/US500.csv");
+ InputStream in = new FileInputStream(inpath.toFile());
+
+ ResultSet resultSet = FilterCSVColumns.transform(in, sql);
+
+ int nrofColumns = resultSet.getMetaData().getColumnCount();
+
+ for (int i = 1; i <= nrofColumns; i++) {
+ System.out.print(resultSet.getMetaData().getColumnLabel(i) + "
");
+ }
+ System.out.println();
+
+ while (resultSet.next()) {
+ for (int i = 1; i <= nrofColumns; i++) {
+ System.out.print(resultSet.getString(i)+ " ");
+ }
+ System.out.println();
+ }
+ }
+
+ @Test
+ public void testTransformCalc() throws InitializationException,
IOException, SQLException {
+ String sql = "select ID, AMOUNT1+AMOUNT2+AMOUNT3 as TOTAL from
CSV.A where ID=100";
--- End diff --
what's the big difference between File and InputStream? I thought all
conversation is about streaming query.
> Create FilterCSVColumns Processor
> ---------------------------------
>
> Key: NIFI-1280
> URL: https://issues.apache.org/jira/browse/NIFI-1280
> Project: Apache NiFi
> Issue Type: Task
> Components: Extensions
> Reporter: Mark Payne
> Assignee: Toivo Adams
>
> We should have a Processor that allows users to easily filter out specific
> columns from CSV data. For instance, a user would configure two different
> properties: "Columns of Interest" (a comma-separated list of column indexes)
> and "Filtering Strategy" (Keep Only These Columns, Remove Only These Columns).
> We can do this today with ReplaceText, but it is far more difficult than it
> would be with this Processor, as the user has to use Regular Expressions,
> etc. with ReplaceText.
> Eventually a Custom UI could even be built that allows a user to upload a
> Sample CSV and choose which columns from there, similar to the way that Excel
> works when importing CSV by dragging and selecting the desired columns? That
> would certainly be a larger undertaking and would not need to be done for an
> initial implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
