ASF GitHub Bot commented on NIFI-2897:

GitHub user mattyb149 opened a pull request:


    NIFI-2897: Fixed SelectHiveQL for CSV output of complex types

    The lack of unit tests is due to the fact that we use Derby. When we get 
integration tests in place using MiniHS2, we can automate this kind of testing. 
In the meantime, here are some helpful HiveDDL statements you can use to 
populate a table with the different types, along with some queries that will 
illustrate the problem and solution:
    Create a table with a field that is an array of structs:
    `create table arraytest (a ARRAY<STRUCT<name:STRING, age:INT>>)`
    Insert two elements (this assumes you have a table named "dummy" with at 
least one row):
    `insert into arraytest select 
from dummy limit 1`
    Select the whole array (as one field):
    `SELECT * FROM arraytest`
    Select the structs as individual columns:
    `SELECT a[0], a[1] FROM arraytest`
    Select a struct and a field from the other struct:
    `SELECT a[0], a[1].name FROM arraytest`
    The first SELECT should generate a single field/column in CSV, the other 
two should have two columns, all properly escaped to give valid CSV. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattyb149/nifi NIFI-2897

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1132
commit 58c97651e18f2880b5aa291a1e41b2ab7e14b0f1
Author: Matt Burgess <mattyb...@apache.org>
Date:   2016-10-13T17:27:49Z

    NIFI-2897: Fixed SelectHiveQL for CSV output of complex types


> SelectHiveQL CSV output cannot be used with comma delimiter in the presence 
> of maps or structs
> ----------------------------------------------------------------------------------------------
>                 Key: NIFI-2897
>                 URL: https://issues.apache.org/jira/browse/NIFI-2897
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
> 1. Created a flow GetFIle -> SelectHiveQL -> putFile
> 2. Set the output format in SelectHiveQL as CSV
> 3. Select a struct or a map column
> Result : Unable to run an object mapper or split on ',' as the 
> maps/struct/list contains comma in them.

This message was sent by Atlassian JIRA

Reply via email to