[
https://issues.apache.org/jira/browse/NIFI-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
archon gum updated NIFI-6435:
-----------------------------
Description:
h1. From AVRO
data
{code:java}
{
"date_type": 17897, // 2019-01-01
"timestamp_type": 1546300800000 // 2019-01-01 00:00:00 +0000
}
{code}
schema
{code:java}
{
"name": "test_types",
"type": "record",
"fields": [
{
"name": "date_type",
"type": "int",
"logicalType": "date"
},
{
"name": "timestamp_type",
"type": "long",
"logicalType": "timestamp-millis" // or others
}
]
}
{code}
or schema
{code:java}
{
"name": "test_types",
"type": "record",
"fields": [
{
"name": "date_type",
"type": {
"type": "int",
"logicalType": "date"
}
},
{
"name": "timestamp_type",
"type": {
"type": "long",
"logicalType": "timestamp-millis" // or others
}
}
]
}
{code}
h1. To ORC
schema should be
{code:java}
struct<date_type:date, timestamp_type:timestamp>
{code}
---
Update: 2019-07-15
h1. PutORC
[https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/orc/PutORC.java]
This processor can do the trick.
*PutORC*: Read from RecordReader(eg JsonReader, AvroReader), convert to ORC and
put the ORC file to HDFS.
*PutORC* can convert Avro logical type such like:
>From Avro schema
{code:java}
{
"name": "test_types",
"type": "record",
"fields": [
{
"name": "date_type",
"type": {
"type": "int",
"logicalType": "date"
}
},
{
"name": "timestamp_type",
"type": {
"type": "long",
"logicalType": "timestamp-millis"
}
}
]
}{code}
And the result ORC schema would be:
{code:java}
struct<date_type:date, timestamp_type:timestamp>{code}
h1. Question?
The *PutORC* still in development? Because *nifi-hive3-processors* module is
not including in build.(At least in NiFi-1.9.2)
was:
h1. From AVRO
data
{code:java}
{
"date_type": 17897, // 2019-01-01
"timestamp_type": 1546300800000 // 2019-01-01 00:00:00 +0000
}
{code}
schema
{code:java}
{
"name": "test_types",
"type": "record",
"fields": [
{
"name": "date_type",
"type": "int",
"logicalType": "date"
},
{
"name": "timestamp_type",
"type": "long",
"logicalType": "timestamp-millis" // or others
}
]
}
{code}
or schema
{code:java}
{
"name": "test_types",
"type": "record",
"fields": [
{
"name": "date_type",
"type": {
"type": "int",
"logicalType": "date"
}
},
{
"name": "timestamp_type",
"type": {
"type": "long",
"logicalType": "timestamp-millis" // or others
}
}
]
}
{code}
h1. To ORC
schema should be
{code:java}
struct<date_type:date, timestamp_type:timestamp>
{code}
> ConvertAvroToORC Should support Date and Timestamp type
> -------------------------------------------------------
>
> Key: NIFI-6435
> URL: https://issues.apache.org/jira/browse/NIFI-6435
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: archon gum
> Priority: Major
>
> h1. From AVRO
> data
>
> {code:java}
> {
> "date_type": 17897, // 2019-01-01
> "timestamp_type": 1546300800000 // 2019-01-01 00:00:00 +0000
> }
> {code}
>
> schema
>
> {code:java}
> {
> "name": "test_types",
> "type": "record",
> "fields": [
> {
> "name": "date_type",
> "type": "int",
> "logicalType": "date"
> },
> {
> "name": "timestamp_type",
> "type": "long",
> "logicalType": "timestamp-millis" // or others
> }
> ]
> }
> {code}
> or schema
>
> {code:java}
> {
> "name": "test_types",
> "type": "record",
> "fields": [
> {
> "name": "date_type",
> "type": {
> "type": "int",
> "logicalType": "date"
> }
> },
> {
> "name": "timestamp_type",
> "type": {
> "type": "long",
> "logicalType": "timestamp-millis" // or others
> }
> }
> ]
> }
> {code}
>
> h1. To ORC
> schema should be
> {code:java}
> struct<date_type:date, timestamp_type:timestamp>
> {code}
>
> ---
>
> Update: 2019-07-15
>
> h1. PutORC
> [https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/nifi/processors/orc/PutORC.java]
>
> This processor can do the trick.
> *PutORC*: Read from RecordReader(eg JsonReader, AvroReader), convert to ORC
> and put the ORC file to HDFS.
> *PutORC* can convert Avro logical type such like:
> From Avro schema
> {code:java}
> {
> "name": "test_types",
> "type": "record",
> "fields": [
> {
> "name": "date_type",
> "type": {
> "type": "int",
> "logicalType": "date"
> }
> },
> {
> "name": "timestamp_type",
> "type": {
> "type": "long",
> "logicalType": "timestamp-millis"
> }
> }
> ]
> }{code}
> And the result ORC schema would be:
> {code:java}
> struct<date_type:date, timestamp_type:timestamp>{code}
>
> h1. Question?
> The *PutORC* still in development? Because *nifi-hive3-processors* module is
> not including in build.(At least in NiFi-1.9.2)
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
