[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Christian Tramnitz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964513#comment-15964513
 ] 

Christian Tramnitz commented on ORC-150:


It seems there is a problem with schema detection when floats are used. A 
sample json like
{noformat}
{"float":5.2}
{noformat}
will result in:
{noformat}
$ java -jar orc-tools-1.4.0-SNAPSHOT-uber.jar convert sample_message.json -o 
sample.orc
Scanning sample_message.json for schema
Exception in thread "main" java.lang.IllegalArgumentException: precision 2 is 
out of range 1 .. 10
at 
org.apache.orc.TypeDescription.withPrecision(TypeDescription.java:410)
at org.apache.orc.tools.json.NumericType.getSchema(NumericType.java:105)
at org.apache.orc.tools.json.StructType.getSchema(StructType.java:110)
at 
org.apache.orc.tools.json.JsonSchemaFinder.getSchema(JsonSchemaFinder.java:257)
at 
org.apache.orc.tools.convert.ConvertTool.computeSchema(ConvertTool.java:48)
at org.apache.orc.tools.convert.ConvertTool.main(ConvertTool.java:58)
at org.apache.orc.tools.Driver.main(Driver.java:112)
{noformat}

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: ORC
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.4.0
>
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Christian Tramnitz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964467#comment-15964467
 ] 

Christian Tramnitz commented on ORC-150:


Is this expected to work in combination with ORC-111 (files on hdfs) or only 
local files?

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: ORC
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.4.0
>
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899261#comment-15899261
 ] 

Lefty Leverenz commented on ORC-150:


Thanks Owen, here's the link:

* [Java ORC Tools | https://orc.apache.org/docs/tools.html#java-orc-tools]

By the way, the formatting in paragraph 2 is broken (subcommands).

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: ORC
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.4.0
>
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888560#comment-15888560
 ] 

Owen O'Malley commented on ORC-150:
---

[~leftylev] Ok, I made a pull request at https://github.com/apache/orc/pull/98 
for the updated page.

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.4.0
>
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Lefty Leverenz (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887216#comment-15887216
 ] 

Lefty Leverenz commented on ORC-150:


Should this be documented in the wiki?

* [ORC Tools | http://orc.apache.org/docs/tools.html]

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.4.0
>
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886642#comment-15886642
 ] 

ASF GitHub Bot commented on ORC-150:


Github user asfgit closed the pull request at:

https://github.com/apache/orc/pull/95


> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875354#comment-15875354
 ] 

ASF GitHub Bot commented on ORC-150:


GitHub user omalley opened a pull request:

https://github.com/apache/orc/pull/95

ORC-150. Add tools for finding schema from JSON and converting into ORC

Here is a first pass for making a JSON to ORC converter along with the JSON 
schema discovery.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omalley/orc orc-150

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/orc/pull/95.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #95


commit ffc3f46a5d55287cc2b405fda402d00ce3936c53
Author: Owen O'Malley 
Date:   2017-02-21T04:37:18Z

ORC-150. Add tools for finding schema from JSON documents and converting 
JSON
into ORC files.




> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875095#comment-15875095
 ] 

Owen O'Malley commented on ORC-150:
---

Actually, I have a JSON schema detector too, so I guess we could make the 
schema optional.

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (ORC-150) Add tool to convert from JSON to ORC

[Ben Johnson (JIRA)]

[ 
https://issues.apache.org/jira/browse/ORC-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874879#comment-15874879
 ] 

Ben Johnson commented on ORC-150:
-

Would the schema need to be specified, or could it be automatically determined 
from the JSON structure? If it is automatic, it would be cleaner to separate 
them, allowing someone to use them independently if desired.

> Add tool to convert from JSON to ORC
> 
>
> Key: ORC-150
> URL: https://issues.apache.org/jira/browse/ORC-150
> Project: Orc
>  Issue Type: New Feature
>Reporter: Owen O'Malley
>
> We should have a tool to convert from JSON data to ORC. Something like
> % java -jar orc-tools-*uber.jar create "struct" *.json



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)