mark juchems created PARQUET-1822:
-------------------------------------
Summary: Parquet without Hadoop dependencies
Key: PARQUET-1822
URL: https://issues.apache.org/jira/browse/PARQUET-1822
Project: Parquet
Issue Type: Improvement
Components: parquet-avro
Affects Versions: 1.11.0
Environment: Amazon Fargate (linux), Windows development box.
We are writing Parquet to be read by the Snowflake and Athena databases.
Reporter: mark juchems
I have been trying for weeks to create a parquet file from avro and write to S3
in Java. This has been incredibly frustrating and odd as Spark can do it
easily (I'm told).
I have assembled the correct jars through luck and diligence, but now I find
out that I have to have hadoop installed on my machine. I am currently
developing in Windows and it seems a dll and exe can fix that up but am
wondering about Linus as the code will eventually run in Fargate on AWS.
*Why do I need external dependencies and not pure java?*
The thing really is how utterly complex all this is. I would like to create an
avro file and convert it to Parquet and write it to S3, but I am trapped in
"ParquetWriter" hell!
*Why can't I get a normal OutputStream and write it wherever I want?*
I have scoured the web for examples and there are a few but we really need some
documentation on this stuff. I understand that there may be reasons for all
this but I can't find them on the web anywhere. Any help? Can't we get the
"SimpleParquet" jar that does this:
ParquetWriter writer =
AvroParquetWriter.<GenericData.Record>builder(outputStream)
.withSchema(avroSchema)
.withConf(conf)
.withCompressionCodec(CompressionCodecName.SNAPPY)
.withWriteMode(Mode.OVERWRITE)//probably not good for prod. (overwrites files).
.build();
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
