[
https://issues.apache.org/jira/browse/PARQUET-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852595#comment-17852595
]
Ryan Rupp commented on PARQUET-1822:
------------------------------------
[~wgtmac] thanks, do you have any sample code by any chance? If I use
`hadoop-client-api` as a dependency I can get further along (compiles etc.) but
it does then result in an exception downstream when trying to go from
ParquetConfiguration -> Configuration (Hadoop). It looks like because
`hadoop-client-api` does some shading but it's failing to classload a woodstox
class that it's expecting to be shaded but isn't:
{code:java}
java.lang.NoClassDefFoundError:
org/apache/hadoop/shaded/com/ctc/wstx/io/InputBootstrapper
at
org.apache.parquet.hadoop.util.ConfigurationUtil.createHadoopConfiguration(ConfigurationUtil.java:60)
at
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:929)
// test code
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.shaded.com.ctc.wstx.io.InputBootstrapper
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 74 more {code}
I'm guessing because this dependency is really just meant for API users and at
runtime the "hadoop-client" with all relevant transitive dependencies would be
in place? I'm actually not familiar when you would use hadoop-client-api vs
hadoop-client.
> Parquet without Hadoop dependencies
> -----------------------------------
>
> Key: PARQUET-1822
> URL: https://issues.apache.org/jira/browse/PARQUET-1822
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-avro
> Affects Versions: 1.11.0
> Environment: Amazon Fargate (linux), Windows development box.
> We are writing Parquet to be read by the Snowflake and Athena databases.
> Reporter: mark juchems
> Assignee: Atour Mousavi Gourabi
> Priority: Minor
> Labels: documentation, newbie
> Fix For: 1.14.0
>
>
> I have been trying for weeks to create a parquet file from avro and write to
> S3 in Java. This has been incredibly frustrating and odd as Spark can do it
> easily (I'm told).
> I have assembled the correct jars through luck and diligence, but now I find
> out that I have to have hadoop installed on my machine. I am currently
> developing in Windows and it seems a dll and exe can fix that up but am
> wondering about Linus as the code will eventually run in Fargate on AWS.
> *Why do I need external dependencies and not pure java?*
> The thing really is how utterly complex all this is. I would like to create
> an avro file and convert it to Parquet and write it to S3, but I am trapped
> in "ParquetWriter" hell!
> *Why can't I get a normal OutputStream and write it wherever I want?*
> I have scoured the web for examples and there are a few but we really need
> some documentation on this stuff. I understand that there may be reasons for
> all this but I can't find them on the web anywhere. Any help? Can't we get
> the "SimpleParquet" jar that does this:
>
> ParquetWriter writer =
> AvroParquetWriter.<GenericData.Record>builder(outputStream)
> .withSchema(avroSchema)
> .withConf(conf)
> .withCompressionCodec(CompressionCodecName.SNAPPY)
> .withWriteMode(Mode.OVERWRITE)//probably not good for prod. (overwrites
> files).
> .build();
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
