[
https://issues.apache.org/jira/browse/AVRO-3472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17512207#comment-17512207
]
Jacob Tolar commented on AVRO-3472:
-----------------------------------
DataFileWriter has
{code:java}
create(Schema schema, OutputStream outs)
{code}
Did you try that (for example, with a {{{}ByteArrayOutputStream{}}})?
[https://avro.apache.org/docs/1.11.0/api/java/org/apache/avro/file/DataFileWriter.html#create-org.apache.avro.Schema-java.io.OutputStream-]
> Writing to an object store (without intermediate file)
> ------------------------------------------------------
>
> Key: AVRO-3472
> URL: https://issues.apache.org/jira/browse/AVRO-3472
> Project: Apache Avro
> Issue Type: New Feature
> Components: java
> Environment: OpenJDK version 11 on Ubuntu
> Reporter: Kevin A Shaw
> Priority: Minor
> Labels: Java11
>
> We need to write a large number files for a production system to an Object
> Store (AWS S3 in this case) which cannot be represented by a java.io.File
> object. We are currently using a DataFileWriter() to create a file in a /tmp
> directory, putting it to S3 and then deleting the file. This is becoming
> cumbersome and just seems inefficient. Is there a way to instead create the
> file using an in-memory object, instead of writing to disk?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
