This wasn't terribly complicated to handle using a ByteArrayOutputStream,
once I fixed the callers to not closeQuietly()...
Here's the calling code, that has to return an OutputStream:
public OutputStream getOutputStream(String containerName, String
resourceName) throws IOException {
return new
JcloudsObjectWritingByteArrayOutputStream(this.blobStoreContext.getBlobStore(),
containerName, resourceName);
}
And here's what JcloudsObjectWritingByteArrayOutputStream looks like (it's
a bit long, so I put it in a gist):
https://gist.github.com/skingsland/d2341cd52cd36c6cbb6f
It's working ok with filesystem and in-memory object stores, but I'm
running into some (apparently-unrelated) errors with the particular object
store I'm trying to use (Ceph via S3 API). I'll save those for another
email...
I'd love to hear feedback on this approach. And thanks everyone for your
help!
*Steve Kingsland*
Senior Software Engineer
*Opower * <http://www.opower.com/>
*We’re hiring! See jobs here <http://www.opower.com/careers> *
On Tue, Aug 5, 2014 at 5:52 PM, Adrian Cole <[email protected]> wrote:
> jclouds currently doesn't have a direct path to the outputstream (or
> channel), and even if it did, things mentioned by gaul would still be
> true (ex. may need content-length up front).
>
> jclouds doesn't have a direct path to becoming netty, so I wouldn't
> get too excited about full-bore async. Chunking, multipart, etc. over
> streams are very possible, though.
>
> Personally, I'd recommend using something like okio buffer (or some
> other buffer) and making that easier to work with (if it isn't
> already). https://github.com/square/okio
>
> Hope this helps,
> -A
>
> On Tue, Aug 5, 2014 at 2:33 PM, Zack Shoylev <[email protected]>
> wrote:
> > With buffered streams, for example, close() causes buffers to be flushed
> > (which is technically what you are doing).
> > So yes, you can get some serious exceptions when closing.
> >
> > ________________________________
> > From: Steve Kingsland [[email protected]]
> > Sent: Tuesday, August 05, 2014 9:06 AM
> >
> > To: [email protected]
> > Subject: Re: How to write a Blob using an OutputStream?
> >
> > org.apache.commons.io.output.ByteArrayOutputStream sounds like a nice
> > improvement over java.io.ByteArrayOutputStream (at least for my
> purposes),
> > thanks Zack!
> >
> > The problem I'm running into is actually with the caller's
> > Closeables.closeQuietly(documentOutputStream); call. That catches any
> > IOException that's thrown from close() and logs it, instead of throwing
> it.
> > That won't work for me, since I won't know if there was an error writing
> to
> > the blob store until close() is called on my OutputStream. I can of
> course
> > change the caller to use different error-handling for closing the stream,
> > but it makes me wonder if using the close() method to upload the blob is
> the
> > right approach. If you're given an OutputStream to write to, you'd expect
> > the real errors to come from the write() methods, and not the close()
> > method, right?
> >
> >
> > Steve Kingsland
> >
> >
> > Senior Software Engineer
> >
> > Opower
> >
> >
> > We’re hiring! See jobs here
> >
> >
> >
> > On Tue, Aug 5, 2014 at 7:21 AM, Zack Shoylev <[email protected]
> >
> > wrote:
> >>
> >> Your code seems fine. I have used
> >>
> http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/output/ByteArrayOutputStream.html
> >> in the past to convert between stream types, but it seems like it
> doesn't
> >> match your case very well.
> >>
> >> Note you might have to do writeBytesToBlob() before super.close(), but
> you
> >> can test that.
> >>
> >> Let us know how it turns out!
> >> ________________________________
> >> From: Steve Kingsland [[email protected]]
> >> Sent: Monday, August 04, 2014 9:22 PM
> >> To: [email protected]
> >> Subject: Re: How to write a Blob using an OutputStream?
> >>
> >> OK, then it appears that my calling code (which would be difficult and
> >> risky to change) is incompatible with jclouds' BlobStore API: my caller
> >> wants to obtain an OutputStream for writing to the blob store, and
> jclouds
> >> wants to obtain an InputStream for reading the blob's content that
> should be
> >> written. Therefore, my only solution is to buffer the blob data, either
> in
> >> memory or on disk, before uploading it to the blob store.
> >>
> >> Given that the documents I'm trying to write to the blob store will
> >> generally be small (1KB to 1MB), I'm going with a simple approach, for
> >> providing my caller with an OutputStream that they can use to write the
> >> blob's payload:
> >>
> >> class BlobWritingByteArrayOutputStream extends
> >> java.io.ByteArrayOutputStream {
> >>
> >> // these are all set in the constructor
> >> private BlobStore blobStore;
> >> private String containerName, blobName;
> >>
> >> // the client will have to call this when he's finished writing, so
> >> this is our chance to upload the blob,
> >> // now that we have the full payload in memory
> >> @Override
> >> public void close() throws IOException {
> >> super.close();
> >>
> >> writeBytesToBlob();
> >> }
> >>
> >> private void writeBytesToBlob() {
> >> byte[] payload = toByteArray();
> >>
> >> Blob blob = blobStore.blobBuilder(blobName)
> >> .payload(payload)
> >> .contentLength(payload.size)
> >> .build();
> >> blobStore.putBlob(containerName, blob);
> >> }
> >> }
> >>
> >> Aside from the weird inversion of control going on and the requirement
> >> that close() be called, I think something simple like this - to buffer
> the
> >> bytes being written before uploading them to the blob store - might
> work for
> >> me.
> >>
> >> Thoughts?
> >>
> >>
> >>
> >>
> >> Steve Kingsland
> >>
> >>
> >> Senior Software Engineer
> >>
> >> Opower
> >>
> >>
> >> We’re hiring! See jobs here
> >>
> >>
> >>
> >> On Mon, Aug 4, 2014 at 9:05 PM, Andrew Gaul <[email protected]> wrote:
> >>>
> >>> On Mon, Aug 04, 2014 at 08:46:37PM -0400, Steve Kingsland wrote:
> >>> > Here is Kevin's example using PipedInputStream and PipedOutputStream:
> >>> > https://groups.google.com/d/msg/jclouds/F2pCt9i7TSg/AUF4AqOO0TMJ
> >>> >
> >>> > I don't have the need to use different threads, though, so instead
> I'd
> >>> > do
> >>> > something like this?
> >>>
> >>> This will not work; putBlob blocks until the operation completes.
> >>> Further you must use PipedInputStream/PipedOutputStream with separate
> >>> threads to avoid deadlock, as its Javadoc states:
> >>>
> >>> http://docs.oracle.com/javase/7/docs/api/java/io/PipedInputStream.html
> >>>
> >>> Unfortunately jclouds has poor support for asynchronous operations and
> >>> you can really only fake the desired behavior with various InputStream.
> >>> I strongly recommend trying to cast your solution into some kind of
> >>> ByteSource or InputStream.
> >>>
> >>> > And then when close() or flush() is called on the returned
> >>> > OutputStream,
> >>> > the blob is uploaded like magic? Is it OK that I'm not setting the
> >>> > content
> >>> > length?
> >>>
> >>> Some blobstores, specifically Amazon S3, require a content length,
> while
> >>> others such as OpenStack Swift do not.
> >>>
> >>> --
> >>> Andrew Gaul
> >>> http://gaul.org/
> >>
> >>
> >
>
