Indeed. FYI & for anyone coming on this thread, below is running code (in the
DevServer),
invoked from a test page via DWR, and returning the String result to be
displayed in an alert;
the result is:
startBytesSize = 1160921
bytesReadSize = 1160921
Code:
public static String doGCSTest()
throws IOException
{
String result = "";
String testPDFFile = "IRAD1-2.pdf"; // a 1.2MB PDF file accessible
as a resource
String testPDFPath = "samples/" + testPDFFile;
byte[] startBytes = CaptureWARSamples.getBytesFromWARFile(testPDFPath);
int startBytesSize = startBytes.length;
result += "startBytesSize = " + startBytesSize;
String BUCKETNAME = "formrunnerbucket-r7yh23nb2";
final GcsService gcsService =
GcsServiceFactory.createGcsService(new RetryParams.Builder()
.initialRetryDelayMillis(10)
.retryMaxAttempts(10)
.totalRetryPeriodMillis(15000)
.build()) ;
String gcsFilename = "TEST-IRAD1-2pdf";
GcsFilename fullGcsFilename = new GcsFilename(BUCKETNAME, gcsFilename);
GcsOutputChannel outputChannel =
gcsService.createOrReplace(fullGcsFilename,
GcsFileOptions.getDefaultInstance());
outputChannel.write(ByteBuffer.wrap(startBytes));
outputChannel.close();
int fileSize = (int)
gcsService.getMetadata(fullGcsFilename).getLength();
ByteBuffer byteBufferResult = ByteBuffer.allocate(fileSize);
GcsInputChannel readChannel =
gcsService.openReadChannel(fullGcsFilename, 0);
try {
readChannel.read(byteBufferResult);
} finally {
readChannel.close();
}
byte[] bytesRead = byteBufferResult.array();
int bytesReadSize = bytesRead.length;
result += "\nbytesReadSize = " + bytesReadSize;
return result;
}
--Ken
On Nov 21, 2013, at 7:05 PM, Tom Kaitchuck wrote:
> Correct. It does not have to be directly in the servlet. For example App
> Engine MapReduce: https://code.google.com/p/appengine-mapreduce/
> uses the GCS client to write out data, but it is many levels removed from a
> servlet.
>
>
> On Thu, Nov 21, 2013 at 4:44 PM, Ken Bowen <[email protected]> wrote:
> Hi Tom,
>
> Re:
> > you don't need to use the LocalServiceTestHelper
>
> Understood. I ran LocalExample once when I began, just to follow the
> documentation.
>
> The original description I posted is from a running app on the DevServer, no
> TestHelper involved. In fact, the test jars are not even in the project.
>
> After posting, I was reviewing LocalExample to make comparisons, and the
> phrase
>
> "...run locally as opposed to in a deployed servlet"
>
> caught my eye. So I wrote a version of what I posted as a servlet (grabs a
> 1.2MB PDF out of a resource, stores it with the GCS library, retrieves it,
> and writes out the number of bytes it got). That works fine.
>
> If you say this doesn't /have/ to be directly in a servlet, I'll dig in
> further to my original.
>
> Cheers,
> --Ken
>
> On Nov 21, 2013, at 5:07 PM, Tom Kaitchuck wrote:
>
> > I'm a bit confused by your statement.
> > If you want to run in the DevAppserver or in AppEngine then you don't need
> > to use the LocalServiceTestHelper at all. The LocalExample only does that
> > as a demo of how to run it as a local executable or if you want to use it
> > within Junit.
> > If you don't include such a line, like in GcsExampleServlet then it should
> > work as part of your deployed application.
> >
> >
> > On Thu, Nov 21, 2013 at 3:41 PM, Ken Bowen <[email protected]> wrote:
> > Thanks Tom.
> > I went through pretty much the same servlet exercise a couple of hours
> > before seeing your post this afternoon.
> > The problem was that I missed the point the the docs that the code /must/
> > run in a servlet.
> > (I'm not clear where it's stated, but I guessed from the comment about the
> > test harness in LocalExample.)
> > I assumed that it would be ok to run in a deployed app on AppEngine.
> > Thanks again,
> > --Ken
> >
> > On Nov 21, 2013, at 2:51 PM, Tom Kaitchuck wrote:
> >
> > > Using the 1.8.8 version of the SDK and depending on appengine-gcs-client
> > > 0.3.3, the following servlet based on your above example works:
> > >
> > > public class GcsTest extends HttpServlet {
> > > private void writeToFile(GcsService gcsService, GcsFilename
> > > fullGcsFilename, byte[] content)
> > > throws IOException {
> > > System.out.println("writeToFile:full=" + fullGcsFilename.toString());
> > > GcsOutputChannel outputChannel =
> > > gcsService.createOrReplace(fullGcsFilename,
> > > GcsFileOptions.getDefaultInstance());
> > > outputChannel.write(ByteBuffer.wrap(content));
> > > outputChannel.close();
> > > }
> > >
> > >
> > > private byte[] readFromFile(GcsService gcsService, GcsFilename
> > > fullGcsFilename)
> > > throws IOException {
> > > System.out.println("readFromFile:full=" + fullGcsFilename.toString());
> > > int fileSize = (int)
> > > gcsService.getMetadata(fullGcsFilename).getLength();
> > > ByteBuffer result = ByteBuffer.allocate(fileSize);
> > > GcsInputChannel readChannel =
> > > gcsService.openReadChannel(fullGcsFilename, 0);
> > > try {
> > > readChannel.read(result);
> > > } finally {
> > > readChannel.close();
> > > }
> > > return result.array();
> > > }
> > >
> > > @Override
> > > public void doGet(HttpServletRequest req, HttpServletResponse resp)
> > > throws IOException {
> > > GcsService gcsService = GcsServiceFactory.createGcsService();
> > > GcsFilename fullGcsFilename = new GcsFilename("Foo", "Bar");
> > > byte[] content = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};
> > > writeToFile(gcsService, fullGcsFilename, content);
> > > byte[] result = readFromFile(gcsService, fullGcsFilename);
> > > PrintWriter writer = resp.getWriter();
> > > writer.append(Arrays.toString(content));
> > > writer.append(Arrays.toString(result));
> > > }
> > > }
> > >
> > > I'm not really sure what could be different about your setup to cause
> > > that. Try creating a new project with the minimal possible dependencies
> > > and run the servlet above and see what happens.
> > >
> > >
> > >
> > > On Tue, Nov 19, 2013 at 12:57 PM, <[email protected]> wrote:
> > > I've encountered a problem using the Google Cloud Storage GCS Client API,
> > > running on the local development server. I'm trying to write the bytes
> > > from a PDF file, and then read them back.
> > >
> > > The code appears to write the (local fake)GCS file ok: (1) There appears
> > > to be an appropriate entry in the Development Console, and (2) there's a
> > > physical file in ~war/WEB-APP/appengine-generated (details below).
> > > However, when I attempt to read the bytes from the GCS file, it throws a
> > > FileNotFound exception when it attempts to get the metadata (filesize).
> > >
> > > First, here's the core code, versions of GCS client read/write, with my
> > > debugging stmts left in for reference:
> > >
> > > private void writeToFile(GcsFilename fullGcsFilename, byte[] content)
> > > throws IOException
> > > {
> > > System.out.println("writeToFile:full="+fullGcsFilename.toString());
> > > GcsOutputChannel outputChannel =
> > > gcsService.createOrReplace(fullGcsFilename,
> > > GcsFileOptions.getDefaultInstance());
> > > outputChannel.write(ByteBuffer.wrap(content));
> > > outputChannel.close();
> > > }
> > >
> > > private byte[] readFromFile(GcsFilename fullGcsFilename)
> > > throws IOException
> > > {
> > > System.out.println("readFromFile:full="+fullGcsFilename.toString());
> > > int fileSize = (int)
> > > gcsService.getMetadata(fullGcsFilename).getLength(); [*][Exception
> > > thrown here]
> > > ByteBuffer result = ByteBuffer.allocate(fileSize);
> > > GcsInputChannel readChannel =
> > > gcsService.openReadChannel(fullGcsFilename, 0);
> > > try {
> > > readChannel.read(result);
> > > } finally {
> > > readChannel.close();
> > > }
> > > return result.array();
> > > }
> > >
> > > Here's the debugging output (in/out filenames appear to be the same):
> > > ----
> > > writeToFile:full=GcsFilename(formrunnerbucket-r7yh23nb2,
> > > FA/MasterFormStore-6649846324789248)
> > >
> > > ----
> > > readFromFile:full=GcsFilename(formrunnerbucket-r7yh23nb2,
> > > FA/MasterFormStore-6649846324789248)
> > >
> > > ----
> > > Here's the observed results:
> > >
> > > IN ~war/WEB-APP/appengine-generated:
> > >
> > > -rw-r--r-- 1 ken staff 1679407 Nov 19 09:19
> > > encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4
> > > This is the expected PDF file, which can be opened (with Preview on a
> > > Mac).
> > >
> > > ----
> > > In the Development Console (http://localhost:8888/_ah/admin/datastore):
> > > In __GsFileInfo__ (in the empty Namespace)
> > >
> > > Key:
> > > ag5mb3JtcnVubmVyLWhyZHJ7CxIOX19Hc0ZpbGVJbmZvX18iZ2VuY29kZWRfZ3Nfa2V5OkwyZHpMMlp2Y20xeWRXNXVaWEppZFdOclpYUXRjamQ1YURJemJtSXlMMFpCTDAxaGMzUmxja1p2Y20xVGRHOXlaUzAyTmpRNU9EUTJNekkwTnpnNU1qUTQM
> > >
> > > ID/Name:
> > > encoded_gs_key:L2dzL2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyL0ZBL01hc3RlckZvcm1TdG9yZS02NjQ5ODQ2MzI0Nzg5MjQ4
> > >
> > > content_type: application/octet-stream
> > >
> > > filename:
> > > /gs/formrunnerbucket-r7yh23nb2/FA/MasterFormStore-6649846324789248
> > >
> > > size: 1679407
> > >
> > > [[The ID/Name appears to be identical to the filename appearing in
> > > ~war/WEB-APP/appengine-generated]]
> > >
> > > ---------------
> > > In the Development Console (http://localhost:8888/_ah/admin/datastore),
> > > in Namespace 5629499534213120 (where everything ran):
> > > In _ah_FakeCloudStorate_formrunnerbucket-r7yh23nb2:
> > >
> > > Key:
> > > ag5mb3JtcnVubmVyLWhyZHJZCxIwX2FoX0Zha2VDbG91ZFN0b3JhZ2VfX2Zvcm1ydW5uZXJidWNrZXQtcjd5aDIzbmIyIiNGQS9NYXN0ZXJGb3JtU3RvcmUtNjY0OTg0NjMyNDc4OTI0OAyiARA1NjI5NDk5NTM0MjEzMTIw
> > >
> > > ID/Name: FA/MasterFormStore-6649846324789248
> > >
> > > options: <Blob: 483 bytes>
> > >
> > > ---------------
> > > Here's the exception thrown in readFromFile above, at:
> > >
> > > int fileSize = (int)
> > > gcsService.getMetadata(fullGcsFilename).getLength(); [*][Exception
> > > thrown here]
> > >
> > > java.lang.RuntimeException: java.io.FileNotFoundException
> > > at
> > > com.formrunner.pdf.PDFMaker.makePDFfromFormInstance(PDFMaker.java:132)
> > > .........
> > > Caused by: java.io.FileNotFoundException
> > > at
> > > com.google.appengine.api.files.FileServiceImpl.translateException(FileServiceImpl.java:614)
> > > at
> > > com.google.appengine.api.files.FileServiceImpl.makeSyncCall(FileServiceImpl.java:593)
> > > at
> > > com.google.appengine.api.files.FileServiceImpl.stat(FileServiceImpl.java:383)
> > > at
> > > com.google.appengine.tools.cloudstorage.dev.LocalRawGcsService.getObjectMetadata(LocalRawGcsService.java:215)
> > > at
> > > com.google.appengine.tools.cloudstorage.GcsServiceImpl$2.run(GcsServiceImpl.java:73)
> > > at
> > > com.google.appengine.tools.cloudstorage.GcsServiceImpl$2.run(GcsServiceImpl.java:70)
> > > at
> > > com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:93)
> > > at
> > > com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:138)
> > > at
> > > com.google.appengine.tools.cloudstorage.GcsServiceImpl.getMetadata(GcsServiceImpl.java:70)
> > > at
> > > com.formrunner.data.GoogleCloudStorage.readFromFile(GoogleCloudStorage.java:62)
> > > at
> > > com.formrunner.data.GoogleCloudStorage.getBytesAt(GoogleCloudStorage.java:55)
> > > at
> > > com.formrunner.db.MasterFormStore.getContent(MasterFormStore.java:92)
> > > at com.formrunner.forms.FormUtils.getFormBlob(FormUtils.java:314)
> > > at
> > > com.formrunner.pdf.PDFUtils.getPDFDocFromInstance(PDFUtils.java:50)
> > > at
> > > com.formrunner.pdf.PDFMaker.makePDFfromFormInstance(PDFMaker.java:75)
> > >
> > > It may be worth noting that the LocalExample from the GCS Client API runs
> > > fine, but does not leave any trace in the
> > > folder ~war/WEB-APP/appengine-generated nor in the Development Consolte
> > > (so far as I can find).
> > >
> > > I hope someone can point out to me where I'm stumbling.
> > > Thanks in advance,
> > > Ken Bowen
> > >
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "Google App Engine" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [email protected].
> > > To post to this group, send email to [email protected].
> > > Visit this group at http://groups.google.com/group/google-appengine.
> > > For more options, visit https://groups.google.com/groups/opt_out.
> > >
> > >
> > > --
> > > You received this message because you are subscribed to a topic in the
> > > Google Groups "Google App Engine" group.
> > > To unsubscribe from this topic, visit
> > > https://groups.google.com/d/topic/google-appengine/TJ1d7wqZngo/unsubscribe.
> > > To unsubscribe from this group and all its topics, send an email to
> > > [email protected].
> > > To post to this group, send email to [email protected].
> > > Visit this group at http://groups.google.com/group/google-appengine.
> > > For more options, visit https://groups.google.com/groups/opt_out.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at http://groups.google.com/group/google-appengine.
> > For more options, visit https://groups.google.com/groups/opt_out.
> >
> >
> > --
> > You received this message because you are subscribed to a topic in the
> > Google Groups "Google App Engine" group.
> > To unsubscribe from this topic, visit
> > https://groups.google.com/d/topic/google-appengine/TJ1d7wqZngo/unsubscribe.
> > To unsubscribe from this group and all its topics, send an email to
> > [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at http://groups.google.com/group/google-appengine.
> > For more options, visit https://groups.google.com/groups/opt_out.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> --
> You received this message because you are subscribed to a topic in the Google
> Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/TJ1d7wqZngo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To post to this group, send email to [email protected].
> Visit this group at http://groups.google.com/group/google-appengine.
> For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
