[GitHub] [beam] suztomo edited a comment on pull request #14527: [BEAM-11205] Libraries-BOM 20.0.0 with gRPC and protobuf

[GitBox]

suztomo edited a comment on pull request #14527:
URL: https://github.com/apache/beam/pull/14527#issuecomment-819578305


   Other errors
   
   ```
   java.lang.AssertionError: 
   Expected: an instance of java.io.FileNotFoundException
        but: <java.io.IOException: Error executing batch GCS request> is a 
java.io.IOException
   Stacktrace was: java.io.IOException: Error executing batch GCS request
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtil.executeBatches(GcsUtil.java:585)
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtil.getObjects(GcsUtil.java:314)
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtil.fileSizes(GcsUtil.java:363)
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtilTest.testGetSizeBytesWhenFileNotFoundBatchRetry(GcsUtilTest.java:523)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
        at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
        at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
        at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
        at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
        at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
        at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
        at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
        at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
        at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
        at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
        at 
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
        at 
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at 
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at 
org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.lang.Thread.run(Thread.java:748)
   Caused by: java.util.concurrent.ExecutionException: 
java.lang.NullPointerException
        at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:60)
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtil.executeBatches(GcsUtil.java:577)
        ... 54 more
   Caused by: java.lang.NullPointerException
        at 
com.google.api.client.googleapis.batch.BatchUnparsedResponse.trimCrlf(BatchUnparsedResponse.java:276)
        at 
com.google.api.client.googleapis.batch.BatchUnparsedResponse.readLine(BatchUnparsedResponse.java:272)
        at 
com.google.api.client.googleapis.batch.BatchUnparsedResponse.parseNextResponse(BatchUnparsedResponse.java:97)
        at 
com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:267)
        at 
org.apache.beam.sdk.extensions.gcp.util.GcsUtil.lambda$executeBatches$0(GcsUtil.java:573)
        at 
org.apache.beam.sdk.util.MoreFutures.lambda$runAsync$2(MoreFutures.java:140)
        at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
   ```
   
   <img width="733" alt="Screen Shot 2021-04-14 at 10 43 55" 
src="https://user-images.githubusercontent.com/28604/114730546-f7f40700-9d0e-11eb-906e-c63953116b3a.png";>
   
   trimCrlf is getting `null`. This implies the mock response doesn't have 
enough values.
   
   <img width="619" alt="Screen Shot 2021-04-14 at 10 49 21" 
src="https://user-images.githubusercontent.com/28604/114730677-13f7a880-9d0f-11eb-92f7-a50ead8c744e.png";>
   
   
   <img width="1198" alt="Screen Shot 2021-04-14 at 11 11 14" 
src="https://user-images.githubusercontent.com/28604/114734631-65556700-9d12-11eb-87ab-4b6edcf9b413.png";>
   
   The content in the mock is the following. 
   
   ```
   --batch_foobarbaz
   Content-Type: application/http
   
   HTTP/1.1 404 Not Found
   Content-Length: -1
   
   {"error":{"code":404}}
   
   --batch_foobarbaz--
   ```
   
   <img width="1095" alt="Screen Shot 2021-04-14 at 11 28 58" 
src="https://user-images.githubusercontent.com/28604/114737173-bf572c00-9d14-11eb-9612-f09cdde94d1e.png";>
   
   
   `batchResponse.hasNext` is true
   `batchResponse`'s inputStream has 5 bytes ("error") and the position is 
already 5 (nothing to read).
   
   
   BatchRequest and BatchUnparsedResponse (the classes appearing in the stack 
trace) are from google-api-client:1.31.1. This PR does not touch the 
dependency. Why is this?
   
   
   # How about in master branch?
   
   `batchResponse`'s inputStream has 137 bytes.
   
   <img width="1153" alt="Screen Shot 2021-04-14 at 11 50 01" 
src="https://user-images.githubusercontent.com/28604/114740160-984e2980-9d17-11eb-91a7-63dc64aa4bcd.png";>
   
   The contentStream comes from google-http-client 1.39.2's HttpResponse.
   
   <img width="1162" alt="Screen Shot 2021-04-14 at 11 57 21" 
src="https://user-images.githubusercontent.com/28604/114741321-a3ee2000-9d18-11eb-8fc6-e5a9d3a52197.png";>
   
   `contentStream` is `response.content`.
   
   
   
   In the bad case, the content from the mocked http response is only retrieved 
once, and that's 5 bytes.
   
   <img width="662" alt="Screen Shot 2021-04-14 at 12 18 53" 
src="https://user-images.githubusercontent.com/28604/114744397-a736db00-9d1b-11eb-8732-ef0b678e037a.png";>
   
   In the good case, the content from the mocked http response is read first in 
`HttpResponse response = batchRequest.execute();` and then `InputStream 
contentStream = response.getContent();`.
   
   I think this is the diffrerence between `google-http-client:1.39.2` and 
`google-http-client:1.38.1`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org