Christian, thanks for that interesting (and kind of scary) work-around ( https://gist.github.com/cmaan/7752e3c4fd0b1ba90a745cb6db232206). I was having the same problem but with the PubSub client rather than the Spanner client. By switching the thread factory I was able to avoid the NPE at ApiProxyImpl.java:1267, but now I'm getting a slightly different problem:
Caused by: io.grpc.StatusRuntimeException: UNAUTHENTICATED at io.grpc.Status.asRuntimeException(Status.java:543 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2FStatus.java&line=543&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:442 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Fstub%2FClientCalls.java&line=442&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-stub-1.4.0.jar:1.4.0] at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FClientCallImpl.java&line=426&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.ClientCallImpl.access$100( ClientCallImpl.java:76 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FClientCallImpl.java&line=76&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close( ClientCallImpl.java:512 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FClientCallImpl.java&line=512&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700( ClientCallImpl.java:429 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FClientCallImpl.java&line=429&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext( ClientCallImpl.java:544 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FClientCallImpl.java&line=544&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.ContextRunnable.run( ContextRunnable.java:52 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FContextRunnable.java&line=52&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at io.grpc.internal.SerializingExecutor.run( SerializingExecutor.java:117 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Finternal%2FSerializingExecutor.java&line=117&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-core-1.4.0.jar:1.4.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_112-google-v7] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_112-google-v7] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_112-google-v7] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:295) ~[na:1.8.0_112-google-v7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112-google-v7] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112-google-v7] ... 1 common frames omitted Caused by: java.io.IOException: Could not get the access token. at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken( AppEngineCredentials.java:136 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=com%2Fgoogle%2Fauth%2Foauth2%2FAppEngineCredentials.java&line=136&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[google-auth-library-oauth2-http-0.7.0.jar:na] at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:149 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=com%2Fgoogle%2Fauth%2Foauth2%2FOAuth2Credentials.java&line=149&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[google-auth-library-oauth2-http-0.7.0.jar:na] at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata( OAuth2Credentials.java:135 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=com%2Fgoogle%2Fauth%2Foauth2%2FOAuth2Credentials.java&line=135&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[google-auth-library-oauth2-http-0.7.0.jar:na] at io.grpc.auth.GoogleAuthLibraryCallCredentials$1.run( GoogleAuthLibraryCallCredentials.java:110 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=io%2Fgrpc%2Fauth%2FGoogleAuthLibraryCallCredentials.java&line=110&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[grpc-auth-1.4.0.jar:1.4.0] ... 7 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112-google-v7] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112-google-v7] at com.google.auth.oauth2.AppEngineCredentials.refreshAccessToken( AppEngineCredentials.java:131 <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=726e5a9947359106976a136abc8e0ca6b35fb27c&file=com%2Fgoogle%2Fauth%2Foauth2%2FAppEngineCredentials.java&line=131&logInsertId=5978aba20009e3592c609b7b&logNanos=1501080480694147000&nestedLogIndex=2&project=mabl-prod-app&organizationId=283129770220>) ~[google-auth-library-oauth2-http-0.7.0.jar:na] ... 10 common frames omitted Caused by: com.google.apphosting.api.ApiProxy$CallNotFoundException: Can't make API call memcache.Get in a thread that is neither the original request thread nor a thread created by ThreadManager at com.google.apphosting.api.ApiProxy$CallNotFoundException.foreignThread(ApiProxy.java:844) ~[runtime-shared.jar:na] at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:183) ~[runtime-shared.jar:na] at com.google.apphosting.api.ApiProxy$1.get(ApiProxy.java:180) ~[runtime-shared.jar:na] at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java:93) ~[appengine-api.jar:na] at com.google.appengine.api.memcache.MemcacheServiceImpl.quietGet(MemcacheServiceImpl.java:28) ~[appengine-api.jar:na] at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java:51) ~[appengine-api.jar:na] at com.google.appengine.api.appidentity.AppIdentityServiceImpl.getAccessToken(AppIdentityServiceImpl.java:300) ~[appengine-api.jar:na] ... 14 common frames omitted I'll keep investigating. Maybe there is another thread factory that needs to be updated somewhere... On Thursday, July 20, 2017 at 12:37:56 PM UTC-4, Christian Maan wrote: > > We switched to the java8 runtime recently, and for the most part didn't > experience any major issues (yet). Performance did definitely improve for > us, compared to java7. On standard environment, we use a bunch of Google > cloud services, including Datastore, Vision API, Storage, Compute Engine, > Memcache, Pubsub, UserService, SearchService and BigQuery. All of them seem > to work without issues after upgrading to java8. > > Currently, we try to make use if some of the new grpc feature java8 is > advertised with. Unfortunately, things seem to be considerably more > unstable there. > > Based on the example here [1], we tried to get Spanner to work on java8, > using the ideomatic Java client libraries from google-cloud-java. The > example itself executes without problems, but any slight variation in code > structure will throw off the grpc initialization code and break the > example. We tracked the problem down to GrpcUtils.TIMER_SERVICE and > GrpcUtils.SHARED_CHANNEL_EXECUTOR. In the official example, classloading > side effects seem to determine the type of ThreadFactory that backs these > final static fields. If the effective ThreadFactory turns out to be a > ThreadManager.currentRequestThreadFactory(), because the initialization > happened in a context where MoreExecutors.isAppEngine() returns true, > everything falls apart and we are left with a NullPointerException in > ApiProxyImpl:1267 later on, when the Spanner session pool tries to create a > session [2]. > > If we - somewhat brutally - force GrpcUtils.TIMER_SERVICE and > GrpcUtils.SHARED_CHANNEL_EXECUTOR to use a Executors.defaultThreadFactory() > like this [3], we get a working grpc Spanner setup on Appengine Standard! > > So java8 looks promising, but from this experience we think it still needs > some work. > > - c > > > [1] > https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/appengine-java8/spanner > [2] https://gist.github.com/cmaan/6aed2228516a9ae3c98a883ea9d6d815 > [3] https://gist.github.com/cmaan/7752e3c4fd0b1ba90a745cb6db232206 > > On Thursday, July 6, 2017 at 9:45:38 PM UTC+2, Patrick Jackson wrote: >> >> Super excited to see Java 8 support. The removal of the white-list is >> also huge! Curious about other's experiences with it so far. Any >> instability, things to note or consider. >> I know this is beta and no SLA until GA and all that, just would be >> interested to hear from any devs that have been using it for real apps and >> traffic. >> >> -Patrick Jackson >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/6624f879-2511-493b-819e-6cd4783ff971%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
