suneet-s commented on pull request #10599: URL: https://github.com/apache/druid/pull/10599#issuecomment-736097807
Here is a snippet. Since I need to provide my gcs credentials, I'm a little apprehensive with sharing the entire logs - I can't be certain I've scrubbed it correctly. However, you can run the tests locally with the instructions in my previous comment. ``` [2020-11-30T17:44:46.939Z] 2020-11-30T17:44:25,231 WARN [[single_phase_sub_task_wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?_cigonojm_2020-11-30T17:43:59.726Z]-appenderator-merge] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 4) in 854ms. [2020-11-30T17:44:46.939Z] java.lang.RuntimeException: org.apache.druid.java.util.common.RE: Failed to upload [/tmp/index990328993385419658.zip] to [gcs-test-13676c60-e503-4b35-a65a-c41ba22aa4bf//wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?/2013-09-01T00:00:00.000Z_2013-09-02T00:00:00.000Z/2020-11-30T17:42:23.415Z/5/index.zip] [2020-11-30T17:44:46.939Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:127) ~[?:?] [2020-11-30T17:44:46.939Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:794) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:790) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:660) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?] [2020-11-30T17:44:46.939Z] at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?] [2020-11-30T17:44:46.939Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] [2020-11-30T17:44:46.939Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] [2020-11-30T17:44:46.939Z] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] [2020-11-30T17:44:46.939Z] Caused by: org.apache.druid.java.util.common.RE: Failed to upload [/tmp/index990328993385419658.zip] to [gcs-test-13676c60-e503-4b35-a65a-c41ba22aa4bf//wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?/2013-09-01T00:00:00.000Z_2013-09-02T00:00:00.000Z/2020-11-30T17:42:23.415Z/5/index.zip] [2020-11-30T17:44:46.939Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:98) ~[?:?] [2020-11-30T17:44:46.939Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.939Z] ... 11 more [2020-11-30T17:44:46.939Z] Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J [2020-11-30T17:44:46.939Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:97) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.939Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:84) ~[?:?] [2020-11-30T17:44:46.939Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.939Z] ... 11 more [2020-11-30T17:44:46.939Z] Caused by: java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J [2020-11-30T17:44:46.939Z] at com.google.api.client.http.ConsumingInputStream.close(ConsumingInputStream.java:40) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.939Z] at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227) ~[?:1.8.0_232] [2020-11-30T17:44:46.940Z] at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) ~[?:1.8.0_232] [2020-11-30T17:44:46.940Z] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:252) ~[jackson-core-2.10.2.jar:2.10.2] [2020-11-30T17:44:46.940Z] at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:369) ~[jackson-core-2.10.2.jar:2.10.2] [2020-11-30T17:44:46.940Z] at com.google.api.client.json.jackson2.JacksonParser.close(JacksonParser.java:47) ~[google-http-client-jackson2-1.37.0.jar:?] [2020-11-30T17:44:46.940Z] at com.google.api.client.json.JsonParser.parse(JsonParser.java:363) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.json.JsonParser.parse(JsonParser.java:335) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:451) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:346) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:397) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:494) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:217) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:880) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:550) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:567) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeUploadInitiation(MediaHttpUploader.java:524) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:383) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:335) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:550) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.940Z] at org.apache.druid.storage.google.GoogleStorage.insert(GoogleStorage.java:43) ~[?:?] [2020-11-30T17:44:46.940Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.lambda$insert$0(GoogleDataSegmentPusher.java:86) ~[?:?] [2020-11-30T17:44:46.940Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:84) ~[?:?] [2020-11-30T17:44:46.940Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.940Z] ... 11 more [2020-11-30T17:44:46.940Z] 2020-11-30T17:44:26,176 WARN [[single_phase_sub_task_wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?_cigonojm_2020-11-30T17:43:59.726Z]-appenderator-merge] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 4) in 1,713ms. [2020-11-30T17:44:46.940Z] java.lang.RuntimeException: org.apache.druid.java.util.common.RE: Failed to upload [/tmp/index5422991695199046872.zip] to [gcs-test-13676c60-e503-4b35-a65a-c41ba22aa4bf//wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?/2013-09-01T00:00:00.000Z_2013-09-02T00:00:00.000Z/2020-11-30T17:42:23.415Z/5/index.zip] [2020-11-30T17:44:46.940Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:127) ~[?:?] [2020-11-30T17:44:46.940Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:794) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:105) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:790) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:660) ~[druid-server-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.940Z] at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?] [2020-11-30T17:44:46.940Z] at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?] [2020-11-30T17:44:46.940Z] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] [2020-11-30T17:44:46.940Z] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] [2020-11-30T17:44:46.940Z] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] [2020-11-30T17:44:46.940Z] Caused by: org.apache.druid.java.util.common.RE: Failed to upload [/tmp/index5422991695199046872.zip] to [gcs-test-13676c60-e503-4b35-a65a-c41ba22aa4bf//wikipedia_index_test_80cd406e-edac-4b94-80b8-2962ab9a4073 Россия 한국 中国!?/2013-09-01T00:00:00.000Z_2013-09-02T00:00:00.000Z/2020-11-30T17:42:23.415Z/5/index.zip] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:98) ~[?:?] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.941Z] ... 11 more [2020-11-30T17:44:46.941Z] Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J [2020-11-30T17:44:46.941Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:97) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.941Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:84) ~[?:?] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.941Z] ... 11 more [2020-11-30T17:44:46.941Z] Caused by: java.lang.NoSuchMethodError: com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J [2020-11-30T17:44:46.941Z] at com.google.api.client.http.ConsumingInputStream.close(ConsumingInputStream.java:40) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227) ~[?:1.8.0_232] [2020-11-30T17:44:46.941Z] at java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) ~[?:1.8.0_232] [2020-11-30T17:44:46.941Z] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:252) ~[jackson-core-2.10.2.jar:2.10.2] [2020-11-30T17:44:46.941Z] at com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:369) ~[jackson-core-2.10.2.jar:2.10.2] [2020-11-30T17:44:46.941Z] at com.google.api.client.json.jackson2.JacksonParser.close(JacksonParser.java:47) ~[google-http-client-jackson2-1.37.0.jar:?] [2020-11-30T17:44:46.941Z] at com.google.api.client.json.JsonParser.parse(JsonParser.java:363) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.json.JsonParser.parse(JsonParser.java:335) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:451) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:346) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:397) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:494) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.auth.oauth2.Credential.intercept(Credential.java:217) ~[google-oauth-client-1.31.0.jar:1.31.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:880) ~[google-http-client-1.37.0.jar:1.37.0] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequestWithoutGZip(MediaHttpUploader.java:550) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeCurrentRequest(MediaHttpUploader.java:567) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.media.MediaHttpUploader.executeUploadInitiation(MediaHttpUploader.java:524) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:383) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:335) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:550) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) ~[google-api-client-1.30.10.jar:1.30.10] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleStorage.insert(GoogleStorage.java:43) ~[?:?] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.lambda$insert$0(GoogleDataSegmentPusher.java:86) ~[?:?] [2020-11-30T17:44:46.941Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:87) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.941Z] at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:115) ~[druid-core-0.20.0-iap5-SNAPSHOT.jar:0.20.0-iap5-SNAPSHOT] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.insert(GoogleDataSegmentPusher.java:84) ~[?:?] [2020-11-30T17:44:46.941Z] at org.apache.druid.storage.google.GoogleDataSegmentPusher.push(GoogleDataSegmentPusher.java:122) ~[?:?] [2020-11-30T17:44:46.941Z] ... 11 more ``` Since the integration tests indicate that ingesting data is broken with gcs as deep storage, can we revert this change while we investigate why the test/ ingestion is broken? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
