sure, attached the logs 2025-01-09, kt, 14:02 Zoltán Borók-Nagy <borokna...@cloudera.com> rašė:
> Thanks Saulius, this is very interesting. Trino seems to be much better > optimized for small queries on huge tables. This is definitely a room > for improvement for us. > > Where most time were being spent: > - Metadata of all 2 tables cached: 42s528ms (42s528ms) > - Single node plan created: 1m13s (30s493ms) > - Completed admission: 1m55s (28s320ms) > > Since the query has a LIMIT 10 clause, we could probably short circuit the > first two (I guess there are no delete files), and then admission would be > also faster probably. > > Is it possible to filter the coordinator logs for the query id and send it > to us, if it doesn't contain any sensitive information (column names, > etc.)? That'll provide us with more hints about where the time was spent. > > Cheers, > Zoltan > > > On Thu, Jan 9, 2025 at 12:47 PM Saulius Valatka <saulius...@gmail.com> > wrote: > > > Yeah, sorry, I was probably wrong about the "blocked" part: it's the > select > > statement without a filter that's taking very long, regardless of any > > running refreshes in parallel. Adding a partition filter for the last day > > brings down the execution time to ~10 seconds (still way below Trino, it > > can do it in ~2 seconds). > > > > 2025-01-09, kt, 13:31 Saulius Valatka <saulius...@gmail.com> rašė: > > > > > The table being "blocked" is just my hypothesis, maybe I'm wrong? I can > > > see that when a REFRESH statement is running, issuing a simple "select > * > > > from table limit 10" sits in "CREATED Query submitted" state until > > roughly > > > when the refresh statement finishes. > > > Just out of curiosity I launched a Trino cluster on the same servers as > > > Impala and hooked it up to the same Iceberg catalog, issuing an > identical > > > "select * from table limit 10" runs in ~2 seconds, whereas in Impala > it's > > > ~2 minutes. > > > > > > Here's the details for the select query execution: > > > > > > Query Compilation: 1m14s > > > - Metadata of all 2 tables cached: 42s528ms (42s528ms) > > > - Analysis finished: 42s539ms (10.112ms) > > > - Authorization finished (ranger): 42s540ms (1.246ms) > > > - Value transfer graph computed: 42s540ms (609.596us) > > > - Single node plan created: 1m13s (30s493ms) > > > - Runtime filters computed: 1m13s (44.325us) > > > - Distributed plan created: 1m13s (29.075us) > > > - Parallel plans created: 1m13s (266.400us) > > > - Planning finished: 1m14s (1s727ms) > > > Query Timeline: 2m13s > > > - Query submitted: 50.627us (50.627us) > > > - Planning finished: 1m27s (1m27s) > > > - Submit for admission: 1m27s (273.383us) > > > - Completed admission: 1m55s (28s320ms) > > > - Ready to start on 121 backends: 1m55s (91.685ms) > > > - All 121 execution backends (1921 fragment instances) started: 2m7s > > > (11s578ms) > > > - Rows available: 2m7s (1.412ms) > > > - First row fetched: 2m7s (291.051ms) > > > - Last row fetched: 2m7s (225.054ms) > > > - Released admission control resources: 2m13s (5s433ms) > > > - Unregister query: 2m13s (100.938ms) > > > > > > Catalog Server Operation: 49s120ms > > > - Got Metastore client: 3.116us (3.116us) > > > - Got catalog version read lock: 2.250ms (2.247ms) > > > - Got catalog version write lock and table write lock: 2.315ms > > > (65.074us) > > > - Got Metastore client: 2.320ms (4.258us) > > > - Fetched table from Metastore: 12.885ms (10.565ms) > > > - Loaded Iceberg API table: 72.570ms (59.684ms) > > > - Loaded schema from Iceberg: 72.751ms (181.127us) > > > - Loaded Iceberg files: 7s925ms (7s852ms) > > > - Loaded all column stats: 7s953ms (28.404ms) > > > - Loaded table schema: 7s958ms (5.226ms) > > > - Start refreshing file metadata: 7s959ms (431.115us) > > > - Loaded file metadata for 1 partitions: 22s081ms (14s122ms) > > > - Loaded all column stats: 49s120ms (27s038ms) > > > - Loaded table: 49s120ms (30.027us) > > > - Finished resetMetadata request: 49s120ms (312.849us) > > > Query Compilation: 53s727ms > > > - Metadata of all 2 tables cached: 53s726ms (53s726ms) > > > - Analysis finished: 53s726ms (67.048us) > > > - Authorization finished (ranger): 53s727ms (422.569us) > > > - Planning finished: 53s727ms (137.463us) > > > Query Timeline: 1m44s > > > - Query submitted: 42.071us (42.071us) > > > - Planning finished: 53s735ms (53s735ms) > > > - CatalogDdlRequest finished: 1m42s (49s122ms) > > > - Applied catalog updates from DDL: 1m43s (1s049ms) > > > - Request finished: 1m43s (14.695ms) > > > - Unregister query: 1m44s (912.626ms) > > > > > > 2025-01-09, kt, 12:33 Zoltán Borók-Nagy <borokna...@cloudera.com> > rašė: > > > > > >> Thanks for the update. > > >> > > >> So the whole REFRESH operation took 1m50s. From this CatalogDdlRequest > > was > > >> only 46s313ms. This 46s313ms is the time when the table is blocked, > > right? > > >> From CatalogDdlRequest the longest operation was loading column stats > > >> which > > >> took 27s300ms. This is a single RPC (getTableColumnStatistics()) > toward > > >> HMS, it would be good to know why it took so long. Especially given > that > > >> loading file metadata for this huge table was around 19 seconds. > > >> It's also interesting that "Loaded all column stats" appears twice in > > the > > >> catalog timeline. At first it took 11.697ms, and the second invocation > > was > > >> the one that took 27s300ms. Hopefully we can get rid of the second > > >> invocation but that'll require a code change. > > >> > > >> I also wonder why the table is not queryable from the Coordinator > cache > > >> while it is being reloaded in CatalogD, I hope we can fix this as > well. > > >> > > >> Is the table expected to grow indefinitely? Or do you drop/relocate > old > > >> partitions after some time? > > >> > > >> Cheers, > > >> Zoltan > > >> > > >> > > >> On Wed, Jan 8, 2025 at 9:19 PM Saulius Valatka <saulius...@gmail.com> > > >> wrote: > > >> > > >> > Hi, > > >> > > > >> > so I just tried applying IMPALA-13254 on top of 4.4.1, redeployed > and > > >> > refresh times for the largest table went down from ~80 minutes to ~2 > > >> > minutes! > > >> > That's waaay better, but still not ideal: if we issue a refresh > every > > 15 > > >> > minutes, there's still a lot of time the table is blocked for a > minute > > >> or > > >> > two, but at least now it's queryable. > > >> > > > >> > Here's an example REFRESH timeline: > > >> > > > >> > Catalog Server Operation: 46s213ms > > >> > - Got Metastore client: 5.771us (5.771us) > > >> > - Got catalog version read lock: 2.334ms (2.328ms) > > >> > - Got catalog version write lock and table write lock: 2.459ms > > >> > (125.470us) > > >> > - Got Metastore client: 2.465ms (6.181us) > > >> > - Fetched table from Metastore: 12.831ms (10.366ms) > > >> > - Loaded Iceberg API table: 139.024ms (126.192ms) > > >> > - Loaded schema from Iceberg: 139.175ms (150.949us) > > >> > - Loaded Iceberg files: 5s036ms (4s897ms) > > >> > - Loaded all column stats: 5s047ms (11.697ms) > > >> > - Loaded table schema: 5s053ms (5.279ms) > > >> > - Start refreshing file metadata: 5s053ms (291.959us) > > >> > - Loaded file metadata for 1 partitions: 18s912ms (13s859ms) > > >> > - Loaded all column stats: 46s213ms (27s300ms) > > >> > - Loaded table: 46s213ms (36.600us) > > >> > - Finished resetMetadata request: 46s213ms (485.329us) > > >> > Query Compilation: 1m3s > > >> > - Metadata of all 2 tables cached: 1m3s (1m3s) > > >> > - Analysis finished: 1m3s (237.705us) > > >> > - Authorization finished (ranger): 1m3s (808.267us) > > >> > - Planning finished: 1m3s (581.713us) > > >> > Query Timeline: 1m51s > > >> > - Query submitted: 40.227us (40.227us) > > >> > - Planning finished: 1m3s (1m3s) > > >> > - CatalogDdlRequest finished: 1m50s (46s313ms) > > >> > - Applied catalog updates from DDL: 1m50s (24.206ms) > > >> > - Request finished: 1m50s (202.949us) > > >> > - Unregister query: 1m51s (763.412ms) > > >> > > > >> > > > >> > 2025-01-08, tr, 17:50 Zoltán Borók-Nagy <borokna...@cloudera.com> > > rašė: > > >> > > > >> > > Thanks for the info, Saulius. > > >> > > > > >> > > If you try out IMPALA-13254, please let us know how much it helps > in > > >> > > your case. > > >> > > Hopefully it speeds up table loading times enough so it won't > cause > > >> too > > >> > > much turbulence. > > >> > > Some table loading statistics would be also helpful to know where > > the > > >> > time > > >> > > is being spent. > > >> > > > > >> > > Do you use local catalog mode? > > >> > > > > https://impala.apache.org/docs/build/html/topics/impala_metadata.html > > >> > > I'm not sure how much it will help, but it could be worth trying > > out. > > >> > > > > >> > > Cheers, > > >> > > Zoltan > > >> > > > > >> > > > > >> > > On Wed, Jan 8, 2025 at 2:45 PM Saulius Valatka < > > saulius...@gmail.com> > > >> > > wrote: > > >> > > > > >> > > > Hi, > > >> > > > > > >> > > > sorry, maybe I worded my question wrong: I understand that > > >> refreshing > > >> > is > > >> > > > needed (either automatic or manual), main concerns are the > latency > > >> of > > >> > the > > >> > > > refresh and the fact that the table is not queryable while it's > > >> being > > >> > > > refreshed - for large tables that are being updated frequently > > this > > >> > > > combination makes them essentially un-queryable. > > >> > > > > > >> > > > 2025-01-08, tr, 15:17 Gabor Kaszab <gaborkas...@apache.org> > rašė: > > >> > > > > > >> > > > > Hi, > > >> > > > > > > >> > > > > I don't think that the issue you describe is specific to > Iceberg > > >> in a > > >> > > > sense > > >> > > > > that even for Hive tables if you make changes using an engine > > that > > >> > > > doesn't > > >> > > > > trigger HMS events, one has to issue refresh/invalidate > metadata > > >> to > > >> > see > > >> > > > the > > >> > > > > changes reflected in Impala. > > >> > > > > Could you share what catalog you use for your Iceberg tables? > > And > > >> > what > > >> > > > tool > > >> > > > > do you use for data ingestion into these tables? > > >> > > > > If you use the HMS backed HiveCatalog as a catalog and an > engine > > >> that > > >> > > > > triggers HMS notifications, like Spark or Hive then even for > > >> Iceberg > > >> > > > tables > > >> > > > > you can avoid executing refresh manually. > > >> > > > > > > >> > > > > Gabor > > >> > > > > > > >> > > > > On Wed, Jan 8, 2025 at 1:48 PM Saulius Valatka < > > >> saulius...@gmail.com > > >> > > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Hi, > > >> > > > > > > > >> > > > > > If I understand correctly, once an Iceberg table is mutated > > >> outside > > >> > > of > > >> > > > > > Impala one has to run a refresh or invalidate statement. We > > >> noticed > > >> > > > that > > >> > > > > > running refresh on huge tables can take minutes and while > that > > >> is > > >> > > > > happening > > >> > > > > > querying them is blocked. We have large event tables that > are > > >> being > > >> > > > > updated > > >> > > > > > very frequently in real-time, by default we run a refresh > > after > > >> > each > > >> > > > > > update, so effectively this means such tables are > > un-queryable, > > >> as > > >> > > > > they're > > >> > > > > > constantly being refreshed. > > >> > > > > > > > >> > > > > > Is there something I'm missing? What would the > recommendation > > >> here > > >> > > be? > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
I0109 13:05:47.163920 3333202 impala-server.cc:1439] ab492505992cf378:6565071e00000000] Registered query query_id=ab492505992cf378:6565071e00000000 session_id=0742de357beddb9e:5e5dc38fe94d3692 I0109 13:05:47.164153 3333202 Frontend.java:2175] ab492505992cf378:6565071e00000000] Analyzing query: I0109 13:05:47.164196 3333202 Frontend.java:2187] ab492505992cf378:6565071e00000000] The original executor group sets from executor membership snapshot: [TExecutorGroupSet(curr_num_executors:120, expected_num_executors:20, exec_group_name_prefix:)] I0109 13:05:47.164222 3333202 RequestPoolService.java:200] ab492505992cf378:6565071e00000000] Default pool only, scheduler allocation is not specified. I0109 13:05:47.164243 3333202 Frontend.java:2207] ab492505992cf378:6565071e00000000] A total of 1 executor group sets to be considered for auto-scaling: [TExecutorGroupSet(curr_num_executors:120, expected_num_executors:20, exec_group_name_prefix:, max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647)] I0109 13:05:47.164258 3333202 Frontend.java:2248] ab492505992cf378:6565071e00000000] Consider executor group set: TExecutorGroupSet(curr_num_executors:120, expected_num_executors:20, exec_group_name_prefix:, max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647) with assumption of 16 cores per node. I0109 13:07:09.519587 3333202 BaseMetastoreTableOperations.java:199] ab492505992cf378:6565071e00000000] Refreshing table metadata from new version: hdfs://ams1-prod/user/hive/warehouse/events.db/list_show_item/metadata/142364-4f8efbf3-9b23-4489-885f-15fb50e88bc3.metadata.json I0109 13:07:09.557911 3333202 BaseMetastoreCatalog.java:67] ab492505992cf378:6565071e00000000] Table loaded by catalog: ImpalaHiveCatalog.events.list_show_item I0109 13:07:13.189919 3333202 MetastoreShim.java:677] ab492505992cf378:6565071e00000000] Table list_show_item,#bucket=0,isBucketed:false,tableType=EXTERNAL_TABLE,tableCapabilities=null I0109 13:07:13.189967 3333202 MetastoreShim.java:683] ab492505992cf378:6565071e00000000] Table has no specific required capabilities I0109 13:07:13.189988 3333202 MetastoreShim.java:698] ab492505992cf378:6565071e00000000] External unbucketed table with EXTREAD/WRITE capability:RW I0109 13:07:13.190986 3333202 MetastoreShim.java:677] ab492505992cf378:6565071e00000000] Table list_show_item,#bucket=0,isBucketed:false,tableType=EXTERNAL_TABLE,tableCapabilities=null I0109 13:07:13.191015 3333202 MetastoreShim.java:683] ab492505992cf378:6565071e00000000] Table has no specific required capabilities I0109 13:07:13.191031 3333202 MetastoreShim.java:698] ab492505992cf378:6565071e00000000] External unbucketed table with EXTREAD/WRITE capability:RW I0109 13:07:13.191761 3333202 AnalysisContext.java:514] ab492505992cf378:6565071e00000000] Analysis took 3 ms I0109 13:07:13.192126 3333202 BaseAuthorizationChecker.java:114] ab492505992cf378:6565071e00000000] Authorization check took 0 ms I0109 13:07:13.192178 3333202 Frontend.java:2569] ab492505992cf378:6565071e00000000] Analysis and authorization finished. I0109 13:07:45.637831 3333202 Analyzer.java:3016] ab492505992cf378:6565071e00000000] null slots cache size: 0, median entry: 0.0, 99th percentile entry: 0.0, hit rate: 1.0 I0109 13:07:47.509709 3333202 Frontend.java:2465] ab492505992cf378:6565071e00000000] Selected executor group: TExecutorGroupSet(curr_num_executors:120, expected_num_executors:20, exec_group_name_prefix:, max_mem_limit:9223372036854775807, num_cores_per_executor:2147483647), reason: suitable group found. requirement{MemoryAsk=2.08GB, CpuAsk=-1, CpuAskBounded=-1, EffectiveParallelism=-1} I0109 13:08:01.082475 3333202 impala-server.cc:1496] Query ab492505992cf378:6565071e00000000 has idle timeout of 5m I0109 13:08:22.627317 3333960 admission-controller.cc:1965] ab492505992cf378:6565071e00000000] Trying to admit id=ab492505992cf378:6565071e00000000 in pool_name=default-pool executor_group_name=default per_host_mem_estimate=2.08 GB dedicated_coord_mem_estimate=160.43 MB max_requests=-1 max_queued=200 max_mem=-1.00 B is_trivial_query=false I0109 13:08:22.627341 3333960 admission-controller.cc:1974] ab492505992cf378:6565071e00000000] Stats: agg_num_running=2, agg_num_queued=0, agg_mem_reserved=139.85 MB, local_host(local_mem_admitted=0, local_trivial_running=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=242.00 B, topN_query_stats: queries=[e64edbb7a25e92d8:8f74485300000000], total_mem_consumed=242.00 B, fraction_of_pool_total_mem=1; pool_level_stats: num_running=1, min=242.00 B, max=242.00 B, pool_total_mem=242.00 B, average_per_query=242.00 B) I0109 13:08:22.627360 3333960 admission-controller.cc:1313] ab492505992cf378:6565071e00000000] Admitting query id=ab492505992cf378:6565071e00000000 I0109 13:08:26.034152 3333960 impala-server.cc:2400] ab492505992cf378:6565071e00000000] Registering query locations I0109 13:08:26.034222 3333960 coordinator.cc:153] ab492505992cf378:6565071e00000000] Exec() query_id=ab492505992cf378:6565071e00000000 stmt= I0109 13:08:26.131637 3333960 coordinator.cc:529] ab492505992cf378:6565071e00000000] starting execution on 121 backends for query_id=ab492505992cf378:6565071e00000000 I0109 13:08:26.132484 2944702 control-service.cc:148] ab492505992cf378:6565071e00000000] ExecQueryFInstances(): query_id=ab492505992cf378:6565071e00000000 coord=gs1-hdp-meta1.ams1.vinted.infra:27002 #instances=1 I0109 13:08:26.136976 3334037 query-state.cc:989] ab492505992cf378:6565071e00000000] Executing instance. instance_id=ab492505992cf378:6565071e00000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=2 I0109 13:08:28.781505 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data102.ams1.vinted.infra:27002 remaining=121 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:28.781545 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:29.053280 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data56.ams1.vinted.infra:27002 remaining=120 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:29.053289 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:29.611016 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data39.ams1.vinted.infra:27002 remaining=119 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:29.611027 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:29.725065 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data61.ams1.vinted.infra:27002 remaining=118 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:29.725075 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:29.806106 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data64.ams1.vinted.infra:27002 remaining=117 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:29.806113 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:30.033536 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data6.ams1.vinted.infra:27002 remaining=116 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:30.033552 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:30.258028 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data105.ams1.vinted.infra:27002 remaining=115 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:30.258049 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:30.465395 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data48.ams1.vinted.infra:27002 remaining=114 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:30.465435 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:30.611119 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data51.ams1.vinted.infra:27002 remaining=113 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:30.611126 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.181686 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data59.ams1.vinted.infra:27002 remaining=112 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.181700 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.219131 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data74.ams1.vinted.infra:27002 remaining=111 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.219141 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.311091 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data5.ams1.vinted.infra:27002 remaining=110 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.311152 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.335904 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data85.ams1.vinted.infra:27002 remaining=109 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.335940 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.444757 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data103.ams1.vinted.infra:27002 remaining=108 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.444782 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.621869 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data66.ams1.vinted.infra:27002 remaining=107 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.621883 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.847242 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data104.ams1.vinted.infra:27002 remaining=106 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.847251 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:31.915413 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data108.ams1.vinted.infra:27002 remaining=105 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:31.915421 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.059258 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data83.ams1.vinted.infra:27002 remaining=104 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.059269 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.072188 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data17.ams1.vinted.infra:27002 remaining=103 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.072196 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.149850 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data87.ams1.vinted.infra:27002 remaining=102 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.149873 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.202813 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data68.ams1.vinted.infra:27002 remaining=101 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.202843 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.320941 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data27.ams1.vinted.infra:27002 remaining=100 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.320956 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.609344 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data4.ams1.vinted.infra:27002 remaining=99 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.609352 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.649752 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data29.ams1.vinted.infra:27002 remaining=98 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.649760 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.665685 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data72.ams1.vinted.infra:27002 remaining=97 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.665694 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.681927 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data53.ams1.vinted.infra:27002 remaining=96 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.681936 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.717797 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data106.ams1.vinted.infra:27002 remaining=95 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.717804 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.810206 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data76.ams1.vinted.infra:27002 remaining=94 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.810213 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:32.828117 2944705 coordinator.cc:1119] Backend completed: host=gs1-hdp-data70.ams1.vinted.infra:27002 remaining=93 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:32.828125 2944705 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.074481 3334063 admission-controller.cc:1974] ce4d9e1a335bc5b0:ebd9786000000000] Stats: agg_num_running=4, agg_num_queued=0, agg_mem_reserved=228.30 GB, local_host(local_mem_admitted=1542.06 GB, local_trivial_running=0, num_admitted_running=2, num_queued=0, backend_mem_reserved=89.02 KB, topN_query_stats: queries=[ab492505992cf378:6565071e00000000, ff411cf5c8505b78:b6dd4fb900000000, 8d424df63fbbcf14:69c152f500000000], total_mem_consumed=89.02 KB, fraction_of_pool_total_mem=1; pool_level_stats: num_running=3, min=278.00 B, max=80.00 KB, pool_total_mem=89.02 KB, average_per_query=29.67 KB) I0109 13:08:33.269745 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data62.ams1.vinted.infra:27002 remaining=92 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.269759 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.356721 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data36.ams1.vinted.infra:27002 remaining=91 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.356737 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.415619 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data71.ams1.vinted.infra:27002 remaining=90 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.415633 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.538529 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data91.ams1.vinted.infra:27002 remaining=89 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.538540 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.567719 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data20.ams1.vinted.infra:27002 remaining=88 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.567727 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.665575 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data86.ams1.vinted.infra:27002 remaining=87 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.665585 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.805544 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data115.ams1.vinted.infra:27002 remaining=86 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.805560 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.810258 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data99.ams1.vinted.infra:27002 remaining=85 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.810267 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:33.990379 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data93.ams1.vinted.infra:27002 remaining=84 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:33.990396 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.353696 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data82.ams1.vinted.infra:27002 remaining=83 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.353835 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.376679 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data114.ams1.vinted.infra:27002 remaining=82 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.376724 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.429914 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data47.ams1.vinted.infra:27002 remaining=81 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.429929 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.548027 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data120.ams1.vinted.infra:27002 remaining=80 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.548038 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.714259 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data117.ams1.vinted.infra:27002 remaining=79 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.714273 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:34.982574 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data69.ams1.vinted.infra:27002 remaining=78 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:34.982582 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.113451 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data13.ams1.vinted.infra:27002 remaining=77 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.113458 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.224303 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data45.ams1.vinted.infra:27002 remaining=76 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.224314 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.348060 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data32.ams1.vinted.infra:27002 remaining=75 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.348073 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.528587 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data77.ams1.vinted.infra:27002 remaining=74 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.528600 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.669140 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data100.ams1.vinted.infra:27002 remaining=73 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.669173 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.697361 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data52.ams1.vinted.infra:27002 remaining=72 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.697378 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.867619 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data113.ams1.vinted.infra:27002 remaining=71 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.867630 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:35.931090 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data46.ams1.vinted.infra:27002 remaining=70 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:35.931118 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.119619 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data95.ams1.vinted.infra:27002 remaining=69 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.119637 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.264762 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data26.ams1.vinted.infra:27002 remaining=68 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.264773 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.337400 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data43.ams1.vinted.infra:27002 remaining=67 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.337411 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.402230 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data18.ams1.vinted.infra:27002 remaining=66 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.402244 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.434816 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data22.ams1.vinted.infra:27002 remaining=65 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.434831 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.535873 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data81.ams1.vinted.infra:27002 remaining=64 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.535890 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.646411 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data112.ams1.vinted.infra:27002 remaining=63 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.646524 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.682016 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data25.ams1.vinted.infra:27002 remaining=62 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.682076 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.741271 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data96.ams1.vinted.infra:27002 remaining=61 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.741287 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.790048 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data11.ams1.vinted.infra:27002 remaining=60 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.790067 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.872507 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data119.ams1.vinted.infra:27002 remaining=59 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.872525 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:36.933048 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data58.ams1.vinted.infra:27002 remaining=58 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:36.933066 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.161193 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data19.ams1.vinted.infra:27002 remaining=57 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.161209 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.349977 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data1.ams1.vinted.infra:27002 remaining=56 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.350008 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.434109 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data111.ams1.vinted.infra:27002 remaining=55 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.434129 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.670334 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data73.ams1.vinted.infra:27002 remaining=54 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.670364 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.705749 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data65.ams1.vinted.infra:27002 remaining=53 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.705778 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.843677 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data79.ams1.vinted.infra:27002 remaining=52 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.843709 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.935942 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data9.ams1.vinted.infra:27002 remaining=51 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.935961 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:37.990212 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data118.ams1.vinted.infra:27002 remaining=50 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:37.990227 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.031528 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data3.ams1.vinted.infra:27002 remaining=49 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.031539 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.034487 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data12.ams1.vinted.infra:27002 remaining=48 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.034498 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.169459 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data84.ams1.vinted.infra:27002 remaining=47 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.169476 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.214686 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data97.ams1.vinted.infra:27002 remaining=46 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.214699 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.369540 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data23.ams1.vinted.infra:27002 remaining=45 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.369558 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.448426 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data75.ams1.vinted.infra:27002 remaining=44 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.448441 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.571250 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data8.ams1.vinted.infra:27002 remaining=43 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.571291 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.583613 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data94.ams1.vinted.infra:27002 remaining=42 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.583644 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.591964 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data109.ams1.vinted.infra:27002 remaining=41 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.591990 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.599069 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data54.ams1.vinted.infra:27002 remaining=40 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.599113 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.618865 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data10.ams1.vinted.infra:27002 remaining=39 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.618885 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.646982 3333960 coordinator.cc:588] ab492505992cf378:6565071e00000000] started execution on 121 backends for query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.755657 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data44.ams1.vinted.infra:27002 remaining=38 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.755673 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.827246 3333202 impala-hs2-server.cc:978] GetResultSetMetadata(): query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.854773 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data35.ams1.vinted.infra:27002 remaining=37 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.854791 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.872212 3334037 krpc-data-stream-mgr.cc:308] ab492505992cf378:6565071e00000000] DeregisterRecvr(): fragment_instance_id=ab492505992cf378:6565071e00000000, node=1 I0109 13:08:38.872260 3334037 query-state.cc:998] ab492505992cf378:6565071e00000000] Instance completed. instance_id=ab492505992cf378:6565071e00000000 #in-flight=1 status=OK I0109 13:08:38.872290 3334036 query-state.cc:499] ab492505992cf378:6565071e00000000] UpdateBackendExecState(): last report for ab492505992cf378:6565071e00000000 I0109 13:08:38.873744 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data30.ams1.vinted.infra:27002 remaining=36 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.873759 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-meta1.ams1.vinted.infra:27002 I0109 13:08:38.874125 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-meta1.ams1.vinted.infra:27002 remaining=35 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:38.874142 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.012591 3333202 coordinator.cc:794] ExecState: query id=ab492505992cf378:6565071e00000000 execution completed I0109 13:08:39.012658 3333202 coordinator.cc:969] Coordinator waiting for backends to finish, 34 remaining. query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.113078 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data28.ams1.vinted.infra:27002 remaining=34 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.113109 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.119916 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data24.ams1.vinted.infra:27002 remaining=33 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.119928 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.318886 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data42.ams1.vinted.infra:27002 remaining=32 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.318904 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.480057 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data67.ams1.vinted.infra:27002 remaining=31 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.480108 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.738608 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data55.ams1.vinted.infra:27002 remaining=30 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.738628 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.739625 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data57.ams1.vinted.infra:27002 remaining=29 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.739634 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:39.822952 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data14.ams1.vinted.infra:27002 remaining=28 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:39.822971 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:40.040556 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data21.ams1.vinted.infra:27002 remaining=27 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.040603 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:40.042246 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data31.ams1.vinted.infra:27002 remaining=26 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.042289 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:40.052907 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data37.ams1.vinted.infra:27002 remaining=25 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.052923 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:40.306334 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data90.ams1.vinted.infra:27002 remaining=24 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.306354 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data49.ams1.vinted.infra:27002 I0109 13:08:40.323765 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data49.ams1.vinted.infra:27002 remaining=23 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.323783 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:40.716979 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data15.ams1.vinted.infra:27002 remaining=22 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.716998 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:40.787917 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data80.ams1.vinted.infra:27002 remaining=21 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.787940 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:40.841059 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data116.ams1.vinted.infra:27002 remaining=20 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.841077 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:40.958715 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data107.ams1.vinted.infra:27002 remaining=19 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:40.958734 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.031767 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data98.ams1.vinted.infra:27002 remaining=18 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.031785 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.072574 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data34.ams1.vinted.infra:27002 remaining=17 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.072594 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.151324 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data110.ams1.vinted.infra:27002 remaining=16 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.151343 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.171056 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data63.ams1.vinted.infra:27002 remaining=15 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.171072 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.220394 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data38.ams1.vinted.infra:27002 remaining=14 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.220409 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.303170 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data101.ams1.vinted.infra:27002 remaining=13 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.303191 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data88.ams1.vinted.infra:27002 I0109 13:08:41.327658 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data88.ams1.vinted.infra:27002 remaining=12 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.327673 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data78.ams1.vinted.infra:27002 I0109 13:08:41.328217 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data41.ams1.vinted.infra:27002 remaining=11 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.328238 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data78.ams1.vinted.infra:27002 I0109 13:08:41.340873 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data78.ams1.vinted.infra:27002 remaining=10 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.340893 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:41.359424 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data7.ams1.vinted.infra:27002 remaining=9 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.359445 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:41.612321 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data60.ams1.vinted.infra:27002 remaining=8 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.612344 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:41.709527 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data16.ams1.vinted.infra:27002 remaining=7 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.709549 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:41.831023 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data40.ams1.vinted.infra:27002 remaining=6 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:41.831049 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:42.013710 2944615 coordinator.cc:1119] Backend completed: host=gs1-hdp-data92.ams1.vinted.infra:27002 remaining=5 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:42.013733 2944615 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:42.159778 2944591 coordinator.cc:1119] Backend completed: host=gs1-hdp-data89.ams1.vinted.infra:27002 remaining=4 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:42.159849 2944591 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:42.346688 2944702 coordinator.cc:1119] Backend completed: host=gs1-hdp-data33.ams1.vinted.infra:27002 remaining=3 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:42.346709 2944702 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data2.ams1.vinted.infra:27002 I0109 13:08:42.454581 2944601 coordinator.cc:1119] Backend completed: host=gs1-hdp-data2.ams1.vinted.infra:27002 remaining=2 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:42.454597 2944601 coordinator-backend-state.cc:415] query_id=ab492505992cf378:6565071e00000000: first in-progress backend: gs1-hdp-data50.ams1.vinted.infra:27002 I0109 13:08:44.023592 2944586 coordinator.cc:1119] Backend completed: host=gs1-hdp-data50.ams1.vinted.infra:27002 remaining=1 query_id=ab492505992cf378:6565071e00000000 I0109 13:08:44.023686 3333202 coordinator.cc:1463] Release admission control resources for query_id=ab492505992cf378:6565071e00000000 I0109 13:08:44.258500 3333202 impala-hs2-server.cc:946] CloseOperation(): query_id=ab492505992cf378:6565071e00000000 I0109 13:08:44.258523 3333202 impala-server.cc:1540] UnregisterQuery(): query_id=ab492505992cf378:6565071e00000000 I0109 13:08:44.258534 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.30.27:27002: Not cancelling because the backend is already done: I0109 13:08:44.258541 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.7.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258545 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.12.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258549 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.18.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258553 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.10.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258558 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.41.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258560 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.57.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258564 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.33.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258569 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.9.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258572 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.9.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258576 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.44.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258580 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.19.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258584 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.25.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258589 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.25.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258592 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.62.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258596 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.43.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258600 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.45.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258605 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.63.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258608 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.38.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258631 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.49.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258635 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.1.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258639 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.27.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258643 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.11.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258647 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.27.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258651 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.47.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258656 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.21.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258658 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.16.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258662 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.16.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258666 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.46.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258670 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.53.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258674 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.60.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258677 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.22.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258682 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.29.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258685 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.12.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258688 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.24.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258692 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.56.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258697 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.68.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258700 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.49.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258704 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.15.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258708 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.17.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258716 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.31.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258721 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.4.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258724 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.17.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258728 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.37.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258733 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.4.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258736 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.65.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258740 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.21.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258744 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.58.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258747 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.13.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258751 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.48.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258754 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.29.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258759 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.6.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258762 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.46.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258766 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.52.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258770 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.3.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258774 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.47.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258777 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.26.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258781 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.22.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258785 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.55.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258790 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.10.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258793 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.54.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258805 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.15.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258808 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.13.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258811 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.42.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258816 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.66.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258819 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.57.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258822 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.23.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258826 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.51.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258831 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.38.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258834 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.41.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258837 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.48.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258841 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.35.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258845 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.59.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258849 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.23.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258853 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.8.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258857 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.2.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258860 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.36.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258864 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.2.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258868 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.39.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258872 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.56.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258875 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.44.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258879 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.64.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258886 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.14.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258890 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.32.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258894 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.67.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258898 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.18.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258901 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.53.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258906 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.30.47:27002: Not cancelling because the backend is already done: Cancelled I0109 13:08:44.258910 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.32.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258914 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.50.47:27002: Not cancelling because the backend is already done: Cancelled I0109 13:08:44.258919 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.7.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258922 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.5.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258926 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.36.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258930 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.26.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258935 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.30.45:27002: Not cancelling because the backend is already done: Cancelled in ScannerContext I0109 13:08:44.258939 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.43.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258944 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.5.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258947 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.11.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258950 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.50.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258955 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.14.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258957 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.52.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258961 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.35.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258968 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.24.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258972 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.8.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258977 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.3.45:27002: Not cancelling because the backend is already done: Cancelled in ScannerContext I0109 13:08:44.258981 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.51.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258986 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.55.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258989 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.58.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.258993 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.45.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.258997 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.19.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259001 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.54.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.259006 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.39.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259011 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.34.45:27002: Not cancelling because the backend is already done: Cancelled I0109 13:08:44.259016 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.61.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.259018 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.42.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.259022 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.1.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259025 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.34.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.259028 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.6.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259032 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.31.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259035 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.33.47:27002: Not cancelling because the backend is already done: I0109 13:08:44.259038 3333202 coordinator-backend-state.cc:1026] query_id=ab492505992cf378:6565071e00000000 target backend=10.37.37.45:27002: Not cancelling because the backend is already done: I0109 13:08:44.259042 3333202 coordinator.cc:1085] CancelBackends() query_id=ab492505992cf378:6565071e00000000, tried to cancel 0 backends I0109 13:08:44.334363 2945274 impala-server.cc:1577] Query successfully unregistered: query_id=ab492505992cf378:6565071e00000000 I0109 13:08:45.635586 2945274 query-exec-mgr.cc:219] ReleaseQueryState(): deleted query_id=ab492505992cf378:6565071e00000000