Hi,
It's hard to say what might be the problem without additional information.
Could you clarify on the following questions:
1. What was the rate of write operations for the 270M rows you mentioned?
Was that regular 50K rows/sec or something else?
2. Do you still observe the slowness or it's already gone? (if that was
just a spike in ops rate, Kudu tablet servers might be rejecting writes due
to the memory pressure when not being able to flush the data on disk fast
enough).
3. What was special about those 270M rows? Maybe, the order of keys
changed somehow? Kudu ingests data much faster if row keys come in kind of
sequential order: so if the prior write batches had that property, but the
recent 270M chunk didn't, it might be the case.
4. Was there any additional concurrent activity on the nodes where Kudu
tablet servers run?
Thanks,
Alexey
On Thu, Jul 18, 2019 at 7:48 PM Tim Armstrong <tarmstr...@cloudera.com>
wrote:
> Also including the Kudu list in case someone there recognises the problem.
>
> On Thu, Jul 18, 2019 at 8:05 AM lk_hadoop <lk_had...@163.com> wrote:
>
>> I0718 18:42:22.677520 51139 coordinator.cc:357] starting execution on 5
>> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000
>> I0718 18:42:22.679605 12873 impala-internal-service.cc:44]
>> ExecQueryFInstances(): query_id=2e4a3fbec0d7d721:2ec73c1c00000000
>> I0718 18:42:22.679620 12873 query-exec-mgr.cc:46] StartQueryFInstances()
>> query_id=2e4a3fbec0d7d721:2ec73c1c00000000
>> coord=realtimeanalysis-kudu-04-10-8-50-58:22000
>> I0718 18:42:22.679625 12873 query-state.cc:178] Buffer pool limit for
>> 2e4a3fbec0d7d721:2ec73c1c00000000: 17179869184
>> I0718 18:42:22.679675 12873 initial-reservations.cc:60] Successfully
>> claimed initial reservations (4.00 MB) for query
>> 2e4a3fbec0d7d721:2ec73c1c00000000
>> I0718 18:42:22.679769 51332 query-state.cc:309] StartFInstances():
>> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 #instances=2
>> I0718 18:42:22.680196 51332 query-state.cc:322] descriptor table for
>> query=2e4a3fbec0d7d721:2ec73c1c00000000
>> tuples:
>> Tuple(id=2 size=567 slots=[Slot(id=52 type=INT col_path=[] offset=464
>> null=(offset=563 mask=20) slot_idx=29 field_idx=-1), Slot(id=53 type=STRING
>> col_path=[] offset=0 null=(offset=560 mask=1) slot_idx=0 field_idx=-1),
>> Slot(id=54 type=STRING col_path=[] offset=16 null=(offset=560 mask=2)
>> slot_idx=1 field_idx=-1), Slot(id=55 type=STRING col_path=[] offset=32
>> null=(offset=560 mask=4) slot_idx=2 field_idx=-1), Slot(id=56 type=STRING
>> col_path=[] offset=48 null=(offset=560 mask=8) slot_idx=3 field_idx=-1),
>> Slot(id=57 type=STRING col_path=[] offset=64 null=(offset=560 mask=10)
>> slot_idx=4 field_idx=-1), Slot(id=58 type=STRING col_path=[] offset=80
>> null=(offset=560 mask=20) slot_idx=5 field_idx=-1), Slot(id=59 type=STRING
>> col_path=[] offset=96 null=(offset=560 mask=40) slot_idx=6 field_idx=-1),
>> Slot(id=60 type=STRING col_path=[] offset=112 null=(offset=560 mask=80)
>> slot_idx=7 field_idx=-1), Slot(id=61 type=STRING col_path=[] offset=128
>> null=(offset=561 mask=1) slot_idx=8 field_idx=-1), Slot(id=62 type=STRING
>> col_path=[] offset=144 null=(offset=561 mask=2) slot_idx=9 field_idx=-1),
>> Slot(id=63 type=STRING col_path=[] offset=160 null=(offset=561 mask=4)
>> slot_idx=10 field_idx=-1), Slot(id=64 type=INT col_path=[] offset=468
>> null=(offset=563 mask=40) slot_idx=30 field_idx=-1), Slot(id=65 type=INT
>> col_path=[] offset=472 null=(offset=563 mask=80) slot_idx=31 field_idx=-1),
>> Slot(id=66 type=INT col_path=[] offset=476 null=(offset=564 mask=1)
>> slot_idx=32 field_idx=-1), Slot(id=67 type=INT col_path=[] offset=480
>> null=(offset=564 mask=2) slot_idx=33 field_idx=-1), Slot(id=68 type=STRING
>> col_path=[] offset=176 null=(offset=561 mask=8) slot_idx=11 field_idx=-1),
>> Slot(id=69 type=STRING col_path=[] offset=192 null=(offset=561 mask=10)
>> slot_idx=12 field_idx=-1), Slot(id=70 type=STRING col_path=[] offset=208
>> null=(offset=561 mask=20) slot_idx=13 field_idx=-1), Slot(id=71 type=STRING
>> col_path=[] offset=224 null=(offset=561 mask=40) slot_idx=14 field_idx=-1),
>> Slot(id=72 type=STRING col_path=[] offset=240 null=(offset=561 mask=80)
>> slot_idx=15 field_idx=-1), Slot(id=73 type=STRING col_path=[] offset=256
>> null=(offset=562 mask=1) slot_idx=16 field_idx=-1), Slot(id=74 type=INT
>> col_path=[] offset=484 null=(offset=564 mask=4) slot_idx=34 field_idx=-1),
>> Slot(id=75 type=INT col_path=[] offset=488 null=(offset=564 mask=8)
>> slot_idx=35 field_idx=-1), Slot(id=76 type=INT col_path=[] offset=492
>> null=(offset=564 mask=10) slot_idx=36 field_idx=-1), Slot(id=77 type=INT
>> col_path=[] offset=496 null=(offset=564 mask=20) slot_idx=37 field_idx=-1),
>> Slot(id=78 type=INT col_path=[] offset=500 null=(offset=564 mask=40)
>> slot_idx=38 field_idx=-1), Slot(id=79 type=INT col_path=[] offset=504
>> null=(offset=564 mask=80) slot_idx=39 field_idx=-1), Slot(id=80 type=INT
>> col_path=[] offset=508 null=(offset=565 mask=1) slot_idx=40 field_idx=-1),
>> Slot(id=81 type=STRING col_path=[] offset=272 null=(offset=562 mask=2)
>> slot_idx=17 field_idx=-1), Slot(id=82 type=STRING col_path=[] offset=288
>> null=(offset=562 mask=4) slot_idx=18 field_idx=-1), Slot(id=83 type=INT
>> col_path=[] offset=512 null=(offset=565 mask=2) slot_idx=41 field_idx=-1),
>> Slot(id=84 type=STRING col_path=[] offset=304 null=(offset=562 mask=8)
>> slot_idx=19 field_idx=-1), Slot(id=85 type=STRING col_path=[] offset=320
>> null=(offset=562 mask=10) slot_idx=20 field_idx=-1), Slot(id=86 type=STRING
>> col_path=[] offset=336 null=(offset=562 mask=20) slot_idx=21 field_idx=-1),
>> Slot(id=87 type=STRING col_path=[] offset=352 null=(offset=562 mask=40)
>> slot_idx=22 field_idx=-1), Slot(id=88 type=STRING col_path=[] offset=368
>> null=(offset=562 mask=80) slot_idx=23 field_idx=-1), Slot(id=89 type=INT
>> col_path=[] offset=516 null=(offset=565 mask=4) slot_idx=42 field_idx=-1),
>> Slot(id=90 type=INT col_path=[] offset=520 null=(offset=565 mask=8)
>> slot_idx=43 field_idx=-1), Slot(id=91 type=INT col_path=[] offset=524
>> null=(offset=565 mask=10) slot_idx=44 field_idx=-1), Slot(id=92 type=INT
>> col_path=[] offset=528 null=(offset=565 mask=20) slot_idx=45 field_idx=-1),
>> Slot(id=93 type=INT col_path=[] offset=532 null=(offset=565 mask=40)
>> slot_idx=46 field_idx=-1), Slot(id=94 type=INT col_path=[] offset=536
>> null=(offset=565 mask=80) slot_idx=47 field_idx=-1), Slot(id=95 type=INT
>> col_path=[] offset=540 null=(offset=566 mask=1) slot_idx=48 field_idx=-1),
>> Slot(id=96 type=INT col_path=[] offset=544 null=(offset=566 mask=2)
>> slot_idx=49 field_idx=-1), Slot(id=97 type=INT col_path=[] offset=548
>> null=(offset=566 mask=4) slot_idx=50 field_idx=-1), Slot(id=98 type=INT
>> col_path=[] offset=552 null=(offset=566 mask=8) slot_idx=51 field_idx=-1),
>> Slot(id=99 type=INT col_path=[] offset=556 null=(offset=566 mask=10)
>> slot_idx=52 field_idx=-1), Slot(id=100 type=STRING col_path=[] offset=384
>> null=(offset=563 mask=1) slot_idx=24 field_idx=-1), Slot(id=101 type=STRING
>> col_path=[] offset=400 null=(offset=563 mask=2) slot_idx=25 field_idx=-1),
>> Slot(id=102 type=STRING col_path=[] offset=416 null=(offset=563 mask=4)
>> slot_idx=26 field_idx=-1), Slot(id=103 type=DECIMAL(30,6) col_path=[]
>> offset=432 null=(offset=563 mask=8) slot_idx=27 field_idx=-1), Slot(id=104
>> type=DECIMAL(30,6) col_path=[] offset=448 null=(offset=563 mask=10)
>> slot_idx=28 field_idx=-1)] tuple_path=[])
>> Tuple(id=1 size=0 slots=[] tuple_path=[])
>> Tuple(id=0 size=563 slots=[Slot(id=0 type=STRING col_path=[0] offset=0
>> null=(offset=556 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING
>> col_path=[29] offset=16 null=(offset=556 mask=2) slot_idx=1 field_idx=-1),
>> Slot(id=2 type=STRING col_path=[1] offset=32 null=(offset=556 mask=4)
>> slot_idx=2 field_idx=-1), Slot(id=3 type=STRING col_path=[30] offset=48
>> null=(offset=556 mask=8) slot_idx=3 field_idx=-1), Slot(id=4 type=STRING
>> col_path=[51] offset=64 null=(offset=556 mask=10) slot_idx=4 field_idx=-1),
>> Slot(id=5 type=STRING col_path=[7] offset=80 null=(offset=556 mask=20)
>> slot_idx=5 field_idx=-1), Slot(id=6 type=STRING col_path=[2] offset=96
>> null=(offset=556 mask=40) slot_idx=6 field_idx=-1), Slot(id=7 type=STRING
>> col_path=[3] offset=112 null=(offset=556 mask=80) slot_idx=7 field_idx=-1),
>> Slot(id=8 type=STRING col_path=[4] offset=128 null=(offset=557 mask=1)
>> slot_idx=8 field_idx=-1), Slot(id=9 type=STRING col_path=[5] offset=144
>> null=(offset=557 mask=2) slot_idx=9 field_idx=-1), Slot(id=10 type=STRING
>> col_path=[6] offset=160 null=(offset=557 mask=4) slot_idx=10 field_idx=-1),
>> Slot(id=11 type=INT col_path=[8] offset=464 null=(offset=559 mask=20)
>> slot_idx=29 field_idx=-1), Slot(id=12 type=INT col_path=[9] offset=468
>> null=(offset=559 mask=40) slot_idx=30 field_idx=-1), Slot(id=13 type=INT
>> col_path=[10] offset=472 null=(offset=559 mask=80) slot_idx=31
>> field_idx=-1), Slot(id=14 type=INT col_path=[11] offset=476
>> null=(offset=560 mask=1) slot_idx=32 field_idx=-1), Slot(id=15 type=STRING
>> col_path=[12] offset=176 null=(offset=557 mask=8) slot_idx=11
>> field_idx=-1), Slot(id=16 type=STRING col_path=[13] offset=192
>> null=(offset=557 mask=10) slot_idx=12 field_idx=-1), Slot(id=17 type=STRING
>> col_path=[14] offset=208 null=(offset=557 mask=20) slot_idx=13
>> field_idx=-1), Slot(id=18 type=STRING col_path=[15] offset=224
>> null=(offset=557 mask=40) slot_idx=14 field_idx=-1), Slot(id=19 type=STRING
>> col_path=[16] offset=240 null=(offset=557 mask=80) slot_idx=15
>> field_idx=-1), Slot(id=20 type=STRING col_path=[17] offset=256
>> null=(offset=558 mask=1) slot_idx=16 field_idx=-1), Slot(id=21 type=INT
>> col_path=[18] offset=480 null=(offset=560 mask=2) slot_idx=33
>> field_idx=-1), Slot(id=22 type=INT col_path=[19] offset=484
>> null=(offset=560 mask=4) slot_idx=34 field_idx=-1), Slot(id=23 type=INT
>> col_path=[20] offset=488 null=(offset=560 mask=8) slot_idx=35
>> field_idx=-1), Slot(id=24 type=INT col_path=[21] offset=492
>> null=(offset=560 mask=10) slot_idx=36 field_idx=-1), Slot(id=25 type=INT
>> col_path=[22] offset=496 null=(offset=560 mask=20) slot_idx=37
>> field_idx=-1), Slot(id=26 type=INT col_path=[23] offset=500
>> null=(offset=560 mask=40) slot_idx=38 field_idx=-1), Slot(id=27 type=INT
>> col_path=[24] offset=504 null=(offset=560 mask=80) slot_idx=39
>> field_idx=-1), Slot(id=28 type=STRING col_path=[25] offset=272
>> null=(offset=558 mask=2) slot_idx=17 field_idx=-1), Slot(id=29 type=STRING
>> col_path=[26] offset=288 null=(offset=558 mask=4) slot_idx=18
>> field_idx=-1), Slot(id=30 type=INT col_path=[27] offset=508
>> null=(offset=561 mask=1) slot_idx=40 field_idx=-1), Slot(id=31 type=STRING
>> col_path=[31] offset=304 null=(offset=558 mask=8) slot_idx=19
>> field_idx=-1), Slot(id=32 type=STRING col_path=[32] offset=320
>> null=(offset=558 mask=10) slot_idx=20 field_idx=-1), Slot(id=33 type=STRING
>> col_path=[33] offset=336 null=(offset=558 mask=20) slot_idx=21
>> field_idx=-1), Slot(id=34 type=STRING col_path=[34] offset=352
>> null=(offset=558 mask=40) slot_idx=22 field_idx=-1), Slot(id=35 type=STRING
>> col_path=[35] offset=368 null=(offset=558 mask=80) slot_idx=23
>> field_idx=-1), Slot(id=36 type=INT col_path=[36] offset=512
>> null=(offset=561 mask=2) slot_idx=41 field_idx=-1), Slot(id=37 type=INT
>> col_path=[37] offset=516 null=(offset=561 mask=4) slot_idx=42
>> field_idx=-1), Slot(id=38 type=INT col_path=[38] offset=520
>> null=(offset=561 mask=8) slot_idx=43 field_idx=-1), Slot(id=39 type=INT
>> col_path=[39] offset=524 null=(offset=561 mask=10) slot_idx=44
>> field_idx=-1), Slot(id=40 type=INT col_path=[40] offset=528
>> null=(offset=561 mask=20) slot_idx=45 field_idx=-1), Slot(id=41 type=INT
>> col_path=[41] offset=532 null=(offset=561 mask=40) slot_idx=46
>> field_idx=-1), Slot(id=42 type=INT col_path=[42] offset=536
>> null=(offset=561 mask=80) slot_idx=47 field_idx=-1), Slot(id=43 type=INT
>> col_path=[43] offset=540 null=(offset=562 mask=1) slot_idx=48
>> field_idx=-1), Slot(id=44 type=INT col_path=[44] offset=544
>> null=(offset=562 mask=2) slot_idx=49 field_idx=-1), Slot(id=45 type=INT
>> col_path=[45] offset=548 null=(offset=562 mask=4) slot_idx=50
>> field_idx=-1), Slot(id=46 type=INT col_path=[46] offset=552
>> null=(offset=562 mask=8) slot_idx=51 field_idx=-1), Slot(id=47 type=STRING
>> col_path=[47] offset=384 null=(offset=559 mask=1) slot_idx=24
>> field_idx=-1), Slot(id=48 type=STRING col_path=[48] offset=400
>> null=(offset=559 mask=2) slot_idx=25 field_idx=-1), Slot(id=49 type=STRING
>> col_path=[28] offset=416 null=(offset=559 mask=4) slot_idx=26
>> field_idx=-1), Slot(id=50 type=DECIMAL(30,6) col_path=[49] offset=432
>> null=(offset=559 mask=8) slot_idx=27 field_idx=-1), Slot(id=51
>> type=DECIMAL(30,6) col_path=[50] offset=448 null=(offset=559 mask=10)
>> slot_idx=28 field_idx=-1)] tuple_path=[])
>> I0718 18:42:22.680297 51333 query-state.cc:404] Executing instance.
>> instance_id=2e4a3fbec0d7d721:2ec73c1c00000009 fragment_idx=0
>> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=1
>> I0718 18:42:22.680315 51139 coordinator.cc:370] started execution on 5
>> backends for query_id=2e4a3fbec0d7d721:2ec73c1c00000000
>> I0718 18:42:22.680457 51334 query-state.cc:404] Executing instance.
>> instance_id=2e4a3fbec0d7d721:2ec73c1c00000004 fragment_idx=1
>> per_fragment_instance_idx=4 coord_state_idx=3 #in-flight=2
>> I0718 18:42:22.680583 51334 hdfs-scan-node.cc:190] Max row batch queue
>> size for scan node '0' in fragment instance
>> '2e4a3fbec0d7d721:2ec73c1c00000004': 40
>> I0718 18:42:22.687943 51332 query-exec-mgr.cc:155] ReleaseQueryState():
>> query_id=2e4a3fbec0d7d721:2ec73c1c00000000 refcnt=4
>> I0718 18:42:22.689262 51338 coordinator.cc:789] Coordinator waiting for
>> backends to finish, 5 remaining. query_id=2e4a3fbec0d7d721:2ec73c1c00000000
>> I0718 18:51:07.797159 51597 webserver.cc:362] Webserver: error reading:
>> Resource temporarily unavailable
>> I0718 19:22:57.687764 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 47602.1us
>> I0718 19:23:27.786094 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 49243.8us
>> I0718 19:27:28.426831 180697 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
>> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
>> blocked reactor thread for 47536.6us
>> I0718 19:29:28.859411 232946 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request
>> call id 5453218) took 518541ms. Request Metrics: {}
>> I0718 19:29:28.859459 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request
>> call id 5432432) took 518541ms. Request Metrics: {}
>> I0718 19:29:28.859504 232946 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request
>> call id 5500254) took 518534ms. Request Metrics: {}
>> I0718 19:29:28.859417 232945 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request
>> call id 5513959) took 518536ms. Request Metrics: {}
>> I0718 19:29:28.859589 232941 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request
>> call id 5485666) took 518524ms. Request Metrics: {}
>> I0718 20:09:09.275442 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 52958.3us
>> I0718 20:09:29.338707 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 49366.4us
>> I0718 20:14:50.273746 180697 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
>> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
>> blocked reactor thread for 47568.8us
>> I0718 20:15:20.342495 180697 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.59:7050
>> (realtimeanalysis-kudu-05-10-8-50-59), user_credentials={real_user=hive}}
>> blocked reactor thread for 51994us
>> I0718 20:15:56.003931 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request
>> call id 5647298) took 510394ms. Request Metrics: {}
>> I0718 20:15:56.004006 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request
>> call id 5626571) took 510394ms. Request Metrics: {}
>> I0718 20:15:56.004089 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request
>> call id 5661461) took 510386ms. Request Metrics: {}
>> I0718 20:15:56.004143 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request
>> call id 5601664) took 510383ms. Request Metrics: {}
>> I0718 20:15:56.004218 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request
>> call id 5577825) took 510328ms. Request Metrics: {}
>> I0718 21:56:51.345875 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 61974.1us
>> I0718 21:57:41.519013 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.58:7050
>> (realtimeanalysis-kudu-04-10-8-50-58), user_credentials={real_user=hive}}
>> blocked reactor thread for 48016.9us
>> I0718 22:07:02.248558 180696 outbound_call.cc:286] RPC callback for RPC
>> call kudu.tserver.TabletServerService.Write -> {remote=10.8.50.56:7050
>> (realtimeanalysis-kudu-02-10-8-50-56), user_credentials={real_user=hive}}
>> blocked reactor thread for 61139.4us
>> I0718 22:07:56.005770 232949 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.56:49430 (request
>> call id 5804566) took 839901ms. Request Metrics: {}
>> I0718 22:07:56.005787 232937 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.59:36460 (request
>> call id 5749547) took 839899ms. Request Metrics: {}
>> I0718 22:07:56.005823 232934 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.57:60524 (request
>> call id 5820240) took 839850ms. Request Metrics: {}
>> I0718 22:07:56.005831 232940 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.55:34302 (request
>> call id 5712080) took 839898ms. Request Metrics: {}
>> I0718 22:07:56.005977 232939 rpcz_store.cc:255] Call
>> impala.DataStreamService.TransmitData from 10.8.50.58:52438 (request
>> call id 5768794) took 839898ms. Request Metrics: {}
>>
>> *I will write 632 million rows to kudu, normally it's about 50 thousand/s
>> , but today after write about 270 million , then write speed become very
>> slow , I don't know what was wrong.*
>>
>> 2019-07-18
>> ------------------------------
>> lk_hadoop
>>
>
