zhangyue19921010 commented on pull request #10608:
URL: https://github.com/apache/druid/pull/10608#issuecomment-741293890
> If Historical crashed in the process of downloading/creating segment files
before `info_dir file` created, Historical would not try to load this segments
when startup lazily. And the loading action of this segment will be triggered
by Coordinator which is a unLazy action.
> Un-lazy loading action will throw exception because of damaged segment
files.
>
> ```
> 2020-11-26T05:37:45,121 INFO [SimpleDataSegmentChangeHandler-26]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Loading segment
traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3
> 2020-11-26T05:37:45,170 WARN [SimpleDataSegmentChangeHandler-26]
org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - No path to
unannounce
segment[traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3]
> 2020-11-26T05:37:45,171 INFO [SimpleDataSegmentChangeHandler-26]
org.apache.druid.server.SegmentManager - Told to delete a queryable on
dataSource[traffic__ops_feed__realtime__second__dev__lazy__test] for
interval[2020-11-26T03:00:00.000Z/2020-11-26T04:00:00.000Z] and
version[2020-11-26T03:30:17.710Z] that I don't have.
> 2020-11-26T05:37:45,171 INFO [SimpleDataSegmentChangeHandler-26]
org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Deleting
directory[/var/druid/segment-cache/traffic__ops_feed__realtime__second__dev__lazy__test/2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z/2020-11-26T03:30:17.710Z/3]
> 2020-11-26T05:37:45,172 WARN [SimpleDataSegmentChangeHandler-26]
org.apache.druid.segment.loading.StorageLocation -
SegmentDir[/var/druid/segment-cache/traffic__ops_feed__realtime__second__dev__lazy__test/2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z/2020-11-26T03:30:17.710Z/3]
is not found under this location[/var/druid/segment-cache]
> 2020-11-26T05:37:45,172 WARN [SimpleDataSegmentChangeHandler-26]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Unable to delete
segmentInfoCacheFile[/var/druid/segment-cache/info_dir/traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3]
> 2020-11-26T05:37:45,177 ERROR [SimpleDataSegmentChangeHandler-26]
org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load
segment for dataSource:
{class=org.apache.druid.server.coordination.SegmentLoadDropHandler,
exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException,
exceptionMessage=Exception loading
segment[traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3],
segment=DataSegment{binaryVersion=9,
id=traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3,
loadSpec={type=>s3_zip, bucket=>pqm-druid-dev,
key=>rtstorage/segments/traffic__ops_feed__realtime__second__dev__lazy__test/2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z/2020-11-26T03:30:17.710Z/3/630170ed-967b-4158-85a7-d2abc284e738/index.zip,
S3Schema=>s3n}, dimensions=[video_cro_network_id, video_cro_network_name,
distributor
_network_id, distributor_network_name, profile_id, profile_name,
is_active_device, is_filtered, service_type, platform, ad_unit_type, country,
country_name, state, state_name, dma, dma_name, syscode, syscode_name,
tv_network_id, tv_network_name, linear_campaign_type, spot_type],
metrics=[count, req_ad_request, req_ad_request_with_video_slot,
req_ad_request_with_midroll_slot, req_resp_time_lt_100ms,
req_resp_time_lt_300ms, req_resp_time_lt_500ms, req_resp_time_lt_1500ms,
req_resp_time_gt_1500ms, req_empty_response,
req_empty_response_with_video_slot, req_empty_response_with_midroll_slot,
req_err_no_profile, req_err_no_mac_address, req_err_no_signal_id,
req_err_syscode_not_found, req_err_station_not_found,
req_err_schedule_not_found, req_err_signal_no_bind_break,
req_err_break_duration_invalid, req_err_break_no_schedule_ad,
req_err_schedule_creative_validation_failed, slot_avails, slot_unfilled_avails,
ad_delivered_ad_primary, ad_delivered_ad_fallback, ad_delivered_ad,
ad_err_full_ava
il_no_variant_segment, ad_err_fallback_to_evergreen,
ad_err_inactive_addressable_order, ack_ad_impression, ack_ad_complete,
ack_ad_first_quartile, ack_ad_mid_point, ack_ad_third_quartile, ack_ad_click,
ack_err_psn_message_validation_failed, ack_err_psn_asset_info_invalid,
ack_err_psn_unknown_message_reference, ack_err_psn_timeout,
ack_err_psn_insertion_point_time_exceeded,
ack_err_psn_abnormal_termination_of_playout, ack_err_psn_bit_rate_mismatch,
ack_err_adm_e_no_ad, ack_err_adm_e_timeout, ack_err_adm_e_security,
ack_err_adm_e_3p_comp, ack_err_adm_e_unknown, ack_err_adm_e_io,
ack_err_adm_e_no_render, ack_err_adm_e_parse, ack_err_adm_e_device_limit,
ack_err_adm_e_render_init, ack_err_vast_100, ack_err_vast_202,
ack_err_vast_300, ack_err_vast_301, ack_err_vast_302, ack_err_vast_303,
ack_err_vast_400, ack_err_vast_402, ack_err_vast_403, ack_err_vast_900],
shardSpec=NumberedShardSpec{partitionNum=3, partitions=0},
lastCompactionState=null, size=50079060}}
> org.apache.druid.segment.loading.SegmentLoadingException: Exception
loading
segment[traffic__ops_feed__realtime__second__dev__lazy__test_2020-11-26T03:00:00.000Z_2020-11-26T04:00:00.000Z_2020-11-26T03:30:17.710Z_3]
> at
org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:263)
~[druid-server-0.17.1.jar:0.17.1]
> at
org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:307)
~[druid-server-0.17.1.jar:0.17.1]
> at
org.apache.druid.server.coordination.SegmentLoadDropHandler$1.lambda$addSegment$1(SegmentLoadDropHandler.java:513)
~[druid-server-0.17.1.jar:0.17.1]
> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_221]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_221]
> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_221]
> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[?:1.8.0_221]
> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_221]
> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_221]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
> Caused by: java.lang.NullPointerException
> at
org.apache.druid.common.utils.SerializerUtils.readString(SerializerUtils.java:61)
~[druid-core-0.17.1.jar:0.17.1]
> at
org.apache.druid.segment.IndexIO$V9IndexLoader.deserializeColumn(IndexIO.java:666)
~[druid-processing-0.17.1.jar:0.17.1]
> at
org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:617)
~[druid-processing-0.17.1.jar:0.17.1]
> at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:194)
~[druid-processing-0.17.1.jar:0.17.1]
> at
org.apache.druid.segment.loading.MMappedQueryableSegmentizerFactory.factorize(MMappedQueryableSegmentizerFactory.java:48)
~[druid-processing-0.17.1.jar:0.17.1]
> at
org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:150)
~[druid-server-0.17.1.jar:0.17.1]
> at
org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:198)
~[druid-server-0.17.1.jar:0.17.1]
> at
org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:157)
~[druid-server-0.17.1.jar:0.17.1]
> at
org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:259)
~[druid-server-0.17.1.jar:0.17.1]
> ... 9 more
> ```
I just make a PR(https://github.com/apache/druid/pull/10650) to solve the
problem described in the quote and tested on Druid Cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
