[
https://issues.apache.org/jira/browse/IMPALA-10345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238307#comment-17238307
]
ASF subversion and git services commented on IMPALA-10345:
----------------------------------------------------------
Commit 96decf535ba9ae7a6b295012d2be85c3a1ae892c in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=96decf5 ]
IMPALA-10345: Impala hits DCHECK in parquet-column-stats.inline.h
During Parquet file writing, a DCHECK checks if row group stats have
copied the min/max string values into their internal buffers. This check
is at the finalization of each page. The copying of the string values
happened at the end of each row batch.
Thus, if a row batch spans over multiple pages then the min/max
string values don't get copied by the end of the page. Since the
memory is attached to the row batch this isn't really an error.
As a workaround this commit also copies the min/max string values
at the end of the page if they haven't been copied yet.
Testing
* Added e2e test
Change-Id: I4289bd743e951cc4c607d5a5ea75d27825a1c12b
Reviewed-on: http://gerrit.cloudera.org:8080/16771
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Impala hits DCHECK in parquet-column-stats.inline.h result in Impala Daemon
> breakdown
> -------------------------------------------------------------------------------------
>
> Key: IMPALA-10345
> URL: https://issues.apache.org/jira/browse/IMPALA-10345
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.3.0
> Reporter: liangwei wei
> Assignee: Zoltán Borók-Nagy
> Priority: Blocker
> Attachments: a4550b1a-2dce-4fb7-e4c4aa8a-14573111.dmp
>
>
> Impala Daemon sometimes will breakdown with fatal Exception like this:
> {code:java}
> F1103 00:53:52.158277 30701 parquet-column-stats.inline.h:240]
> 4849ebcf56de71cb:1f9acb4800000005] Check failed:
> static_cast<void*>(prev_page_min_value_.ptr) !=
> static_cast<void*>(cs->min_value_.ptr) (0x7f9e8ef7c8e0 vs. 0x7f9e8ef7c8e0)
> {code}
> I think the error was caused by write parquet. The SQL which cause this
> error is：
> I1102 23:58:01.681974 30677 Frontend.java:1243]
> 4849ebcf56de71cb:1f9acb4800000000] Analyzing query: insert into
> vehicle_pic_info_hdfs_all partition (year ,month ,day ) select
> card_position,plate_rect,vehicle_speed,vice_pilot_sunvisor_confidence,crossing_id,link_face_vehicle_id,envprosign_value,recognition_sign,vehicle_lamp,illegal_traffic_event,lane_no,vehicle_color_value,lane_vehicle_dir,datapool_create_time,event_type,front_child_value,vice_pilot_safebelt_value,pilot_safebelt_value,vehicle_len,label_value,detect_result,vehicle_head_value,pendant_value,luggage_rack_value,prtday,tag_value,video_taskId,vehicle_is_main_vehicle,datapool_platform_index,card_num_value,target_sub_url,license_bright,tricycle_canopy_value,device_alias_name,vehicle_state_value,vehicle_color_depth,crossing_index_code,vehicle_entry_exiting_status,copilot_value,vehicle_sub_logo_value,device_latitude,sub_feature,latitude,sunroof_value,main_task_id,plate_pic_url,spare_tire_value,label_num_value,error_msg,event_state,tempplate_no_value,pass_time,vehicle_type,link_vehicle_mac_id,vice_pilot_sunvisor_value,muck_truck_value,decoration_position,label_position,model_data,fix_pass_time,vice_pilot_url,plate_type_value,algorithm_id,camera_index_code,bkg_url,plate_no_value,device_direction,lng_lat_status,eventType_yyyyMMddHHmmss_l,bkg_pic_md5,rowkey,crossing_type,longitude,direction_index,source_from,recv_time,area_code,pilot_sunvisor_value,vehicle_logo_value,vehicle_spray_painted_value,event_desc,send_time,decoration_value,error_code,device_ip_address,pdvs_value,pass_id,camera_address,time_difference,card_value,rect,vice_pilot_position,check_result,mac_address,plate_color_value,pendant_position,pdvs_position,tissue_box_position,device_index_code,vehicle_model_value,device_longitude,eco_label,multi_vehicle,stream_recy_delay,tissue_box_value,uphone_value,datapool_update_time,dangmark_value,pilot_position,cover_plate_value,card_type_value,cast(substr(cast(prtday
> as string),1,4) as int) as year,cast(substr(cast(prtday as string),1,6) as
> int) as month,cast(substr(cast(prtday as string),1,8) as int) as day from
> vehicle_pic_info_all where 20201028 <= prtday AND prtday < 20201029 db:
> default
>
> Minidump file is attached
>
> *version info:*
> impalad version 3.3.0-RELEASE DEBUG (build
> 0f840c5a0f5e673c67cbd482e62065fd47b98e1a)
> Built on Wed Dec 11 15:26:16 CST 2019
> Build Flags: is_ndebug=false cmake_build_type=DEBUG library_link_type=DYNAMIC
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
