Right. Can you try to write the segments after cherry picking #2796? On Wed, Oct 17, 2018 at 3:21 PM xm_zzc <[email protected]> wrote:
> Hi Kunal Kapoor: > 1. No; > 2. query unsuccessfully, I use Carbon SDK Reader to read that wrong > segment and it failed too. > 3. the schema of the table: > > | rt | string > > | timestamp_1min | bigint > > | timestamp_5min | bigint > > | timestamp_1hour | bigint > > | customer_id | bigint > > | transport_id | bigint > > | transport_code | string > > | tcp_udp | int > > | pre_hdt_id | string > > | hdt_id | string > > | status | int > > | is_end_user | int > > | transport_type | string > > | transport_type_nam | string > > | fcip | string > > | host | string > > | cip | string > > | code | int > > | conn_status | int > > | recv | bigint > > | send | bigint > > | msec | bigint > > | dst_prefix | string > > | next_type | int > > | next | string > > | hdt_sid | string > > | from_endpoint_type | int > > | to_endpoint_type | int > > | fcip_view | string > > | fcip_country | string > > | fcip_province | string > > | fcip_city | string > > | fcip_longitude | string > > | fcip_latitude | string > > | fcip_node_name | string > > | fcip_node_name_cn | string > > | host_view | string > > | host_country | string > > | host_province | string > > | host_city | string > > | host_longitude | string > > | host_latitude | string > > | cip_view | string > > | cip_country | string > > | cip_province | string > > | cip_city | string > > | cip_longitude | string > > | cip_latitude | string > > | cip_node_name | string > > | cip_node_name_cn | string > > | dtp_send | string > > | client_port | int > > | server_ip | string > > | server_port | int > > | state | string > > | response_code | int > > | access_domain | string > > | valid | int > > | min_batch_time | bigint > > | update_time | bigint > > | | > > | ##Detailed Table Information | > > | Database Name | hdt_sys > > | Table Name | transport_access_log > > | CARBON Store Path | hdfs://hdtcluster/carbon_store > > | Comment | > > | Table Block Size | 512 MB > > | Table Data Size | 777031634135 > > | Table Index Size | 72894232 > > | Last Update Time | 1539769299990 > > | SORT_SCOPE | local_sort > > | Streaming | true > > | MAJOR_COMPACTION_SIZE | 4096 > > | AUTO_LOAD_MERGE | true > > | COMPACTION_LEVEL_THRESHOLD | 2,8 > > | | > > | ##Detailed Column property | > > | ADAPTIVE | > > | SORT_COLUMNS | > is_end_user,status,customer_id,access_domain,transport_id,timestamp_1hour,timestamp_1min,conn_status,pre_hdt_id,tcp_udp,transport_code,fcip,cip > > | > > > I think maybe the data are written wrongly when generated segment because > of > the issue : MemoryBlock is cleaned by some other thread and result in > wrong data when write data. > Now I deleted the wrong segment and then patch the PR#2796, and continue to > run stream app. If the exception no longer occur, it proves that PR#2796 > works. Right? > > > > > -- > Sent from: > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ >
