[
https://issues.apache.org/jira/browse/IMPALA-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-5680.
-----------------------------------
Resolution: Cannot Reproduce
> Avro scanner still crashing when HDFS seek fails
> ------------------------------------------------
>
> Key: IMPALA-5680
> URL: https://issues.apache.org/jira/browse/IMPALA-5680
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.8.0
> Reporter: Pavas Garg
> Priority: Major
>
> We had a previous Impala Jira
> https://issues.apache.org/jira/browse/IMPALA-4223
> Impala daemon didn't crash but stopped working.
> I0712 16:21:25.917135 11642 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/data/XXX_ETL-f2h/xxx_logs/dt=2017-07-12/000000_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.917220 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf400000000: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0
> at 536859282
> I0712 16:21:25.917259 30331 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0
>
> Error(255): Unknown error 255
> I0712 16:21:25.928112 11640 status.cc:114] Error seeking to 536870912 in
> file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0
>
> Error(255): Unknown error 255
> @ 0x8377f9 (unknown)
> @ 0xa606fb (unknown)
> @ 0xa5bbeb (unknown)
> @ 0xa5c064 (unknown)
> @ 0xbd2cd9 (unknown)
> @ 0xbd36b4 (unknown)
> @ 0xe1b65a (unknown)
> @ 0x3a3b007851 (unknown)
> @ 0x3a3a8e894d (unknown)
> I0712 16:21:25.928170 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf400000000: Problem parsing file
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0
> at 536867240
> I0712 16:21:25.928185 30332 runtime-state.cc:217] Error from query
> 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file:
> hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0
>
> Error(255): Unknown error 255
> I0712 17:09:47.028455 11343 authentication.cc:499] Registering
> impala/[email protected], keytab file
> /var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala.keytab
> ---
> From HDFS namenode logs:
> ---
> 2017-07-12 16:20:43,552 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449009_1099897417390{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-e5b8a4fb-a840-4de2-bd5c-d0de019cb2c8:NORMAL:10.13.68.32:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-5373c471-3289-4563-bff1-65b01c7c906b:NORMAL:10.13.68.30:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-e5988516-2067-4339-b465-f5180869f73b:NORMAL:10.13.68.28:1004|XXX]]}
>
> 2017-07-12 16:21:21,599 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0
> is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_-1168871812_1
> 2017-07-12 16:21:21,729 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/-ext-10001/tmpstats-2
> is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_261413855_1
> 2017-07-12 16:21:24,269 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocateBlock:
> /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0.
> BP-707861034-10.13.68.11-1367011211601
> blk_1319449246_1099897417627{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-15ccd60f-6356-43b0-ae31-11989433f66f:NORMAL:10.13.68.46:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-d40c69bb-7882-42de-a0a7-4a784f4aa14a:NORMAL:10.13.68.73:1004|XXX],
>
> ReplicaUnderConstruction[[DISK]DS-395d3bce-2c3e-4922-9d94-b182dacbd14a:NORMAL:10.13.68.33:1004|XXX]]}
>
> 2017-07-12 16:21:24,396 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> completeFile:
> /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0
> is closed by DFSClient_attempt_1499250888632_48578_m_000002_0_-1637609232_1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
