[ https://issues.apache.org/jira/browse/IMPALA-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-5680. ----------------------------------- Resolution: Cannot Reproduce > Avro scanner still crashing when HDFS seek fails > ------------------------------------------------ > > Key: IMPALA-5680 > URL: https://issues.apache.org/jira/browse/IMPALA-5680 > Project: IMPALA > Issue Type: Bug > Components: Backend > Affects Versions: Impala 2.8.0 > Reporter: Pavas Garg > Priority: Major > > We had a previous Impala Jira > https://issues.apache.org/jira/browse/IMPALA-4223 > Impala daemon didn't crash but stopped working. > I0712 16:21:25.917135 11642 status.cc:114] Error seeking to 536870912 in > file: > hdfs://nameservice1/envs/production/XXX/data/XXX_ETL-f2h/xxx_logs/dt=2017-07-12/000000_0 > > Error(255): Unknown error 255 > @ 0x8377f9 (unknown) > @ 0xa606fb (unknown) > @ 0xa5bbeb (unknown) > @ 0xa5c064 (unknown) > @ 0xbd2cd9 (unknown) > @ 0xbd36b4 (unknown) > @ 0xe1b65a (unknown) > @ 0x3a3b007851 (unknown) > @ 0x3a3a8e894d (unknown) > I0712 16:21:25.917220 30331 runtime-state.cc:217] Error from query > 37402db5e53ce683:96f60bf400000000: Problem parsing file > hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0 > at 536859282 > I0712 16:21:25.917259 30331 runtime-state.cc:217] Error from query > 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file: > hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000000_0 > > Error(255): Unknown error 255 > I0712 16:21:25.928112 11640 status.cc:114] Error seeking to 536870912 in > file: > hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0 > > Error(255): Unknown error 255 > @ 0x8377f9 (unknown) > @ 0xa606fb (unknown) > @ 0xa5bbeb (unknown) > @ 0xa5c064 (unknown) > @ 0xbd2cd9 (unknown) > @ 0xbd36b4 (unknown) > @ 0xe1b65a (unknown) > @ 0x3a3b007851 (unknown) > @ 0x3a3a8e894d (unknown) > I0712 16:21:25.928170 30332 runtime-state.cc:217] Error from query > 37402db5e53ce683:96f60bf400000000: Problem parsing file > hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0 > at 536867240 > I0712 16:21:25.928185 30332 runtime-state.cc:217] Error from query > 37402db5e53ce683:96f60bf400000000: Error seeking to 536870912 in file: > hdfs://nameservice1/envs/production/XXX/out-data/XXX_ETL-f2h/XXX_logs/dt=2017-07-12/000002_0 > > Error(255): Unknown error 255 > I0712 17:09:47.028455 11343 authentication.cc:499] Registering > impala/hostn...@domain.com, keytab file > /var/run/cloudera-scm-agent/process/XXX-impala-IMPALAD/impala.keytab > --- > From HDFS namenode logs: > --- > 2017-07-12 16:20:43,552 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* > allocateBlock: > /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0. > BP-707861034-10.13.68.11-1367011211601 > blk_1319449009_1099897417390{blockUCState=UNDER_CONSTRUCTION, > primaryNodeIndex=-1, > replicas=[ReplicaUnderConstruction[[DISK]DS-e5b8a4fb-a840-4de2-bd5c-d0de019cb2c8:NORMAL:10.13.68.32:1004|XXX], > > ReplicaUnderConstruction[[DISK]DS-5373c471-3289-4563-bff1-65b01c7c906b:NORMAL:10.13.68.30:1004|XXX], > > ReplicaUnderConstruction[[DISK]DS-e5988516-2067-4339-b465-f5180869f73b:NORMAL:10.13.68.28:1004|XXX]]} > > 2017-07-12 16:21:21,599 INFO org.apache.hadoop.hdfs.StateChange: DIR* > completeFile: > /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/_task_tmp.-ext-10000/_tmp.000002_0 > is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_-1168871812_1 > 2017-07-12 16:21:21,729 INFO org.apache.hadoop.hdfs.StateChange: DIR* > completeFile: > /envs/production/XXX/out-data/XXX_ETL-f2h/z_tmp_XXX_logs/dt=2017-07-12/.hive-staging_hive_2017-07-12_16-11-04_471_7577440609227768101-1/-ext-10001/tmpstats-2 > is closed by DFSClient_attempt_1499250888632_48550_r_000002_0_261413855_1 > 2017-07-12 16:21:24,269 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* > allocateBlock: > /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0. > BP-707861034-10.13.68.11-1367011211601 > blk_1319449246_1099897417627{blockUCState=UNDER_CONSTRUCTION, > primaryNodeIndex=-1, > replicas=[ReplicaUnderConstruction[[DISK]DS-15ccd60f-6356-43b0-ae31-11989433f66f:NORMAL:10.13.68.46:1004|XXX], > > ReplicaUnderConstruction[[DISK]DS-d40c69bb-7882-42de-a0a7-4a784f4aa14a:NORMAL:10.13.68.73:1004|XXX], > > ReplicaUnderConstruction[[DISK]DS-395d3bce-2c3e-4922-9d94-b182dacbd14a:NORMAL:10.13.68.33:1004|XXX]]} > > 2017-07-12 16:21:24,396 INFO org.apache.hadoop.hdfs.StateChange: DIR* > completeFile: > /tmp/hive/XXX_prod/f890d0a0-f1bc-4fcd-a828-f882a5aa81e7/hive_2017-07-12_16-14-29_668_7405085067336660683-1/_task_tmp.-mr-10006/_tmp.000002_0 > is closed by DFSClient_attempt_1499250888632_48578_m_000002_0_-1637609232_1 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org