Quanlong Huang created IMPALA-11593:
---------------------------------------
Summary: TestMemReservations.test_per_backend_min_reservation
failed by NullPointerException from libhdfs in S3 builds
Key: IMPALA-11593
URL: https://issues.apache.org/jira/browse/IMPALA-11593
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
Saw this failure on an S3 build:
{noformat}
custom_cluster/test_mem_reservations.py:102: in test_per_backend_min_reservation
assert t.error is None
E assert 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255):
Unknown error 255\nRoot cause: NullPointerException: \n\n' is None
E + where 'ImpalaBeeswaxException:\n Query aborted:Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera....warehouse/alltypes/year=2009/month=9/090901.txt\nError(255):
Unknown error 255\nRoot cause: NullPointerException: \n\n' =
<QuerySubmitThread(Thread-165, stopped 140272709113600)>.error
{noformat}
Impalad logs for the query:
{noformat}
I0915 03:12:33.839942 21677 impala-server.cc:1333]
09439d05a2468038:3816f0f200000000] Registered query
query_id=09439d05a2468038:3816f0f200000000
session_id=874c5100c59607af:a86e04c8f62bb9a9
I0915 03:12:33.889168 21677 Frontend.java:1628]
09439d05a2468038:3816f0f200000000] Analyzing query: select max(t.c1),
avg(t.c2), min(t.c3), avg(c4), avg(c5), avg(c6)
from (select
max(tinyint_col) over (order by int_col) c1,
avg(tinyint_col) over (order by smallint_col) c2,
min(tinyint_col) over (order by smallint_col desc) c3,
rank() over (order by int_col desc) c4,
dense_rank() over (order by bigint_col) c5,
first_value(tinyint_col) over (order by bigint_col desc) c6
from functional.alltypes) t; db: default
I0915 03:12:33.981251 21677 FeSupport.java:315]
09439d05a2468038:3816f0f200000000] Requesting prioritized load of table(s):
functional.alltypes
I0915 03:12:33.986737 21677 thrift-util.cc:99]
09439d05a2468038:3816f0f200000000] TSocket::open() connect() <Host: localhost
Port: 26000>: Connection refused
I0915 03:12:34.582643 21677 BaseAuthorizationChecker.java:113]
09439d05a2468038:3816f0f200000000] Authorization check took 693 ms
I0915 03:12:34.582674 21677 Frontend.java:1671]
09439d05a2468038:3816f0f200000000] Analysis and authorization finished.
I0915 03:12:34.723712 21208 control-service.cc:148]
4a4ebd3b7575254c:eb71cd8000000000] ExecQueryFInstances():
query_id=4a4ebd3b7575254c:eb71cd8000000000
coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000
#instances=1
I0915 03:12:34.738032 21758 query-state.cc:942]
4a4ebd3b7575254c:eb71cd8000000002] Executing instance.
instance_id=4a4ebd3b7575254c:eb71cd8000000002 fragment_idx=1
per_fragment_instance_idx=1 coord_state_idx=1 #in-flight=1
I0915 03:12:34.850791 21820 admission-controller.cc:1819]
09439d05a2468038:3816f0f200000000] Trying to admit
id=09439d05a2468038:3816f0f200000000 in pool_name=default-pool
executor_group_name=default per_host_mem_estimate=1.34 GB
dedicated_coord_mem_estimate=1.10 GB max_requests=-1 max_queued=200
max_mem=-1.00 B
I0915 03:12:34.850811 21820 admission-controller.cc:1827]
09439d05a2468038:3816f0f200000000] Stats: agg_num_running=1, agg_num_queued=0,
agg_mem_reserved=1.56 GB, local_host(local_mem_admitted=0,
num_admitted_running=0, num_queued=0, backend_mem_reserved=192.46 MB,
topN_query_stats: queries=[4a4ebd3b7575254c:eb71cd8000000000],
total_mem_consumed=192.46 MB, fraction_of_pool_total_mem=1; pool_level_stats:
num_running=1, min=192.46 MB, max=192.46 MB, pool_total_mem=192.46 MB,
average_per_query=192.46 MB)
I0915 03:12:34.850852 21820 admission-controller.cc:1218]
09439d05a2468038:3816f0f200000000] Admitting query
id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.850939 21820 impala-server.cc:2159]
09439d05a2468038:3816f0f200000000] Registering query locations
I0915 03:12:34.850998 21820 coordinator.cc:150]
09439d05a2468038:3816f0f200000000] Exec()
query_id=09439d05a2468038:3816f0f200000000 stmt=select max(t.c1), avg(t.c2),
min(t.c3), avg(c4), avg(c5), avg(c6)
from (select
max(tinyint_col) over (order by int_col) c1,
avg(tinyint_col) over (order by smallint_col) c2,
min(tinyint_col) over (order by smallint_col desc) c3,
rank() over (order by int_col desc) c4,
dense_rank() over (order by bigint_col) c5,
first_value(tinyint_col) over (order by bigint_col desc) c6
from functional.alltypes) t;
I0915 03:12:34.851434 21820 coordinator.cc:474]
09439d05a2468038:3816f0f200000000] starting execution on 3 backends for
query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.856995 21208 control-service.cc:148]
09439d05a2468038:3816f0f200000000] ExecQueryFInstances():
query_id=09439d05a2468038:3816f0f200000000
coord=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
#instances=2
I0915 03:12:34.858456 21820 coordinator.cc:533]
09439d05a2468038:3816f0f200000000] started execution on 3 backends for
query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:34.860503 21841 query-state.cc:942]
09439d05a2468038:3816f0f200000002] Executing instance.
instance_id=09439d05a2468038:3816f0f200000002 fragment_idx=1
per_fragment_instance_idx=1 coord_state_idx=0 #in-flight=2
I0915 03:12:34.860591 21843 query-state.cc:942]
09439d05a2468038:3816f0f200000000] Executing instance.
instance_id=09439d05a2468038:3816f0f200000000 fragment_idx=0
per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=3
I0915 03:12:35.057634 21208 coordinator.cc:1032] Backend completed:
host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27002
remaining=3 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.057649 21208 coordinator-backend-state.cc:371]
query_id=09439d05a2468038:3816f0f200000000: first in-progress backend:
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
I0915 03:12:35.149704 21208 coordinator.cc:1032] Backend completed:
host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27000
remaining=2 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.149719 21208 coordinator-backend-state.cc:371]
query_id=09439d05a2468038:3816f0f200000000: first in-progress backend:
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
I0915 03:12:35.106189 21377 status.cc:71] Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to
open HDFS file
s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
@ 0x1f096f4 impala::Status::Status()
@ 0x1f1dfdb impala::Status::Status()
@ 0x2ea28bb impala::io::OpenHdfsFileOp::Execute()
@ 0x2ea3884 impala::SynchronousWorkItem::WorkerExecute()
@ 0x2ea4071 impala::SynchronousThreadPool::Worker()
@ 0x2ea6a89
boost::detail::function::void_function_invoker2<>::invoke()
@ 0x2ea6734 boost::function2<>::operator()()
@ 0x2ea56f1 impala::ThreadPool<>::WorkerThread()
@ 0x2ea8819 boost::_mfi::mf1<>::operator()()
@ 0x2ea8645 boost::_bi::list2<>::operator()<>()
@ 0x2ea82d2 boost::_bi::bind_t<>::operator()()
@ 0x2ea7eaf
boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x221e4f7 boost::function0<>::operator()()
@ 0x29a898f impala::Thread::SuperviseThread()
@ 0x29b12f0 boost::_bi::list5<>::operator()<>()
@ 0x29b1214 boost::_bi::bind_t<>::operator()()
@ 0x29b11d5 boost::detail::thread_data<>::run()
@ 0x42018b1 thread_proxy
@ 0x7f4590612ea4 start_thread
@ 0x7f458d00cb0c __clone
I0915 03:12:35.195694 21854 hdfs-scan-node.cc:515]
09439d05a2468038:3816f0f200000002] Scan node (id=0) ran into a parse error for
scan range
s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt(0:20179).
Processed 0 bytes.
I0915 03:12:35.196115 21841 query-state.cc:951]
09439d05a2468038:3816f0f200000002] Instance completed.
instance_id=09439d05a2468038:3816f0f200000002 #in-flight=1
status=DISK_IO_ERROR: Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to
open HDFS file
s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.196135 21829 query-state.cc:462]
09439d05a2468038:3816f0f200000000] UpdateBackendExecState(): last report for
09439d05a2468038:3816f0f200000000
I0915 03:12:35.198632 21208 coordinator.cc:1032] Backend completed:
host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
remaining=1 query_id=09439d05a2468038:3816f0f200000000
I0915 03:12:35.198649 21208 coordinator.cc:752] ExecState: query
id=09439d05a2468038:3816f0f200000000
finstance=09439d05a2468038:3816f0f200000002 on
host=impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001
(EXECUTING -> ERROR) status=Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to
open HDFS file
s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.198689 21208 coordinator-backend-state.cc:974]
query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27001: Not
cancelling because the backend is already done: Disk I/O error on
impala-ec2-centos79-m6i-4xlarge-ondemand-1db1.vpc.cloudera.com:27001: Failed to
open HDFS file
s3a://impala-test-uswest2-1/test-warehouse/alltypes/year=2009/month=9/090901.txt
Error(255): Unknown error 255
Root cause: NullPointerException:
I0915 03:12:35.198695 21208 coordinator-backend-state.cc:974]
query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27000: Not
cancelling because the backend is already done:
I0915 03:12:35.198702 21208 coordinator-backend-state.cc:974]
query_id=09439d05a2468038:3816f0f200000000 target backend=127.0.0.1:27002: Not
cancelling because the backend is already done:
I0915 03:12:35.198706 21208 coordinator.cc:999] CancelBackends()
query_id=09439d05a2468038:3816f0f200000000, tried to cancel 0 backends
I0915 03:12:35.198752 21208 coordinator.cc:1375] Release admission control
resources for query_id=09439d05a2468038:3816f0f200000000
{noformat}
This could be due to the same cause of IMPALA-11592. Maybe there is an issue
inside the hdfs client. Hadoop version: hadoop-3.1.1.7.2.16.0-171
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
