[
https://issues.apache.org/jira/browse/IMPALA-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16749143#comment-16749143
]
Michael Brown commented on IMPALA-8091:
---------------------------------------
The failure isn't easy to triage from a console log. There, it looks like:
{noformat}
11:40:19 11:40:19 Error executing impala SQL:
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
See:
/data/jenkins/workspace/impala-asf-master-core/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
{noformat}
That log looks like this:
{noformat}
Traceback (most recent call last):
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/bin/load-data.py",
line 207, in exec_impala_query_from_file
result = impala_client.execute(query)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 182, in execute
handle = self.__execute_query(query_string.strip(), user=user)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 357, in __execute_query
handle = self.execute_query_async(query_string, user=user)
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 351, in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
File
"/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/tests/beeswax/impala_beeswax.py",
line 509, in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
ImpalaBeeswaxException: ImpalaBeeswaxException:
INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
MESSAGE: ImpalaRuntimeException: Error creating Kudu table
'impala::tpch_kudu.lineitem'
CAUSED BY: NonRecoverableException: too many attempts:
KuduRpc(method=ListTables, tablet=null, attempt=101,
DeadlineTracker(timeout=180000, elapsed=177706), Traces: [0ms] querying master,
[45ms] Sub rpc
: ConnectToMaster sending RPC to server master-127.0.0.1:7051, [100ms] Sub rpc:
ConnectToMaster received from server master-127.0.0.1:7051 response Network
error: java.net.ConnectException: Connection r
efused: /127.0.0.1:7051, [107ms] delaying RPC due to Service unavailable:
Master config (127.0.0.1:7051) has no leader.
{noformat}
This is the kudu service in the minicluster dropping out from under Impala.
Note btw that Kudu initially started ok.
{noformat}
11:26:26 Starting kudu (Web UI - http://localhost:8051)
11:26:41 The cluster is running
{noformat}
Finally, there's evidence in the minicluster logs for Kudu that all 4 services
(1 master, 3 tablet servers) all failed the same dcheck as pasted above.
> Kudu SIGABRT'ed during dataload on Impala release build
> -------------------------------------------------------
>
> Key: IMPALA-8091
> URL: https://issues.apache.org/jira/browse/IMPALA-8091
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.2.0
> Reporter: David Knupp
> Assignee: Thomas Tauber-Marshall
> Priority: Critical
> Labels: broken-build
>
> *Console log*:
> {noformat}
> 19:10:04 19:10:04 Error executing impala SQL:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql
> See:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
> 19:10:04 Background task Loading functional-query data (pid 4831) failed.
> 19:10:04 Background task Loading TPC-H data (pid 4832) failed.
> 19:10:04 ERROR in
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/testdata/bin/create-load-data.sh
> at line 85: CM_HOST=${2-}
> [...]
> 19:10:05 2019-01-16 19:10:05,461 - archive_core_dumps - INFO - Found core
> files: ['./core.1547693849.30799.kudu-master',
> './core.1547693849.30783.kudu-tserver',
> './core.1547693849.30767.kudu-tserver',
> './core.1547693849.30808.kudu-tserver']
> 19:10:05 2019-01-16 19:10:05,649 - archive_core_dumps - INFO - [New LWP 30799]
> 19:10:05 [New LWP 30834]
> 19:10:05 [New LWP 30835]
> 19:10:05 [New LWP 30838]
> 19:10:05 [New LWP 30837]
> 19:10:05 [New LWP 30836]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x00007f9a2cc611f7 in ?? ()
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,650 - archive_core_dumps - INFO - Found binary
> path through GDB:
> /data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c
> 19:10:05 2019-01-16 19:10:05,893 - archive_core_dumps - WARNING - Failed to
> determine binary because multiple candidate binaries were found and none of
> their paths contained 'latest' to disambiguate:
> 19:10:05 Core:./core.1547693849.30799.kudu-master
> 19:10:05
> Binaries:['./testdata/cluster/node_templates/common/etc/init.d/kudu-master',
> './testdata/cluster/cdh6/node-1/etc/init.d/kudu-master']
> 19:10:05
> 19:10:05 2019-01-16 19:10:05,917 - archive_core_dumps - INFO - [New LWP 30783]
> 19:10:05 [New LWP 30810]
> 19:10:05 [New LWP 30812]
> 19:10:05 [New LWP 30811]
> 19:10:05 [New LWP 30824]
> 19:10:05 [New LWP 30820]
> 19:10:05 Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> 19:10:05 Program terminated with signal SIGABRT, Aborted.
> 19:10:05 #0 0x00007f0b81fb11f7 in ?? ()
> {noformat}
> *Backtraces*:
> {noformat}
> CORE: ./core.1547693849.30799.kudu-master
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f9a2cc611f7 in ?? ()
> #0 0x00007f9a2cc611f7 in ?? ()
> #1 0x00007f9a2cc628e8 in ?? ()
> #2 0x0000000000000020 in ?? ()
> #3 0x0000000000000000 in ?? ()
> CORE: ./core.1547693849.30783.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f0b81fb11f7 in ?? ()
> #0 0x00007f0b81fb11f7 in ?? ()
> #1 0x00007f0b81fb28e8 in ?? ()
> #2 0x0000000000000020 in ?? ()
> #3 0x0000000000000000 in ?? ()
> CORE: ./core.1547693849.30767.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f60b1e2f1f7 in ?? ()
> #0 0x00007f60b1e2f1f7 in ?? ()
> #1 0x00007f60b1e308e8 in ?? ()
> #2 0x0000000000000020 in ?? ()
> #3 0x0000000000000000 in ?? ()
> CORE: ./core.1547693849.30808.kudu-tserver
> BINARY: ./be/build/latest/service/impalad
> Core was generated by
> `/data/jenkins/workspace/impala-asf-master-exhaustive-release/Impala-Toolchain/c'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007fa3cb5591f7 in ?? ()
> #0 0x00007fa3cb5591f7 in ?? ()
> #1 0x00007fa3cb55a8e8 in ?? ()
> #2 0x0000000000000020 in ?? ()
> #3 0x0000000000000000 in ?? ()
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
