Joe McDonnell created IMPALA-12639:
--------------------------------------
Summary: Divert the output of the Hive warm up statement during
dataload
Key: IMPALA-12639
URL: https://issues.apache.org/jira/browse/IMPALA-12639
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 4.4.0
Reporter: Joe McDonnell
Assignee: Joe McDonnell
During dataload in testdata/bin/create-load-data.sh, we run a couple Hive
statements to warm up Hive. These produce hundreds of lines of output. This
should be diverted to a log file to avoid the noise. Using the run-step
function is the standard way to do that.
{noformat}
20:37:29 Running warm up Hive statements
20:37:30 SLF4J: Class path contains multiple SLF4J bindings.
20:37:30 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:30 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:30 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:31 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:32 SLF4J: Class path contains multiple SLF4J bindings.
20:37:32 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:32 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:32 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:32 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:32 Connecting to jdbc:hive2://localhost:11050/default;
20:37:32 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369)
20:37:32 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369)
20:37:32 Transaction isolation: TRANSACTION_REPEATABLE_READ
20:37:34 INFO : Compiling
command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326):
create database if not exists functional
20:37:34 INFO : Semantic Analysis Completed (retrial = false)
20:37:34 INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
20:37:34 INFO : Completed compiling
command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326);
Time taken: 1.139 seconds
20:37:34 INFO : Executing
command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326):
create database if not exists functional
20:37:34 INFO : Starting task [Stage-0:DDL] in serial mode
20:37:34 INFO : Completed executing
command(queryId=jenkins_20231214203732_f74ae90f-84e3-44ef-ae4e-d71d14be1326);
Time taken: 0.226 seconds
20:37:34 INFO : OK
20:37:34 No rows affected (1.572 seconds)
20:37:34 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive
20:37:34 Closing: 0: jdbc:hive2://localhost:11050/default;
20:37:35 SLF4J: Class path contains multiple SLF4J bindings.
20:37:35 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:35 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:35 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:35 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:36 SLF4J: Class path contains multiple SLF4J bindings.
20:37:36 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:36 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:36 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:36 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:37 Connecting to jdbc:hive2://localhost:11050/default;
20:37:37 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369)
20:37:37 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369)
20:37:37 Transaction isolation: TRANSACTION_REPEATABLE_READ
20:37:37 INFO : Compiling
command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7):
create table if not exists hive_warm_up_tbl (i int)
20:37:37 INFO : Semantic Analysis Completed (retrial = false)
20:37:37 INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
20:37:37 INFO : Completed compiling
command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7);
Time taken: 0.075 seconds
20:37:37 INFO : Executing
command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7):
create table if not exists hive_warm_up_tbl (i int)
20:37:37 INFO : Starting task [Stage-0:DDL] in serial mode
20:37:37 INFO : Completed executing
command(queryId=jenkins_20231214203737_686a63ee-b02a-4760-949a-ac10809853a7);
Time taken: 0.101 seconds
20:37:37 INFO : OK
20:37:37 No rows affected (0.253 seconds)
20:37:37 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive
20:37:37 Closing: 0: jdbc:hive2://localhost:11050/default;
20:37:38 SLF4J: Class path contains multiple SLF4J bindings.
20:37:38 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:38 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:38 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:38 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:40 SLF4J: Class path contains multiple SLF4J bindings.
20:37:40 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/apache-hive-3.1.3000.7.2.18.0-369-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:40 SLF4J: Found binding in
[jar:file:/data0/jenkins/workspace/impala-private-basic-parameterized/Impala-Toolchain/cdp_components-45689292/hadoop-3.1.1.7.2.18.0-369/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
20:37:40 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
20:37:40 SLF4J: Actual binding is of type
[org.apache.logging.slf4j.Log4jLoggerFactory]
20:37:40 Connecting to jdbc:hive2://localhost:11050/default;
20:37:40 Connected to: Apache Hive (version 3.1.3000.7.2.18.0-369)
20:37:40 Driver: Hive JDBC (version 3.1.3000.7.2.18.0-369)
20:37:40 Transaction isolation: TRANSACTION_REPEATABLE_READ
20:37:50 INFO : Compiling
command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f):
insert overwrite table hive_warm_up_tbl values (1)
20:37:50 INFO : Semantic Analysis Completed (retrial = false)
20:37:50 INFO : Created Hive schema:
Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null)],
properties:null)
20:37:50 INFO : Completed compiling
command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f);
Time taken: 0.949 seconds
20:37:50 INFO : Executing
command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f):
insert overwrite table hive_warm_up_tbl values (1)
20:37:50 INFO : Query ID =
jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f
20:37:50 INFO : Total jobs = 3
20:37:50 INFO : Launching Job 1 out of 3
20:37:50 INFO : Starting task [Stage-1:MAPRED] in serial mode
20:37:50 INFO : Subscribed to counters: [] for queryId:
jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f
20:37:50 INFO : Tez session hasn't been created yet. Opening session
20:37:50 INFO : Dag name: insert overwrite table hive_warm_up_tb...(1)
(Stage-1)
20:37:50 INFO : HS2 Host:
[impala-ec2-centos79-m6i-4xlarge-xldisk-0293.vpc.cloudera.com], Query ID:
[jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f], Dag ID:
[dag_1702614944517_0001_1], DAG Session ID: [application_1702614944517_0001]
20:37:50 INFO : Status: Running (Executing on YARN cluster with App id
application_1702614944517_0001)
20:37:50
20:37:51
[2K----------------------------------------------------------------------------------------------
20:37:51 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:51
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:51 [2KMap 1 container INITED 1 0
0 1 0 0
20:37:51
[2K----------------------------------------------------------------------------------------------
20:37:51 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0%
ELAPSED TIME: 2.74 s
20:37:51
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:52
[7A[2K----------------------------------------------------------------------------------------------
20:37:52 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:52
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:52 [2KMap 1 container INITED 1 0
0 1 0 0
20:37:52
[2K----------------------------------------------------------------------------------------------
20:37:52 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0%
ELAPSED TIME: 3.74 s
20:37:52
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:53
[7A[2K----------------------------------------------------------------------------------------------
20:37:53 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:53
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:53 [2KMap 1 container RUNNING 1 0
1 0 0 0
20:37:53
[2K----------------------------------------------------------------------------------------------
20:37:53 [2K[31;1mVERTICES: 00/01 [>>--------------------------] 0%
ELAPSED TIME: 4.74 s
20:37:53
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54
[7A[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1
0 0 0 0
20:37:54
[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100%
ELAPSED TIME: 5.75 s
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54
[7A[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1
0 0 0 0
20:37:54
[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100%
ELAPSED TIME: 5.88 s
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54 INFO : Starting task [Stage-7:CONDITIONAL] in serial mode
20:37:54 INFO : Stage-4 is selected by condition resolver.
20:37:54 INFO : Stage-3 is filtered out by condition resolver.
20:37:54 INFO : Stage-5 is filtered out by condition resolver.
20:37:54 INFO : Starting task [Stage-4:MOVE] in serial mode
20:37:54 INFO : Moving data to directory
hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10000
from
hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10002
20:37:54 INFO : Starting task [Stage-2:DEPENDENCY_COLLECTION] in serial mode
20:37:54 INFO : Starting task [Stage-0:MOVE] in serial mode
20:37:54 INFO : Loading data to table default.hive_warm_up_tbl from
hdfs://localhost:20500/test-warehouse/hive_warm_up_tbl/.hive-staging_hive_2023-12-14_20-37-40_924_4404023279633860277-1/-ext-10000
20:37:54 INFO : Completed executing
command(queryId=jenkins_20231214203740_c0783dce-45c9-469e-b087-2b7f2f9ab77f);
Time taken: 12.35 seconds
20:37:54 INFO : OK
20:37:54
[7A[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[36;1m VERTICES MODE STATUS TOTAL COMPLETED
RUNNING PENDING FAILED KILLED
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54 [2KMap 1 .......... container SUCCEEDED 1 1
0 0 0 0
20:37:54
[2K----------------------------------------------------------------------------------------------
20:37:54 [2K[31;1mVERTICES: 01/01 [==========================>>] 100%
ELAPSED TIME: 5.88 s
20:37:54
[22;0m[2K----------------------------------------------------------------------------------------------
20:37:54 1 row affected (13.325 seconds)
20:37:54 Beeline version 3.1.3000.7.2.18.0-369 by Apache Hive
20:37:54 Closing: 0: jdbc:hive2://localhost:11050/default;{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
