dixingxing0 opened a new issue #2198:
URL: https://github.com/apache/iceberg/issues/2198
I got an error while join iceberg table and hive table at same time, most of
mappers succeed, but some mappers got an `cannot find field` error:
```java
Caused by: java.lang.RuntimeException: cannot find field log_src from
[org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector$IcebergRecordStructField@8838736]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:442)
at
org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector.getStructFieldRef(IcebergRecordObjectInspector.java:78)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:980)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1006)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:77)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:498)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:115)
... 22 more
```
Environment:
```java
Iceberg : 0.11
hive : 2.0.1
hadoop : 2.7.2
JDK : 1.8
```
I found some clues:
```java
2021-02-02 15:57:35,036 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
Paths:/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10005/0/emptyFile:0+0InputFormatClass:
org.apache.hadoop.mapred.TextInputFormat
```
The split file is for `hive_table1`(which is an empty table for test), but
still the mapper using `org.apache.iceberg.mr.hive.HiveIcebergSerDe`, the table
properties is also from `iceberg_table1`.
It seems like a bug of hive, hive using wrong serd to deserialize data, I am
not an expert in hive, hope someone could give me some clues .
I've created an hive issue here:
[HIVE-24720](https://issues.apache.org/jira/browse/HIVE-24720)
SQL:
```sql
select count(1),count(a.dt),count(b.dt)
from (
select dt,concat_ws('###',wx_source, log_src, dt, hour) as str
from flink_fdm_iceberg.iceberg_table1
where dt='2021-01-29' and hour='16') a
full outer join (
select dt,concat_ws('###',wx_source, log_src, dt, hour) as str
from flink_fdm_iceberg.hive_table1
where dt='2021-01-29' and hour='16') b on a.str=b.str
where a.str is null or b.str is null;
```
The full log by the failed mapper:
```java
2021-02-02 15:57:30,060 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2021-02-02 15:57:30,118 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2021-02-02 15:57:30,118 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
started
2021-02-02 15:57:30,131 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens:
2021-02-02 15:57:30,131 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: mapreduce.job, Service: job_1605975810748_1427, Ident:
(org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@4b2bac3f)
2021-02-02 15:57:30,167 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns2, Ident:
(HDFS_DELEGATION_TOKEN token 151282235 for flink)
2021-02-02 15:57:30,168 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident:
(HDFS_DELEGATION_TOKEN token 151249944 for flink)
2021-02-02 15:57:30,168 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns4, Ident:
(HDFS_DELEGATION_TOKEN token 151235141 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns3, Ident:
(HDFS_DELEGATION_TOKEN token 151183885 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns6, Ident:
(HDFS_DELEGATION_TOKEN token 168039224 for flink)
2021-02-02 15:57:30,169 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns5, Ident:
(HDFS_DELEGATION_TOKEN token 150801000 for flink)
2021-02-02 15:57:30,881 INFO [main] org.apache.hadoop.mapred.YarnChild:
Sleeping for 0ms before retrying again. Got null now.
2021-02-02 15:57:31,032 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child:
/data1/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data2/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data4/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data5/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data6/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data7/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data8/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data9/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data10/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data11/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427,/data12/nm-local-dir/usercache/flink/appcache/
application_1605975810748_1427
2021-02-02 15:57:31,937 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
2021-02-02 15:57:34,143 INFO [main] org.apache.hadoop.mapred.Task: Using
ResourceCalculatorProcessTree : [ ]
// !!!!!
2021-02-02 15:57:35,036 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split:
Paths:/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10005/0/emptyFile:0+0InputFormatClass:
org.apache.hadoop.mapred.TextInputFormat
2021-02-02 15:57:35,198 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH =
hdfs://ns6/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10006/33cfad1a-7384-4c27-b2d4-2f894ecb845a/map.xml
2021-02-02 15:57:35,209 WARN [main] org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category READ is not supported in state standby
2021-02-02 15:57:35,221 INFO [main]
org.apache.hadoop.hive.ql.exec.SerializationUtilities: Deserializing MapWork
using kryo
2021-02-02 15:57:35,384 INFO [main]
com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library from
the embedded binaries
2021-02-02 15:57:35,386 INFO [main] com.hadoop.compression.lzo.LzoCodec:
Successfully loaded & initialized native-lzo library [hadoop-lzo rev
5e360bdefd8923280b1a0234b845448050bf0caa]
2021-02-02 15:57:35,857 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated.
Instead, use mapreduce.map.input.file
2021-02-02 15:57:35,857 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.start is
deprecated. Instead, use mapreduce.map.input.start
2021-02-02 15:57:35,857 INFO [main]
org.apache.hadoop.conf.Configuration.deprecation: map.input.length is
deprecated. Instead, use mapreduce.map.input.length
2021-02-02 15:57:35,857 INFO [main] org.apache.hadoop.mapred.MapTask:
numReduceTasks: 1009
2021-02-02 15:57:37,109 INFO [main] org.apache.hadoop.mapred.MapTask:
(EQUATOR) 0 kvi 268435452(1073741808)
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask:
mapreduce.task.io.sort.mb: 1024
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask: soft
limit at 858993472
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask:
bufstart = 0; bufvoid = 1073741824
2021-02-02 15:57:37,110 INFO [main] org.apache.hadoop.mapred.MapTask:
kvstart = 268435452; length = 67108864
2021-02-02 15:57:37,114 INFO [main] org.apache.hadoop.mapred.MapTask: Map
output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2021-02-02 15:57:37,119 INFO [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper: conf classpath =
[file:/data12/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/container_e03_1605975810748_1427_01_000559/,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/etc/hadoop/,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-nfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/mysql-connector-java-5.1.31-bin.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/spark-2.0.2-yarn-shuffle.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/data-hadoop-patch.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common
/lib/jaxb-api-2.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-client-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/junit-4.11.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/avro-1.7.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jets3t-0.9.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-httpclient-3.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-math3-3.1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/servl
et-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpclient-4.2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/mockito-all-1.8.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hamcrest-core-1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-recipes-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-collections-3.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.
2-yarn/share/hadoop/common/lib/guava-11.0.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpcore-4.2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsp-api-2.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-json-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/zookeeper-3.4.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-auth-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/ha
doop/common/lib/slf4j-api-1.7.10.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xmlenc-0.52.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jettison-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-util-1.0.0-M20.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/snappy-java-1.0.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsch-0.1.42.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/stax-api-1.0-2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-xc-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/commo
n/lib/paranamer-2.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-configuration-1.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/gson-2.2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-1.7.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/java-xmlbuilder-0.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-net-3.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2
-yarn/share/hadoop/common/lib/commons-digester-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-framework-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/activation-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop-2
.7.2-yarn/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/servlet-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-server-1.9.jar,
file:/data
/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/guava-11.0.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xmlenc-0.52.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoo
p/hdfs/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-
common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/servlet-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-servlet-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-collections-3.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guava-11.0.2.jar,
file:/data
/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/aopalliance-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-json-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-client-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jettison-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/stax-api-1.0-2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yar
n/lib/jackson-xc-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/javax.inject-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-guice-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/activation-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-ya
rn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/junit-4.11.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/avro-1.7.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hadoop-annotation
s-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/aopalliance-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/paranamer-2.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/javax.inject-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduc
e/lib/jersey-guice-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/job.jar,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/classes,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/lib/*,
file:/data2/nm-local-dir/usercache/flink/filecache/299/UserAgentUtils-1.19.jar,
file:/data9/nm-local-dir/usercache/flink/filecache/294/iceberg-hive-runtime-022dc46.dirty.jar,
file:/data7/nm-local-dir/usercache/flink/filecache/292/hive-jdbc-1.2.1.jar,
file:/data12/nm-local-dir/usercache/flink/filecache/297/dc-ua-util-2.0.0-SNAPSHOT.jar,
file:/data11/nm-local-dir/usercache/flink/filecache/296/check-hive-db.jar,
file:/data3/nm-local-dir/usercache/flink
/appcache/application_1605975810748_1427/filecache/10/job.jar/,
file:/data10/nm-local-dir/usercache/flink/filecache/295/data-udf-0.0.1.jar,
file:/data6/nm-local-dir/usercache/flink/filecache/291/sqljdbc4-4.0.jar]
2021-02-02 15:57:37,121 INFO [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper: thread classpath =
[file:/data12/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/container_e03_1605975810748_1427_01_000559/,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/etc/hadoop/,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-nfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/mysql-connector-java-5.1.31-bin.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/spark-2.0.2-yarn-shuffle.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/data-hadoop-patch.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/hadoop-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/comm
on/lib/jaxb-api-2.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-client-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/junit-4.11.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/avro-1.7.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jets3t-0.9.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-httpclient-3.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-math3-3.1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/ser
vlet-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpclient-4.2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/mockito-all-1.8.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hamcrest-core-1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-recipes-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-collections-3.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.
7.2-yarn/share/hadoop/common/lib/guava-11.0.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/httpcore-4.2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsp-api-2.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-json-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/zookeeper-3.4.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/hadoop-auth-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/
hadoop/common/lib/slf4j-api-1.7.10.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xmlenc-0.52.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jettison-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-util-1.0.0-M20.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/snappy-java-1.0.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsch-0.1.42.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/stax-api-1.0-2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-xc-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/com
mon/lib/paranamer-2.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-configuration-1.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/gson-2.2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-beanutils-1.7.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/java-xmlbuilder-0.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-net-3.1.jar,
file:/data/sysdir/servers/hadoop-2.7
.2-yarn/share/hadoop/common/lib/commons-digester-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/curator-framework-2.7.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/activation-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/common/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop
-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/servlet-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-server-1.9.jar,
file:/da
ta/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/guava-11.0.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/xmlenc-0.52.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/had
oop/hdfs/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/hdfs/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-serve
r-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-6.1.26.jar
,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jetty-util-6.1.26.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/servlet-api-2.5.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-logging-1.1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guice-servlet-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-collections-3.2.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/guava-11.0.2.jar,
file:/da
ta/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/aopalliance-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-json-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-client-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-cli-1.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-lang-2.6.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jettison-1.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/stax-api-1.0-2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/y
arn/lib/jackson-xc-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/javax.inject-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jsr305-3.0.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jersey-guice-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/activation-1.1.ja
r,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/yarn/lib/commons-codec-1.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-
yarn/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/junit-4.11.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/avro-1.7.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-server-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/hadoop-annotati
ons-2.7.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/aopalliance-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-io-2.4.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jersey-core-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/asm-3.2.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/paranamer-2.3.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/log4j-1.2.17.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/javax.inject-1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapred
uce/lib/jersey-guice-1.9.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/xz-1.0.jar,
file:/data/sysdir/servers/hadoop-2.7.2-yarn/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/job.jar,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/classes,
file:/data3/nm-local-dir/usercache/flink/appcache/application_1605975810748_1427/filecache/10/job.jar/lib/*,
file:/data2/nm-local-dir/usercache/flink/filecache/299/UserAgentUtils-1.19.jar,
file:/data9/nm-local-dir/usercache/flink/filecache/294/iceberg-hive-runtime-022dc46.dirty.jar,
file:/data7/nm-local-dir/usercache/flink/filecache/292/hive-jdbc-1.2.1.jar,
file:/data12/nm-local-dir/usercache/flink/filecache/297/dc-ua-util-2.0.0-SNAPSHOT.jar,
file:/data11/nm-local-dir/usercache/flink/filecache/296/check-hive-db.jar,
file:/data3/nm-local-dir/usercache/fli
nk/appcache/application_1605975810748_1427/filecache/10/job.jar/,
file:/data10/nm-local-dir/usercache/flink/filecache/295/data-udf-0.0.1.jar,
file:/data6/nm-local-dir/usercache/flink/filecache/291/sqljdbc4-4.0.jar]
2021-02-02 15:57:37,121 INFO [main]
org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH =
hdfs://ns6/tmp/hive/flink/25369d0b-110e-4eca-8553-f2f3953f0303/hive_2021-02-02_15-56-22_060_2459779186626852210-1/-mr-10006/33cfad1a-7384-4c27-b2d4-2f894ecb845a/map.xml
2021-02-02 15:57:37,123 INFO [main]
org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[0]
2021-02-02 15:57:37,128 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties :
{columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string,
location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di,
columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour,
metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json,
previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct
flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id,
string pre_load_id, string sign, string visit_time, string ip, string
province_id, string city_id, string county_id, string appkey, string
session_id, string user_id, string ua, string visits_id, string
first_visit_time, string first_visit_load_id, string first_visit_ref_domain,
string first_visit_subdomain_time, string first_visit_subdomain_load_id, string
first_visit_subdomain_refer, string visits_ref, string visits_keyword, string
visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string
site_ref, string scene_type, string show_id, string ref, string refer_domain,
string key_word, string area_id, string current_url, string current_domain,
string site_id, string category_id, string sub_category_id, string object_id,
string series_id, string spec_id, string dealer_id, string level_id, string
doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string
browser_charset, string browser_language, string browser_cookie_enable, string
extend, string create_time, string id, string custom1, string custom2, string
customer3, string customer4, string longitude, string latitude, string
browse_type, string ac_userid, string ac_app_key, string ac_app_platform,
string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string
cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest,
string wx_openid, string wx_source, string log_src, string dt, string hour},
transient_lastDdlTime=1611904166,
iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]},
columns.comments=, totalSize=302458258710, bucket_count=0,
file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat,
serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe,
storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler,
table_type=ICEBERG,
file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat,
EXTERNAL=TRUE,
name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,906 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties
{"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,907 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:37,926 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema
{"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:37,969 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties :
{columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string,
location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di,
columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour,
metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json,
previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct
flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id,
string pre_load_id, string sign, string visit_time, string ip, string
province_id, string city_id, string county_id, string appkey, string
session_id, string user_id, string ua, string visits_id, string
first_visit_time, string first_visit_load_id, string first_visit_ref_domain,
string first_visit_subdomain_time, string first_visit_subdomain_load_id, string
first_visit_subdomain_refer, string visits_ref, string visits_keyword, string
visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string
site_ref, string scene_type, string show_id, string ref, string refer_domain,
string key_word, string area_id, string current_url, string current_domain,
string site_id, string category_id, string sub_category_id, string object_id,
string series_id, string spec_id, string dealer_id, string level_id, string
doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string
browser_charset, string browser_language, string browser_cookie_enable, string
extend, string create_time, string id, string custom1, string custom2, string
customer3, string customer4, string longitude, string latitude, string
browse_type, string ac_userid, string ac_app_key, string ac_app_platform,
string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string
cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest,
string wx_openid, string wx_source, string log_src, string dt, string hour},
transient_lastDdlTime=1611904166,
iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]},
columns.comments=, totalSize=302458258710, bucket_count=0,
file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat,
serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe,
storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler,
table_type=ICEBERG,
file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat,
EXTERNAL=TRUE,
name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,973 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties
{"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,974 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:37,974 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema
{"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:37,998 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: serDeProperties :
{columns.types=string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string:string,
location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di,
columns=version,load_id,pre_load_id,sign,visit_time,ip,province_id,city_id,county_id,appkey,session_id,user_id,ua,visits_id,first_visit_time,first_visit_load_id,first_visit_ref_domain,first_visit_subdomain_time,first_visit_subdomain_load_id,firs
t_visit_subdomain_refer,visits_ref,visits_keyword,visits_area_id,bt_visits_ref,last_visit_subdomain_refer,site_ref,scene_type,show_id,ref,refer_domain,key_word,area_id,current_url,current_domain,site_id,category_id,sub_category_id,object_id,series_id,spec_id,dealer_id,level_id,doc_title,page_custom_vars,platform,screen,color_depth,browser_charset,browser_language,browser_cookie_enable,extend,create_time,id,custom1,custom2,customer3,customer4,longitude,latitude,browse_type,ac_userid,ac_app_key,ac_app_platform,ac_app_ver,ac_app_deviceid,cur_visit_ref_domain,cookie_content,pre_visits_ref,lt_visits_ref,abtest,wx_openid,wx_source,log_src,dt,hour,
metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00753-b2322cf0-ac32-4eb0-957e-95d2189e0272.metadata.json,
previous_metadata_location=viewfs://AutoLfCluster/team/db/hive_db/flink_fdm_iceberg/flink_fdm_iceberg_fdm_flw_pcm_log_clean_di/metadata/00752-6949a171-403e-4853-9
3a3-666df4305603.metadata.json, numFiles=59177, serialization.ddl=struct
flink_fdm_iceberg_fdm_flw_pcm_log_clean_di { string version, string load_id,
string pre_load_id, string sign, string visit_time, string ip, string
province_id, string city_id, string county_id, string appkey, string
session_id, string user_id, string ua, string visits_id, string
first_visit_time, string first_visit_load_id, string first_visit_ref_domain,
string first_visit_subdomain_time, string first_visit_subdomain_load_id, string
first_visit_subdomain_refer, string visits_ref, string visits_keyword, string
visits_area_id, string bt_visits_ref, string last_visit_subdomain_refer, string
site_ref, string scene_type, string show_id, string ref, string refer_domain,
string key_word, string area_id, string current_url, string current_domain,
string site_id, string category_id, string sub_category_id, string object_id,
string series_id, string spec_id, string dealer_id, string level_id, string
doc_title, string pag
e_custom_vars, string platform, string screen, string color_depth, string
browser_charset, string browser_language, string browser_cookie_enable, string
extend, string create_time, string id, string custom1, string custom2, string
customer3, string customer4, string longitude, string latitude, string
browse_type, string ac_userid, string ac_app_key, string ac_app_platform,
string ac_app_ver, string ac_app_deviceid, string cur_visit_ref_domain, string
cookie_content, string pre_visits_ref, string lt_visits_ref, string abtest,
string wx_openid, string wx_source, string log_src, string dt, string hour},
transient_lastDdlTime=1611904166,
iceberg.mr.table.schema={"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id"
:6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"first_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,
"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required":false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,
"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform","required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"cre
ate_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"i
d":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]},
columns.comments=, totalSize=302458258710, bucket_count=0,
file.outputformat=org.apache.iceberg.mr.hive.HiveIcebergOutputFormat,
serialization.lib=org.apache.iceberg.mr.hive.HiveIcebergSerDe,
storage_handler=org.apache.iceberg.mr.hive.HiveIcebergStorageHandler,
table_type=ICEBERG,
file.inputformat=org.apache.iceberg.mr.hive.HiveIcebergInputFormat,
EXTERNAL=TRUE,
name=flink_fdm_iceberg.flink_fdm_iceberg_fdm_flw_pcm_log_clean_di}
2021-02-02 15:57:37,999 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using schema from serDeProperties
{"type":"struct","fields":[{"id":1,"name":"version","required":false,"type":"string"},{"id":2,"name":"load_id","required":false,"type":"string"},{"id":3,"name":"pre_load_id","required":false,"type":"string"},{"id":4,"name":"sign","required":false,"type":"string"},{"id":5,"name":"visit_time","required":false,"type":"string"},{"id":6,"name":"ip","required":false,"type":"string"},{"id":7,"name":"province_id","required":false,"type":"string"},{"id":8,"name":"city_id","required":false,"type":"string"},{"id":9,"name":"county_id","required":false,"type":"string"},{"id":10,"name":"appkey","required":false,"type":"string"},{"id":11,"name":"session_id","required":false,"type":"string"},{"id":12,"name":"user_id","required":false,"type":"string"},{"id":13,"name":"ua","required":false,"type":"string"},{"id":14,"name":"visits_id","required":false,"type":"string"},{"id":15,"name":"f
irst_visit_time","required":false,"type":"string"},{"id":16,"name":"first_visit_load_id","required":false,"type":"string"},{"id":17,"name":"first_visit_ref_domain","required":false,"type":"string"},{"id":18,"name":"first_visit_subdomain_time","required":false,"type":"string"},{"id":19,"name":"first_visit_subdomain_load_id","required":false,"type":"string"},{"id":20,"name":"first_visit_subdomain_refer","required":false,"type":"string"},{"id":21,"name":"visits_ref","required":false,"type":"string"},{"id":22,"name":"visits_keyword","required":false,"type":"string"},{"id":23,"name":"visits_area_id","required":false,"type":"string"},{"id":24,"name":"bt_visits_ref","required":false,"type":"string"},{"id":25,"name":"last_visit_subdomain_refer","required":false,"type":"string"},{"id":26,"name":"site_ref","required":false,"type":"string"},{"id":27,"name":"scene_type","required":false,"type":"string"},{"id":28,"name":"show_id","required":false,"type":"string"},{"id":29,"name":"ref","required"
:false,"type":"string"},{"id":30,"name":"refer_domain","required":false,"type":"string"},{"id":31,"name":"key_word","required":false,"type":"string"},{"id":32,"name":"area_id","required":false,"type":"string"},{"id":33,"name":"current_url","required":false,"type":"string"},{"id":34,"name":"current_domain","required":false,"type":"string"},{"id":35,"name":"site_id","required":false,"type":"string"},{"id":36,"name":"category_id","required":false,"type":"string"},{"id":37,"name":"sub_category_id","required":false,"type":"string"},{"id":38,"name":"object_id","required":false,"type":"string"},{"id":39,"name":"series_id","required":false,"type":"string"},{"id":40,"name":"spec_id","required":false,"type":"string"},{"id":41,"name":"dealer_id","required":false,"type":"string"},{"id":42,"name":"level_id","required":false,"type":"string"},{"id":43,"name":"doc_title","required":false,"type":"string"},{"id":44,"name":"page_custom_vars","required":false,"type":"string"},{"id":45,"name":"platform"
,"required":false,"type":"string"},{"id":46,"name":"screen","required":false,"type":"string"},{"id":47,"name":"color_depth","required":false,"type":"string"},{"id":48,"name":"browser_charset","required":false,"type":"string"},{"id":49,"name":"browser_language","required":false,"type":"string"},{"id":50,"name":"browser_cookie_enable","required":false,"type":"string"},{"id":51,"name":"extend","required":false,"type":"string"},{"id":52,"name":"create_time","required":false,"type":"string"},{"id":53,"name":"id","required":false,"type":"string"},{"id":54,"name":"custom1","required":false,"type":"string"},{"id":55,"name":"custom2","required":false,"type":"string"},{"id":56,"name":"customer3","required":false,"type":"string"},{"id":57,"name":"customer4","required":false,"type":"string"},{"id":58,"name":"longitude","required":false,"type":"string"},{"id":59,"name":"latitude","required":false,"type":"string"},{"id":60,"name":"browse_type","required":false,"type":"string"},{"id":61,"name":"ac
_userid","required":false,"type":"string"},{"id":62,"name":"ac_app_key","required":false,"type":"string"},{"id":63,"name":"ac_app_platform","required":false,"type":"string"},{"id":64,"name":"ac_app_ver","required":false,"type":"string"},{"id":65,"name":"ac_app_deviceid","required":false,"type":"string"},{"id":66,"name":"cur_visit_ref_domain","required":false,"type":"string"},{"id":67,"name":"cookie_content","required":false,"type":"string"},{"id":68,"name":"pre_visits_ref","required":false,"type":"string"},{"id":69,"name":"lt_visits_ref","required":false,"type":"string"},{"id":70,"name":"abtest","required":false,"type":"string"},{"id":71,"name":"wx_openid","required":false,"type":"string"},{"id":72,"name":"wx_source","required":false,"type":"string"},{"id":73,"name":"log_src","required":false,"type":"string"},{"id":74,"name":"dt","required":false,"type":"string"},{"id":75,"name":"hour","required":false,"type":"string"}]}
2021-02-02 15:57:37,999 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: selectedColumns 1, [wx_source]
2021-02-02 15:57:38,000 INFO [main]
org.apache.iceberg.mr.hive.HiveIcebergSerDe: Using projectedSchema
{"type":"struct","fields":[{"id":72,"name":"wx_source","required":false,"type":"string"}]}
2021-02-02 15:57:38,005 INFO [main]
org.apache.hadoop.hive.ql.exec.mr.ExecMapper:
<MAP>Id =0
<Children>null
<TS>Id =3
<Children>
<FIL>Id =18
<Children>
<SEL>Id =5
<Children>
<RS>Id =7
<Children>
<\Children>
<Parent>Id = 5 null<\Parent>
<\RS>
<\Children>
<Parent>Id = 18 null<\Parent>
<\SEL>
<\Children>
<Parent>Id = 3 null<\Parent>
<\FIL>
<\Children>
<Parent>Id = 0 null<\Parent>
<\TS>
<\Children>
<Parent><\Parent>
<\MAP>
2021-02-02 15:57:38,007 INFO [main]
org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[3]
2021-02-02 15:57:38,008 INFO [main]
org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing operator FIL[18]
2021-02-02 15:57:43,031 INFO [main]
org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing operator SEL[5]
2021-02-02 15:57:43,031 INFO [main]
org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT
struct<version:string,load_id:string,pre_load_id:string,sign:string,visit_time:string,ip:string,province_id:int,city_id:int,county_id:string,appkey:string,session_id:string,user_id:int,ua:string,visits_id:string,first_visit_time:string,first_visit_load_id:string,first_visit_ref_domain:string,first_visit_subdomain_time:string,first_visit_subdomain_load_id:string,first_visit_subdomain_refer:string,visits_ref:string,visits_keyword:string,visits_area_id:string,bt_visits_ref:string,last_visit_subdomain_refer:string,site_ref:string,scene_type:string,show_id:string,ref:string,refer_domain:string,key_word:string,area_id:string,current_url:string,current_domain:string,site_id:int,category_id:int,sub_category_id:int,object_id:int,series_id:int,spec_id:int,dealer_id:int,level_id:int,doc_title:string,page_custom_vars:string,platform:int,screen:string,color_depth:string,browser_charset:string,browser_lang
uage:string,browser_cookie_enable:string,extend:string,create_time:string,id:string,custom1:int,custom2:int,customer3:string,customer4:string,longitude:float,latitude:float,browse_type:string,ac_userid:string,ac_app_key:string,ac_app_platform:string,ac_app_ver:string,ac_app_deviceid:string,cur_visit_ref_domain:string,cookie_content:string,pre_visits_ref:string,lt_visits_ref:string,abtest:string,wx_openid:string,wx_source:string,log_src:string,dt:string,hour:string>
2021-02-02 15:57:43,033 INFO [main]
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing operator RS[7]
2021-02-02 15:57:43,036 INFO [main]
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = 1
2021-02-02 15:57:43,050 INFO [main]
org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing operator TS[0]
2021-02-02 15:57:43,050 INFO [main]
org.apache.hadoop.hive.ql.exec.FilterOperator: Initializing operator FIL[17]
2021-02-02 15:57:43,051 INFO [main]
org.apache.hadoop.hive.ql.exec.SelectOperator: Initializing operator SEL[2]
2021-02-02 15:57:43,051 INFO [main]
org.apache.hadoop.hive.ql.exec.SelectOperator: SELECT struct<wx_source:string>
2021-02-02 15:57:43,052 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.RuntimeException: Error in configuring
object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:136)
... 22 more
Caused by: java.lang.RuntimeException: cannot find field log_src from
[org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector$IcebergRecordStructField@8838736]
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:442)
at
org.apache.iceberg.mr.hive.serde.objectinspector.IcebergRecordObjectInspector.getStructFieldRef(IcebergRecordObjectInspector.java:78)
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:55)
at
org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:139)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:980)
at
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1006)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:77)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:355)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:504)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:457)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:365)
at
org.apache.hadoop.hive.ql.exec.MapOperator.initializeMapOperator(MapOperator.java:498)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:115)
... 22 more
2021-02-02 15:57:43,053 INFO [main] org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
