[
https://issues.apache.org/jira/browse/TEZ-4442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Authur Wang updated TEZ-4442:
-----------------------------
Environment: we use CDP7.1.7SP1 with the 0.91 tez version (was: we use
CDP7.1.7SP1 with the 0.91 tez version, and parameters are as follows:
beeline -u
'jdbc:hive2://bg21146.hadoop.com:10000/default;principal=hive/[bg21146.hadoop....@bg.com|mailto:bg21146.hadoop....@bg.com]'
--hiveconf tez.queue.name=root.000kjb.bdhmgmas_bas -e "
create temporary function get_card_rank as
'com.unionpay.spark.udf.GenericUDFCupsCardMediaProc' using jar
'hdfs:///user/lib/spark-udf-0.0.1-SNAPSHOT.jar';
set tez.am.log.level=debug;
set tez.am.resource.memory.mb=8192;
set hive.tez.container.size=8192;
set tez.task.resource.memory.mb=2048;
set tez.runtime.io.sort.mb=1200;
set hive.auto.convert.join.noconditionaltask.size=500000000;
set tez.runtime.unordered.output.buffer.size-mb=800;
set tez.grouping.min-size=33554432;
set tez.grouping.max-size=536870912;
set hive.tez.auto.reducer.parallelism=true;
set hive.tez.min.partition.factor=0.25;
set hive.tez.max.partition.factor=2.0;
set hive.exec.reducers.bytes.per.reducer=268435456;
set mapreduce.map.memory.mb=4096;
set ipc.maximum.response.length=1536000000;
select
get_card_rank(ext_pri_acct_no) as ext_card_media_proc_md,
coun(*)
from bs_comdb.tmp_bscom_glhis_ct_settle_dtl_bas_swt a
where a.hp_settle_dt = '20200910'
group by get_card_rank(ext_pri_acct_no)
;
")
> tez unable to control the memory size when UDF occupies 100MB memory
> ---------------------------------------------------------------------
>
> Key: TEZ-4442
> URL: https://issues.apache.org/jira/browse/TEZ-4442
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Environment: we use CDP7.1.7SP1 with the 0.91 tez version
> Reporter: Authur Wang
> Priority: Critical
>
> We have a UDF which loads about 5 million records into memory, and
> matchs the data in the memory according to the user's input, and finally
> return the output. Each input record of the UDF will lead to one output.
> Based on heapdump analysis, this udf occupies about 100MB of
> memory. The UDF runs stably in hive on MR, hive on spark and native spark,
> and only needs about 4GB of memory for that situation. However, if we use tez
> engine, we adjust the memory from 4G to 8g, the task will fail. Even if we
> adjust the memory to 12g, the task will fail with a high probability. Why
> does tez engine need so much memory compared to Mr and spark? Is there a good
> tuning method to control the amount of memory ?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
