答复: 退订
如果需要取消订阅 user-zh@flink.apache.org 邮件组，是发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 发件人: 谢振爵 日期: 星期五, 2021年7月30日 15:11 收件人: user-zh 主题: 退订 退订
答复: 退订
如果需要取消订阅 user-zh@flink.apache.org 邮件组，是发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 发件人: 赵珠峰 日期: 星期五, 2021年7月30日 15:15 收件人: user-zh@flink.apache.org 主题: 退订 退订 本邮件载有秘密信息，请您恪守保密义务。未经许可不得披露、使用或允许他人使用。谢谢合作。 This email contains confidential information. Recipient is obliged to keep the information confidential. Any unauthorized disclosure, use, or distribution of the information in this email is strictly prohibited. Thank you.
flink 1.13.1 ????hive??????like????????????
hihive??sql??like ?? org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call: like(VARCHAR(255), STRING NOT NULL) org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call: like(STRING, STRING NOT NULL) If you think this function should be supported, you can create an issue and start a discussion for it. at org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$5$$anonfun$apply$2.apply(ExprCodeGenerator.scala:845) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$5$$anonfun$apply$2.apply(ExprCodeGenerator.scala:845) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at scala.Option.getOrElse(Option.scala:121) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$5.apply(ExprCodeGenerator.scala:844) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator$$anonfun$generateCallExpression$5.apply(ExprCodeGenerator.scala:849) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at scala.Option.getOrElse(Option.scala:121) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:837) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:529) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:56) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.calcite.rex.RexCall.accept(RexCall.java:174) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateExpression(ExprCodeGenerator.scala:155) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateProcessCode(CalcCodeGenerator.scala:173) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateCalcOperator(CalcCodeGenerator.scala:50) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.codegen.CalcCodeGenerator.generateCalcOperator(CalcCodeGenerator.scala) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:94) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.plan.nodes.exec.batch.BatchExecSink.translateToPlanInternal(BatchExecSink.java:58) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:134) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$translateToPlan$1.apply(BatchPlanner.scala:80) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.delegation.BatchPlanner$$anonfun$translateToPlan$1.apply(BatchPlanner.scala:79) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.Iterator$class.foreach(Iterator.scala:891) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.AbstractIterable.foreach(Iterable.scala:54) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at scala.collection.AbstractTraversable.map(Traversable.scala:104) ~[flink-dist_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.delegation.BatchPlanner.translateToPlan(BatchPlanner.scala:79) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:165) ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] at org.apache.flink.table.api.in
Re: Over窗口聚合性能调优问题
（小的）tumbling window + （大的）over window 这样会不会好一些。 Wanghui (HiCampus) 于2021年7月30日周五 下午3:17写道： > Hi all: >我在测试Over窗口时，当窗口是5秒~15s级别时，处理速度能够达到2000/s。 > 但是当我把窗口调整为10分钟以上时，处理速度从2000开始急速下降，几分钟后下降至230/s。 > 请问下： >Over窗口的性能该如何优化，因为我后续会将窗口调整为24小时，按照目前的情况来看，性能会下降很快。 >我的测试节点配置：8C + 16G >Flink配置： taskmanager process memory: 8G > Best regards > WangHui > > -- ** tivanli **
flink 1.13.1 TM内存超用严重问题
问题： flink 计算任务配置的TM内存是 6G，但是，TM进程占用的内存实际达到了 8G。 是什么超用那么多内存？ flink rocksdb内存超用那么多吗？ 还是我的配置有什么问题 ？ 内存使用： > top - 19:05:36 up 304 days, 9:12, 0 users, load average: 7.24, 5.99, > 5.25 > Tasks: 5 total, 1 running, 4 sleeping, 0 stopped, 0 zombie > %Cpu(s): 5.8 us, 0.9 sy, 0.0 ni, 92.8 id, 0.0 wa, 0.0 hi, 0.6 si, > 0.0 st > MiB Mem : 192409.0 total, 63550.7 free, 60642.7 used, 68215.6 buff/cache > MiB Swap: 0.0 total, 0.0 free, 0.0 used. 135410.0 avail Mem > > PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ > COMMAND > > > 300 root 20 0 23.9g 7.8g 49564 S 52.8 4.2 5099:45 > java > TM 配置： > rest.flamegraph.enabled: true > akka.framesize: 20485760b > taskmanager.numberOfTaskSlots: 1 > blob.server.port: 6124 > jobmanager.rpc.port: 6123 > taskmanager.rpc.port: 6122 > jobmanager.heap.size: 2G > taskmanager.memory.process.size: 6g > taskmanager.memory.managed.fraction: 0.5 > taskmanager.debug.memory.log-interval: 1 > jobmanager.execution.failover-strategy: full > 作业情况，作业计算的key比较多亿级别。 [image: image.png] -- ** tivanli **
场景题：Flink SQL 不支持 INSERT INTO… ON DUPLICATE KEY UPDATE ?
现有table CREATE TABLE t ( abigint, bbigint, cbigint, PRIMARY KEY (a) NOT ENFORCED ) WITH ( ... ); 我们的场景只想根据主键a更新部分字段b，其余的字段保持不变，例如 mysql 支持 insert into t(a,b,c) select '1','2','3' on duplicate key update b='4'; 主键重复的时候只更新字段b,字段c的值不变 我在官方文档中没找到这个用法，sql-client也测试了一下也不支持 on duplicate key update，会报错。 请问这种部分字段更新的场景 使用flink sql应该怎么处理？
Re: Flink Yarn Session模式，多任务不同Kerberos认证问题
这个不知道未来怎么规划 Paul Lam 于2021年7月30日周五 下午2:51写道： > > 现在是不能共享的。Flink JobManager 的 principal 在启动时就确定了。 > > Best, > Paul Lam > > > 2021年7月30日 14:46，Ada Luna 写道： > > > > 在Flink Yarn Session中每次提交Job都更换principal。因为要做权限隔离，每个用户有自己的principal。 > > > > 现在 Flink Session模式是不是无法满足多个principal共享一个Flink Session集群，只能走perjob。 > > 或者每个持有独立principal的用户独享一个Session。 >
flink on yarn报错
有没有大佬帮忙看看这个问题 The RMClient's and YarnResourceManagers internal state about the number of pending container requests for resource has diverged .Number client's pending container requests 1 !=Number RM's pending container requests 0;
Re: 退订
Send anything to user-zh-unsubscr...@flink.apache.org hihl 于2021年7月27日周二 下午5:50写道： > 退订
Over窗口聚合性能调优问题
Hi all: 我在测试Over窗口时，当窗口是5秒~15s级别时，处理速度能够达到2000/s。 但是当我把窗口调整为10分钟以上时，处理速度从2000开始急速下降，几分钟后下降至230/s。 请问下： Over窗口的性能该如何优化，因为我后续会将窗口调整为24小时，按照目前的情况来看，性能会下降很快。 我的测试节点配置：8C + 16G Flink配置： taskmanager process memory: 8G Best regards WangHui
退订
退订 本邮件载有秘密信息，请您恪守保密义务。未经许可不得披露、使用或允许他人使用。谢谢合作。 This email contains confidential information. Recipient is obliged to keep the information confidential. Any unauthorized disclosure, use, or distribution of the information in this email is strictly prohibited. Thank you.
退订
退订
?????? flink 1.13.1 ????hive??????????hive sql????????
CREATE TABLE `cosldatacenter.ods_emp_maindata_iadc_paramvalue`( `paramvalue_id` string COMMENT '', `platform_id` string COMMENT '', `equipment_id` string COMMENT '', `param_id` string COMMENT '', `param_value` string COMMENT '', `remark` string COMMENT '', `create_time` string COMMENT '', `creator` string COMMENT '', `update_time` string COMMENT '', `update_person` string COMMENT '', `record_flag` double COMMENT '', `subject_id` string COMMENT '', `output_unit` string COMMENT '', `show_seq` double COMMENT '') COMMENT '' ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'=',', 'serialization.format'=',') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://hadoop02:8020/user/hive/warehouse/cosldatacenter.db/ods_emp_maindata_iadc_paramvalue' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='false', 'last_modified_by'='root', 'last_modified_time'='1621834335', 'numFiles'='0', 'numRows'='-1', 'rawDataSize'='-1', 'totalSize'='0', 'transient_lastDdlTime'='1621834335') CREATE TABLE `cosldatacenter.ods_emp_md_large_equip`( `large_equip_id` string COMMENT '', `equip_name` string COMMENT '', `equip_type` string COMMENT '', `equip_function` string COMMENT '', `equip_board` string COMMENT '', `ship_yard` string COMMENT '', `manufacturer_date` string COMMENT '', `enqueue_date` string COMMENT '', `dockrepair_date` string COMMENT '', `scrap_date` string COMMENT '', `enqueue_mode` string COMMENT '', `work_for_org` string COMMENT '', `work_in_org` string COMMENT '', `old_age` string COMMENT '', `create_time` date COMMENT '', `creator` string COMMENT '', `update_time` date COMMENT '', `update_person` string COMMENT '', `record_flag` double COMMENT '', `data_timestamp` string COMMENT '', `work_unit_id` string COMMENT '', `work_status` string COMMENT '', `work_location` string COMMENT '', `work_area` string COMMENT '', `equip_code` string COMMENT '', `shi_main_power` double COMMENT '', `shi_total_len` double COMMENT '', `shi_type_width` double COMMENT '', `shi_type_depth` double COMMENT '', `shi_design_draft` double COMMENT '', `shi_total_tonnage` double COMMENT '', `shi_load_tonnage` double COMMENT '', `remark` string COMMENT '', `unit_classification1` string COMMENT '', `unit_classification2` string COMMENT '') COMMENT '' ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'=',', 'serialization.format'=',') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://hadoop02:8020/user/hive/warehouse/cosldatacenter.db/ods_emp_md_large_equip' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='false', 'last_modified_by'='root', 'last_modified_time'='1621834338', 'numFiles'='0', 'numRows'='-1', 'rawDataSize'='-1', 'totalSize'='0', 'transient_lastDdlTime'='1621834338') CREATE TABLE `cosldatacenter.ods_emp_maindata_iadc_paramdef`( `param_id` string COMMENT '', `iadc_id` string COMMENT '', `param_code` string COMMENT '', `param_en` string COMMENT '', `param_cn` string COMMENT '', `output_standard` string COMMENT '', `output_unit` string COMMENT '', `param_type` string COMMENT '', `param_value` string COMMENT '', `remark` string COMMENT '', `create_time` string COMMENT '', `creator` string COMMENT '', `update_time` string COMMENT '', `update_person` string COMMENT '', `record_flag` double COMMENT '') COMMENT '' ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'=',', 'serialization.format'=',') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://hadoop02:8020/user/hive/warehouse/cosldatacenter.db/ods_emp_maindata_iadc_paramdef' TBLPROPERTIES ( 'COLUMN_STATS_ACCURATE'='false', 'last_modified_by'='root', 'last_modified_time'='1621834335', 'numFiles'='0', 'numRows'='-1', 'rawDataSize'='-1', 'totalSize'='0', 'transient_lastDdlTime'='1621834335') CREATE TABLE `cosldatacenter.dw_riginfoparam`( `large_equip_id` string, `equip_code` string, `equip_name` string, `enqueue_date` string, `shi_total_len` double, `shi_type_width` double, `shi_type_depth` double, `moonpool` string, `maxwindvelocity` string, `maxwaveheight` string, `airgap` string, `maxopewaterdepth` string, `drilldepthcap` string, `drillvl` string, `drillwater` string, `potablewater` string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'=',', 'ser
