-pyexec 指定的是集群端所用的Python环境，客户端需要编译Flink作业，也会依赖Python环境。可以看一下这个文档: https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/dependency_management/#python-interpreter-of-client
On Thu, Nov 18, 2021 at 9:00 PM Asahi Lee <[email protected]> wrote: > Hi ! > 我在java Table api中使用python udf > 函数，通过下面的命令提交应用，报无法启动python服务错误，请问我的提交方式对吗？jm日志为/bin/python: No module named > pyflink。 > > > ./flink-1.13.2/bin/flink > run-application -t yarn-application > -Dyarn.provided.lib.dirs="hdfs://nameservice1/user/flink/flinklib" > -Dyarn.application.queue=d > -p 1 > -pyarch /opt/venv.zip > -pyexec venv.zip/venv/bin/python > -pyfs /opt/test.py > -c test.PyUDFTest > /opt/flink-python-test-1.0-SNAPSHOT.jar > > > > 错误： > Caused by: java.lang.RuntimeException: Python callback server start failed! > at > org.apache.flink.client.python.PythonFunctionFactory.createPythonFunctionFactory(PythonFunctionFactory.java:167) > ~[flink-python_2.11-1.13.2.jar:1.13.2] > at > org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:88) > ~[flink-python_2.11-1.13.2.jar:1.13.2] > at > org.apache.flink.client.python.PythonFunctionFactory$1.load(PythonFunctionFactory.java:84) > ~[flink-python_2.11-1.13.2.jar:1.13.2] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.client.python.PythonFunctionFactory.getPythonFunction(PythonFunctionFactory.java:129) > ~[flink-python_2.11-1.13.2.jar:1.13.2] > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_111] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_111] > at > java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111] > at > org.apache.flink.table.functions.python.utils.PythonFunctionUtils.getPythonFunction(PythonFunctionUtils.java:45) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.functions.UserDefinedFunctionHelper.instantiateFunction(UserDefinedFunctionHelper.java:206) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.catalog.FunctionCatalog.getFunctionDefinition(FunctionCatalog.java:659) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.catalog.FunctionCatalog.resolveAmbiguousFunctionReference(FunctionCatalog.java:606) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:362) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.planner.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:97) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:67) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1200) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1169) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:945) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:704) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:152) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:110) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:201) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:101) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:704) > ~[flink-table-blink_2.11-1.13.1.jar:1.13.1] > at test.PyUDFTest.main(PyUDFTest.java:22) > ~[flink-python-test-1.0-SNAPSHOT.jar:?] > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:1.8.0_111] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:1.8.0_111] > at > java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:242) > ~[flink-dist_2.11-1.13.1.jar:1.13.1] > ... 10 more > 2021-11-18 20:48:44,458 INFO > org.apache.flink.runtime.blob.BlobServer > [] - Stopped BLOB server at > 0.0.0.0:45070 > 2021-11-18 20:48:44,458 INFO > org.apache.flink.runtime.entrypoint.ClusterEntrypoint > [] - Shutting YarnApplicationClusterEntryPoint down with application > status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally.. > 2021-11-18 20:48:44,458 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting > down rest endpoint. > 2021-11-18 20:48:44,474 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing > cache directory > /tmp/flink-web-dd82c3c0-f457-492d-8e64-5ae74fe9abbd/flink-web-ui > 2021-11-18 20:48:44,474 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - > http://cdh5node3:40216 lost leadership > 2021-11-18 20:48:44,474 INFO > org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut > down complete. > 2021-11-18 20:48:44,474 INFO > org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent > [] - Closing components. > 2021-11-18 20:48:44,475 INFO > org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess > [] - Stopping SessionDispatcherLeaderProcess. > 2021-11-18 20:48:44,475 INFO > org.apache.flink.runtime.dispatcher.StandaloneDispatcher > [] - Stopping dispatcher akka.tcp://flink@cdh5node3 > :34697/user/rpc/dispatcher_1. > 2021-11-18 20:48:44,476 INFO > org.apache.flink.runtime.dispatcher.StandaloneDispatcher > [] - Stopping all currently running jobs of dispatcher > akka.tcp://flink@cdh5node3:34697/user/rpc/dispatcher_1. > > > LogType:jobmanager.out > Log Upload Time:星期四 十一月 18 20:48:45 +0800 2021 > LogLength:37 > Log Contents: > /bin/python: No module named pyflink > > > > > > > > > > > > > > ------------------ 原始邮件 ------------------ > 发件人: > "user-zh" > < > [email protected]>; > 发送时间: 2021年11月18日(星期四) 下午3:34 > 收件人: "user-zh"<[email protected]>; > > 主题: Re:哪里可以下载到downloads/setup-pyflink-virtual-env.sh脚本 > > > > Hi!<br/>我在下面的旧版本中找到了，你试下能不能用：<br/> > https://nightlies.apache.org/flink/flink-docs-release-1.12/downloads/setup-pyflink-virtual-env.sh > 在 2021-11-18 15:05:03，"Asahi Lee" <[email protected]> 写道： > >Hi! > >&nbsp; &nbsp; > 我在flink官方文档中看到通过&nbsp;&nbsp;setup-pyflink-virtual-env.sh > 脚本制作python虚拟环境，请问再哪里可以下载到？文档地址如下： > > > https://nightlies.apache.org/flink/flink-docs-release-1.14/zh/docs/dev/python/faq/
