[
https://issues.apache.org/jira/browse/KYLIN-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215306#comment-17215306
]
ASF GitHub Bot commented on KYLIN-4736:
---------------------------------------
zhangayqian commented on pull request #1451:
URL: https://github.com/apache/kylin/pull/1451#issuecomment-709952465
I test this branch in my local env. I use the
$KYLIN_HOME/bin/download-flink.sh download Flink 1.11.1. Then I start kylin and
submit a Flink build job. But there is a error in step `Build cube with flink`.
The error information is as follows:
```
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: error execute
org.apache.kylin.engine.flink.FlinkCubingByLayer. Root cause: Filesystem closed
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:302)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:198)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:149)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:699)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:232)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
Caused by: java.lang.RuntimeException: error execute
org.apache.kylin.engine.flink.FlinkCubingByLayer. Root cause: Filesystem closed
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:288)
... 11 more
Caused by: java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:476)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1683)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1714)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1711)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1726)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1718)
at
org.apache.kylin.engine.mr.JobBuilderSupport.scanFiles(JobBuilderSupport.java:495)
at
org.apache.kylin.engine.mr.JobBuilderSupport.getFileSize(JobBuilderSupport.java:510)
at
org.apache.kylin.engine.flink.FlinkCubingByLayer.execute(FlinkCubingByLayer.java:212)
at
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 17 more
The command is:
export HADOOP_CONF_DIR=/etc/hadoop/conf && export
HADOOP_CLASSPATH=/etc/hadoop &&
/root/zhangyaqian/tar/apache-kylin-3.1.2-SNAPSHOT-bin/flink/bin/flink run -m
yarn-cluster -yjm 2G -ys 1 -ytm 4G -ynm
Build-Cube-with-Flink:kylin_sales_cube_clone[20120101000000_20120201000000] -c
org.apache.kylin.common.util.FlinkEntry -p 1
/root/zhangyaqian/tar/apache-kylin-3.1.2-SNAPSHOT-bin/lib/kylin-job-3.1.2-SNAPSHOT.jar
-className org.apache.kylin.engine.flink.FlinkCubingByLayer -hiveTable
default.kylin_intermediate_kylin_sales_cube_clone_3ca685b1_049d_4377_9c9f_1698e0cbcbc6
-output
hdfs://cdh-master:8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_sales_cube_clone/cuboid/
-input
hdfs://cdh-master:8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_intermediate_kylin_sales_cube_clone_3ca685b1_049d_4377_9c9f_1698e0cbcbc6
-enableObjectReuse false -segmentId 3ca685b1-049d-4377-9c9f-1698e0cbcbc6
-metaUrl
kylin_zyq@hdfs,path=hdfs://cdh-master:8020/kylin/yaqian/kylin_zyq/kylin-44637f80-4271-4dcd-0fb2-f673a418b80e/kylin_sales_cube_clone/metadata
-cubename kylin_sales_cube_clone
at
org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
at
org.apache.kylin.engine.flink.FlinkExecutable.lambda$doWork$1(FlinkExecutable.java:246)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-10-16 14:23:39,537 INFO [Scheduler 737950975 Job
44637f80-4271-4dcd-0fb2-f673a418b80e-123] execution.ExecutableManager:485 : job
id:44637f80-4271-4dcd-0fb2-f673a418b80e-06 from RUNNING to ERROR
2020-10-16 14:23:39,553 INFO [Scheduler 737950975 Job
44637f80-4271-4dcd-0fb2-f673a418b80e-123] execution.ExecutableManager:485 : job
id:44637f80-4271-4dcd-0fb2-f673a418b80e from RUNNING to ERROR
```
@chenjie-sau Do you know the reason for this error?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Upgrade Flink version to 1.11.1
> -------------------------------
>
> Key: KYLIN-4736
> URL: https://issues.apache.org/jira/browse/KYLIN-4736
> Project: Kylin
> Issue Type: Sub-task
> Components: Flink Engine
> Reporter: vinoyang
> Assignee: chenjie
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
