[
https://issues.apache.org/jira/browse/FLINK-28231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17560999#comment-17560999
]
bianqi commented on FLINK-28231:
--------------------------------
I have successfully integrated Flink1.16.0 and Ozone.Below is the sample output
of the test
{code:java}
[root@jykj0 bin]# ./flink run-application -t yarn-application
/soft/flink-1.16.0/flink-1.16-SNAPSHOT/examples/batch/WordCount.jar --input
ofs://jykj0.yarn.com/volume/bucket/warehouse/input --output
ofs://jykj0.yarn.com/volume/bucket/warehouse/output21
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/soft/flink-1.16.0/flink-1.16-SNAPSHOT/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/soft/hadoop-3.2.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2022-06-30 19:50:04,390 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
[] - Found Yarn properties file under
/tmp/flink-1.13.5/.yarn-properties-root.
2022-06-30 19:50:04,390 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
[] - Found Yarn properties file under
/tmp/flink-1.13.5/.yarn-properties-root.
2022-06-30 19:50:04,546 WARN
org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The
configuration directory ('/soft/flink-1.16.0/flink-1.16-SNAPSHOT/conf') already
contains a LOG4J config file.If you want to use logback, then please delete or
rename the log configuration file.
2022-06-30 19:50:04,609 INFO org.apache.hadoop.yarn.client.RMProxy
[] - Connecting to ResourceManager at
jykj0.yarn.com/192.168.149.101:8050
2022-06-30 19:50:04,918 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - No path for the flink jar passed. Using the location of class
org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2022-06-30 19:50:05,044 INFO org.apache.hadoop.conf.Configuration
[] - resource-types.xml not found
2022-06-30 19:50:05,045 INFO
org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to
find 'resource-types.xml'.
2022-06-30 19:50:05,081 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured JobManager memory is 782 MB. YARN will allocate
1024 MB to make up an integer multiple of its minimum allocation memory (1024
MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 242 MB
may not be used by Flink.
2022-06-30 19:50:05,081 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - The configured TaskManager memory is 1653 MB. YARN will
allocate 2048 MB to make up an integer multiple of its minimum allocation
memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The
extra 395 MB may not be used by Flink.
2022-06-30 19:50:05,081 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Cluster specification:
ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1653,
slotsPerTaskManager=4}
2022-06-30 19:50:09,632 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Adding delegation tokens to the AM container.
2022-06-30 19:50:09,636 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Delegation tokens added to the AM container.
2022-06-30 19:50:09,640 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Submitting application master application_1656324008705_0012
2022-06-30 19:50:09,814 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted
application application_1656324008705_0012
2022-06-30 19:50:09,814 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Waiting for the cluster to be allocated
2022-06-30 19:50:09,816 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Deploying cluster, current state ACCEPTED
2022-06-30 19:50:16,619 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - YARN application has been deployed successfully.
2022-06-30 19:50:16,620 INFO org.apache.flink.yarn.YarnClusterDescriptor
[] - Found Web Interface jykj1.yarn.com:38881 of application
'application_1656324008705_0012'.
[root@jykj0 bin]# hdfs dfs -cat
ofs://jykj0.yarn.com/volume/bucket/warehouse/output21
2022-06-30 19:51:09,831 INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2022-06-30 19:51:09,885 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot
period at 10 second(s).
2022-06-30 19:51:09,885 INFO impl.MetricsSystemImpl: XceiverClientMetrics
metrics system started
22 2
232 1
323 1
332 2
abc 2
bb 1
bbb 2
cdd 1 {code}
I think it would be nice to just write the Ozone integration documentation. No
need to develop code to integrate Ozone。
[~martijnvisser]
What do you think about just updating the documentation? Because I think there
will be more and more companies using Ozone in the future。
> Support Apache Ozone FileSystem
> -------------------------------
>
> Key: FLINK-28231
> URL: https://issues.apache.org/jira/browse/FLINK-28231
> Project: Flink
> Issue Type: New Feature
> Components: fs
> Affects Versions: 1.12.1
> Environment: * Flink1.12.1
> * Hadoop3.2.2
> * Ozone1.2.1
> Reporter: bianqi
> Priority: Major
> Attachments: abc.png, image-2022-06-24-10-05-42-193.png,
> image-2022-06-24-10-08-26-010.png, mapreduce-webui.png,
> 微信图片_20220623224142.png
>
>
> h3. Background：
> Apache Ozone has become a new generation object storage system. The future is
> promising.
> Currently, Flink already supports S3, aliyunOSS, and Google Cloud Storage.
> But Apache Ozone is not supported.
> h3. Purpose：
>
> Flink supports the Apache Ozone file system
> h3. work content：
> We need to develop Flink to support Apache Ozone FileSystem. Also improve the
> documentation.
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/filesystems/overview/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
