[ https://issues.apache.org/jira/browse/FLINK-7365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119873#comment-16119873 ]
Steve Loughran commented on FLINK-7365: --------------------------------------- There's a special log you can turn off to slience all deprecation warnings: {{org.apache.hadoop.conf.Configuration.deprecation}} {code} log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=ERROR {code} > excessive warning logs of attempt to override final parameter: > fs.s3.buffer.dir > ------------------------------------------------------------------------------- > > Key: FLINK-7365 > URL: https://issues.apache.org/jira/browse/FLINK-7365 > Project: Flink > Issue Type: Bug > Components: Configuration > Affects Versions: 1.3.0 > Reporter: Bowen Li > > I'm seeing hundreds of line of the following log in my JobManager log file: > {code:java} > 2017-08-03 19:48:45,330 WARN org.apache.hadoop.conf.Configuration > - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to > override final parameter: fs.s3.buffer.dir; Ignoring. > 2017-08-03 19:48:45,485 WARN org.apache.hadoop.conf.Configuration > - /etc/hadoop/conf/core-site.xml:an attempt to override final > parameter: fs.s3.buffer.dir; Ignoring. > 2017-08-03 19:48:45,486 WARN org.apache.hadoop.conf.Configuration > - /usr/lib/hadoop/etc/hadoop/core-site.xml:an attempt to > override final parameter: fs.s3.buffer.dir; Ignoring. > 2017-08-03 19:48:45,626 WARN org.apache.hadoop.conf.Configuration > - /etc/hadoop/conf/core-site.xml:an attempt to override final > parameter: fs.s3.buffer.dir; Ignoring > ...... > {code} > Info of my Flink cluster: > - Running on EMR with emr-5.6.0 > - Using FSStateBackend, writing checkpointing data files to s3 > - Configured s3 with S3AFileSystem according to > https://ci.apache.org/projects/flink/flink-docs-release-1.4/setup/aws.html#set-s3-filesystem > - AWS forbids resetting 'fs.s3.buffer.dir' value (it has a <final> tag on > this property in core-site.xml), so I set 'fs.s3a.buffer.dir' as '/tmp' > Here's my core-site.xml file: > {code:java} > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <!-- Licensed to the Apache Software Foundation (ASF) under one or more > --> > <!-- contributor license agreements. See the NOTICE file distributed with > --> > <!-- this work for additional information regarding copyright ownership. > --> > <!-- The ASF licenses this file to You under the Apache License, Version 2.0 > --> > <!-- (the "License"); you may not use this file except in compliance with > --> > <!-- the License. You may obtain a copy of the License at > --> > <!-- > --> > <!-- http://www.apache.org/licenses/LICENSE-2.0 > --> > <!-- > --> > <!-- Unless required by applicable law or agreed to in writing, software > --> > <!-- distributed under the License is distributed on an "AS IS" BASIS, > --> > <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > --> > <!-- See the License for the specific language governing permissions and > --> > <!-- limitations under the License. > --> > <configuration> > <property> > <name>fs.s3.buffer.dir</name> > <value>/mnt/s3,/mnt1/s3</value> > <final>true</final> > </property> > <property> > <name>fs.s3.impl</name> > <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> > </property> > <property> > <name>fs.s3n.impl</name> > <value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value> > </property> > <property> > <name>ipc.client.connect.max.retries.on.timeouts</name> > <value>5</value> > </property> > <property> > <name>hadoop.security.key.default.bitlength</name> > <value>256</value> > </property> > <property> > <name>hadoop.proxyuser.hadoop.groups</name> > <value>*</value> > </property> > <property> > <name>hadoop.tmp.dir</name> > <value>/mnt/var/lib/hadoop/tmp</value> > </property> > <property> > <name>hadoop.proxyuser.hadoop.hosts</name> > <value>*</value> > </property> > <property> > <name>io.file.buffer.size</name> > <value>65536</value> > </property> > <property> > <name>fs.AbstractFileSystem.s3.impl</name> > <value>org.apache.hadoop.fs.s3.EMRFSDelegate</value> > </property> > <property> > <name>fs.s3a.buffer.dir</name> > <value>/tmp</value> > </property> > <property> > <name>fs.s3bfs.impl</name> > <value>org.apache.hadoop.fs.s3.S3FileSystem</value> > </property> > </configuration> > {code} > This bug is about excessive logging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)