[
https://issues.apache.org/jira/browse/FLINK-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891871#comment-15891871
]
ASF GitHub Bot commented on FLINK-5815:
---------------------------------------
Github user wenlong88 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3388#discussion_r103878523
--- Diff:
flink-fs-tests/src/test/java/org/apache/flink/hdfstests/FileUtilsTest.java ---
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.hdfstests;
+
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.test.util.AbstractTestBase;
+import org.apache.flink.util.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.*;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+
+import static org.junit.Assert.fail;
+
+/**
+ * Created by wenlong.lwl on 2017/2/21.
+ */
+public class FileUtilsTest {
+ @ClassRule
+ public static TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private static File TEMP_DIR;
+
+ private static String HDFS_ROOT_URI;
+
+ private static MiniDFSCluster HDFS_CLUSTER;
+
+ private static FileSystem FS;
+
+ //
------------------------------------------------------------------------
+ // startup / shutdown
+ //
------------------------------------------------------------------------
+
+ @BeforeClass
+ public static void createHDFS() {
+ try {
+ TEMP_DIR = temporaryFolder.newFolder();
+
+ Configuration hdConf = new Configuration();
+ hdConf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR,
TEMP_DIR.getAbsolutePath());
+ MiniDFSCluster.Builder builder = new
MiniDFSCluster.Builder(hdConf);
+ HDFS_CLUSTER = builder.build();
+
+ HDFS_ROOT_URI = "hdfs://" +
HDFS_CLUSTER.getURI().getHost() + ":"
+ + HDFS_CLUSTER.getNameNodePort() + "/";
+
+ FS = FileSystem.get(new URI(HDFS_ROOT_URI));
+ }
+ catch (Exception e) {
+ e.printStackTrace();
+ fail("Could not create HDFS mini cluster " +
e.getMessage());
--- End diff --
ok
> Add resource files configuration for Yarn Mode
> ----------------------------------------------
>
> Key: FLINK-5815
> URL: https://issues.apache.org/jira/browse/FLINK-5815
> Project: Flink
> Issue Type: Improvement
> Components: Client, YARN
> Affects Versions: 1.3.0
> Reporter: Wenlong Lyu
> Assignee: Wenlong Lyu
>
> Currently in flink, when we want to setup a resource file to distributed
> cache, we need to make the file accessible remotely by a url, which is often
> difficult to maintain a service like that. What's more, when we want do add
> some extra jar files to job classpath, we need to copy the jar files to blob
> server when submitting the jobgraph. In yarn, especially in flip-6, the blob
> server is not running yet when we try to start a flink job.
> Yarn has a efficient distributed cache implementation for application running
> on it, what's more we can be easily share the files stored in hdfs in
> different application by distributed cache without extra IO operations.
> I suggest to introduce -yfiles, -ylibjars -yarchives options to FlinkYarnCLI
> to enable yarn user setup their job resource files by yarn distributed cache.
> The options is compatible with what is used in mapreduce, which make it easy
> to use for yarn user who generally has experience on using mapreduce.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
