I just tried to put a file(only size about 400M) to IGFS using the code bellow.
The code is quite same as the offical examples， it cant end with a reslut.
While using “hadoop fs –put xxxx”, it only take about 7s.
public class IgfsWrite {
public static void main(String[] args) throws IOException {
Ignite ignite =
Ignition.start("examples/config/filesystem/yq-example-igfs.xml");
System.out.println("Ignite is started");
Long t1 = System.currentTimeMillis();
File file = new File(args[0]);
IgniteFileSystem fs = ignite.fileSystem("igfs");
IgfsPath workDir = new IgfsPath("/examples/fs");
IgfsPath fsPath = new IgfsPath(workDir, file.getName());
System.out.println("Start copying file: " + file.getAbsolutePath());
IgfsOutputStream os = fs.create(fsPath, true);
FileInputStream fis = new FileInputStream(file);
byte[] buf = new byte[2048];
int read = fis.read(buf);
while (read != -1) {
os.write(buf, 0, read);
read = fis.read(buf);
}
System.out.println(System.currentTimeMillis()-t1);
Ignition.stop(true);
}
}
发件人: Vladimir Ozerov [mailto:[email protected]]
发送时间: 2015年9月7日 20:16
收件人: [email protected]
抄送: Hejun (Jun He)
主题: Re: 答复: How to access data stored in IGFS or IGFS as Hadoop cache
>> 1. If the way accessing IGFS is same as HDFS, how can I ensure where the
>> data is stored , in IGFS or HDFS? (consider IGFS as HDFS cache mode)
IGFS doesn't provide the way to check whether file is stored in it, in
secondary file system, or in both. You can query HDFS and check whether file
exists there. IGFS is a caching layer. Normally you should only need to know
the mode in which partiuclar path works: PRIMARY, DUAL_SYNC, DUAL_ASYNC or
PROXY. Please check documentation here:
https://apacheignite.readme.io/docs/modes
>> And if I want to compare the performance of these two file system, what
>> shall I do?
I have no answer to this question, because it highly depends in what and how
are you going to measure. Please provide more information on your case and I
will try to give you some ideas.
>> 2. I tried to set fs.defaultFS as “igfs:///”, but hadoop could not start
>> for no namenode detected
Yes, sorry, HDFS cannot start normally when fs.defaultFS in core-site.xml is
set to non-HDFS file system. In this case just set fs.defaultFS back to default
and access file system using fully qualified paths. E.g., instead of
"/path/to/my/file" try using "igfs:///path/to/my/file".
On Mon, Sep 7, 2015 at 2:57 PM, yaoqin
<[email protected]<mailto:[email protected]>> wrote:
1. If the way accessing IGFS is same as HDFS, how can I ensure where the
data is stored , in IGFS or HDFS? (consider IGFS as HDFS cache mode)
And if I want to compare the performance of these two file system, what shall I
do?
2. I tried to set fs.defaultFS as “igfs:///”, but hadoop could not start
for no namenode detected
发件人: Vladimir Ozerov [mailto:[email protected]<mailto:[email protected]>]
发送时间: 2015年9月7日 15:13
收件人: [email protected]<mailto:[email protected]>
抄送: Hejun (Jun He)
主题: Re: How to access data stored in IGFS or IGFS as Hadoop cache
>> 1. How to access(set/get) data stored in IGFS or IGFS as Hadoop cache?
This depeds on how do you want to access it. If you are accessing data form
Hadoop, then it is not different from any other Hadoop file systems: you can
use FileSystem API, "dfs" command line tool, etc.. If you want to access data
through Ignite API, use IgniteFileSystem API:
https://apacheignite.readme.io/docs/igfs
>> 2. In IGFS as Hadoop cache mode, is core-site.xml need to be configured?
>> (hadoop v2)
Yes, it needs to be configured.
>> - what shall fs.defaultFS be?
"fs.defaultFS" parameter should be set to the same value as
"fs.default.name<http://fs.default.name>" in v1. It should be URI to running
IGFS instance. Please see "Configure Hadoop" and "File System URI" paragraphs
here https://apacheignite.readme.io/docs/file-system for examples.
Let us know if you still have outstanding questions.
>> - where shall this core-site.xml file put, in hadoop path or ignite path or
>> both?
core-site.xml is only needed by Hadoop. Please put it into Hadoop path as usual.
On Sun, Sep 6, 2015 at 7:14 AM, yaoqin
<[email protected]<mailto:[email protected]>> wrote:
1. How to access(set/get) data stored in IGFS or IGFS as Hadoop cache?
2. In IGFS as Hadoop cache mode, is core-site.xml need to be configured?
(hadoop v2)
- what shall fs.defaultFS be?
- where shall this core-site.xml file put, in hadoop path or ignite path or
both?
