You wrote:
I’m running drill as user “drill”.
How are you invoking sqllline? Are you specifying a user "drill"?
You should be able to query the file with two steps:
1) use mfs;
this invokes the plugin
2) select * from `x.pcap`;
Since x.pcap is in the root directory, you don't need to reference mfs again
Thanks.
--Robert
________________________________
From: Takeo Ogawara <ta-ogaw...@kddi-research.jp>
Sent: Wednesday, September 13, 2017 9:17 PM
To: user
Subject: Re: Query Error on PCAP over MapR FS
I used storage plugin named “mfs” with “maprfs:///“.
I modified plugin name from “mfs” to “dfs” and tested a query, but the result
was the same (No such file).
"Hadoop fs -ls / " can find x.pcap
[drill@node21 log]$ hadoop fs -ls / | grep x.pcap
-rw-r--r-- 3 root root 6083 2017-09-13 16:14 /x.pcap
Show files in drill
0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
+---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+
| name | isDirectory | isFile | length | owner | group | permissions |
accessTime | modificationTime |
+---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+
| x.pcap | false | true | 6083 | root | root | rw-r--r-- |
2017-09-13 16:14:52.0 | 2017-09-13 16:14:52.24 |
+---------+--------------+---------+---------+--------+--------+--------------+------------------------+-------------------------+
1 row selected (0.328 seconds)
I’m running drill as user “drill”.
Is there something wrong with file permissions?
> For your testing you can just use the default root volume, but with MapR-FS
> it is a good idea to create volumes for different data/use cases and then
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also
> ease of use for enterprise features like mirroring, snapshots, etc in the
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with
Volumes<https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html>
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance.
A volume is a logical unit that allows you to apply policies to a set of files,
directories, and sub-volumes. A ...
Thank you for the information.
I’ll separate the volume for PCAP from other services.
Thank you.
> 2017/09/13 23:48、Andries Engelbrecht <aengelbre...@mapr.com>のメ�`ル:
>
> Drill is not seeing the file in the location you pointed it.
>
> What did you name the storage plugin?
> The default is normally dfs for the distributed filesystem.
>
> Also did you place the file in the root directory of the dfs?
> What do you get back if you run Hadoop fs �Cls /
>
> For your testing you can just use the default root volume, but with MapR-FS
> it is a good idea to create volumes for different data/use cases and then
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also
> ease of use for enterprise features like mirroring, snapshots, etc in the
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with
Volumes<https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html>
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance.
A volume is a logical unit that allows you to apply policies to a set of files,
directories, and sub-volumes. A ...
>
>
> --Andries
>
>
> On 9/13/17, 12:38 AM, "Takeo Ogawara" <ta-ogaw...@kddi-research.jp> wrote:
>
> Hi,
>
> I modified storage config like this.
>
> "type": "file",
> "enabled": true,
> "connection": "maprfs:///",
> "config": null,
> "workspaces": {
> "root": {
> "location": "/",
> "writable": false,
> "defaultInputFormat": null
> }
> }
>
> But query like “select * from mfs.`x.pcap`” failed.
> Is there any other configuration I should modify?
>
> This is drillbit.log and it seems java.io.FileInputStream is going to open
> MapR FS file path.
>
> Thank you.
>
> 2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
> 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 9 classes for
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
> 2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.c.s.persistence.ScanResult - loading 10 classes for
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
> 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0
> ms, numFiles: 1
> 2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0
> ms, numFiles: 1
> 2017-09-13 16:20:06,178 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0
> ms, numFiles: 1
> 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of
> 1 using 1 threads. Time: 0ms total, 0.847323ms avg, 0ms max.
> 2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman]
> INFO o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of
> 1 using 1 threads. Earliest start: 1.522000 μs, Latest start: 1.522000 μs,
> Average start: 1.522000 μs .
> 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0]
> INFO o.a.d.e.store.pcap.PcapRecordReader - User Error Occurred: /x.pcap (No
> such file or directory) (/x.pcap (No such file or directory))
> org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: /x.pcap
> (No such file or directory)
>
> File name: /x.pcap
>
> [Error Id: 48be766a-8706-407f-8dff-eb563271a4a3 ]
> at
> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
> ~[drill-common-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:109)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:104)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:166)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207)
> [drill-java-exec-1.11.0.jar:1.11.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
> [drill-common-1.11.0.jar:1.11.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_141]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_141]
> at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141]
> Caused by: java.io.FileNotFoundException: /x.pcap (No such file or
> directory)
> at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_141]
> at java.io.FileInputStream.<init>(FileInputStream.java:146)
> ~[na:1.7.0_141]
> at java.io.FileInputStream.<init>(FileInputStream.java:101)
> ~[na:1.7.0_141]
> at
> org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:103)
> [drill-java-exec-1.11.0.jar:1.11.0]
> ... 15 common frames omitted
> 2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested
> AWAITING_ALLOCATION --> FAILED
> 2017-09-13 16:20:06,200 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0: State change requested FAILED -->
> FINISHED
> 2017-09-13 16:20:06,213 [BitServer-4] WARN
> o.a.drill.exec.work.foreman.Foreman - Dropping request to move to COMPLETED
> state as query is already at FAILED state (which is terminal).
> 2017-09-13 16:20:06,214 [BitServer-4] WARN
> o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
> 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0 does not exist.
>
> ―
>
> 97 @Override
> 98 public void setup(final OperatorContext context, final OutputMutator
> output) throws ExecutionSetupException {
> 99 try {
> 100
> 101 this.output = output;
> 102 this.buffer = new byte[100000];
> 103 this.in = new FileInputStream(inputPath);
> 104 this.decoder = new PacketDecoder(in);
> 105 this.validBytes = in.read(buffer);
> 106 this.projectedCols = getProjectedColsIfItNull();
> 107 setColumns(projectedColumns);
> 108 } catch (IOException io) {
> 109 throw UserException.dataReadError(io)
> 110 .addContext("File name:", inputPath)
> 111 .build(logger);
> 112 }
> 113 }
>
>
>> 2017/09/11 23:16、Andries Engelbrecht <aengelbre...@mapr.com>のメ�`ル:
>>
>> Typically when you use the MapR-FS plugin you don’t need to specify the
>> cluster root path in the dfs workspace.
>>
>> Instead of "location": "/mapr/cluster3", use "location": "/",
>>
>> "connection": "maprfs:///", already points to the default MapR cluster root.
>>
>> --Andries
>>
>>
>>
>> On 9/11/17, 2:23 AM, "Takeo Ogawara" <ta-ogaw...@kddi-research.jp> wrote:
>>
>> Dear all,
>>
>> I’m using PCAP storage plugin over MapR FS(5.2.0) with Drill(1.11.0)
>> compiled as follows.
>> $ mvn clean install -DskipTests -Pmapr
>>
>> Some queries caused errors as following.
>> Does anyone know how to solve these errors?
>>
>> 1. Query error when cluster-name is not specified
>> Storage “mfs” setting is this.
>>
>>> "type": "file",
>>> "enabled": true,
>>> "connection": "maprfs:///",
>>> "config": null,
>>> "workspaces": {
>>> "root": {
>>> "location": "/mapr/cluster3",
>>> "writable": false,
>>> "defaultInputFormat": null
>>> }
>>> }
>>
>>
>> With this setting, the following query failed.
>>> select * from mfs.`x.pcap` ;
>>> Error: DATA_READ ERROR: /x.pcap (No such file or directory)
>>>
>>> File name: /x.pcap
>>> Fragment 0:0
>>>
>>> [Error Id: 70b73062-c3ed-4a10-9a88-034b4e6d039a on node21:31010]
>>> (state=,code=0)
>>
>> But these queries passed.
>>> select * from mfs.root.`x.pcap` ;
>>> select * from mfs.`x.csv`;
>>> select * from mfs.root.`x.csv`;
>>
>> 2. Large PCAP file
>> Query on very large PCAP file (larger than 100GB) failed with following
>> error message.
>>> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
>>>
>>> Fragment 1:169
>>>
>>> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010]
>>> (state=,code=0)
>>
>> This happens even on Linux FS not MapR FS
>>
>> Thank you.
>>
>>
>>
>>
>>
>>
>
> ―――――――――――――――――――――――
> <KDDI�t合研究所 ビジョン>
> Challenge for the future �Nかな未来への挑��
> ―――――――――――――――――――――――
> 英雄だけの夏。
> https://www.au.com/pr/cm/3taro/
> ―――――――――――――――――――――――
> 小河原 健生（Takeo Ogawara）
> (株)KDDI�t合研究所
> コネクティッドカ�`1G
>
> TEL:049-278-7495 / 070-3623-9914
>
>
>
