Hi,
I modified storage config like this.
"type": "file",
"enabled": true,
"connection": "maprfs:///",
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null
}
}
But query like “select * from mfs.`x.pcap`” failed.
Is there any other configuration I should modify?
This is drillbit.log and it seems java.io.FileInputStream is going to open MapR
FS file path.
Thank you.
2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 9 classes for
org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 10 classes for
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 10 classes for
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 10 classes for
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 9 classes for
org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 10 classes for
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.c.s.persistence.ScanResult - loading 10 classes for
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-09-13 16:20:06,178 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
numFiles: 1
2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Time: 0ms total, 0.847323ms avg, 0ms max.
2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using
1 threads. Earliest start: 1.522000 μs, Latest start: 1.522000 μs, Average
start: 1.522000 μs .
2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO
o.a.d.e.store.pcap.PcapRecordReader - User Error Occurred: /x.pcap (No such
file or directory) (/x.pcap (No such file or directory))
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: /x.pcap (No
such file or directory)
File name: /x.pcap
[Error Id: 48be766a-8706-407f-8dff-eb563271a4a3 ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
~[drill-common-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:109)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ScanBatch.<init>(ScanBatch.java:104)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:166)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.11.0.jar:1.11.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_141]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_141]
at java.lang.Thread.run(Thread.java:748) [na:1.7.0_141]
Caused by: java.io.FileNotFoundException: /x.pcap (No such file or directory)
at java.io.FileInputStream.open(Native Method) ~[na:1.7.0_141]
at java.io.FileInputStream.<init>(FileInputStream.java:146)
~[na:1.7.0_141]
at java.io.FileInputStream.<init>(FileInputStream.java:101)
~[na:1.7.0_141]
at
org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:103)
[drill-java-exec-1.11.0.jar:1.11.0]
... 15 common frames omitted
2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0:
State change requested AWAITING_ALLOCATION --> FAILED
2017-09-13 16:20:06,200 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0:
State change requested FAILED --> FINISHED
2017-09-13 16:20:06,213 [BitServer-4] WARN o.a.drill.exec.work.foreman.Foreman
- Dropping request to move to COMPLETED state as query is already at FAILED
state (which is terminal).
2017-09-13 16:20:06,214 [BitServer-4] WARN o.a.d.e.w.b.ControlMessageHandler -
Dropping request to cancel fragment. 264723d8-bcba-6330-c9be-1c9c95dd2aa6:0:0
does not exist.
—
97 @Override
98 public void setup(final OperatorContext context, final OutputMutator
output) throws ExecutionSetupException {
99 try {
100
101 this.output = output;
102 this.buffer = new byte[100000];
103 this.in = new FileInputStream(inputPath);
104 this.decoder = new PacketDecoder(in);
105 this.validBytes = in.read(buffer);
106 this.projectedCols = getProjectedColsIfItNull();
107 setColumns(projectedColumns);
108 } catch (IOException io) {
109 throw UserException.dataReadError(io)
110 .addContext("File name:", inputPath)
111 .build(logger);
112 }
113 }
> 2017/09/11 23:16、Andries Engelbrecht <[email protected]>のメール:
>
> Typically when you use the MapR-FS plugin you don’t need to specify the
> cluster root path in the dfs workspace.
>
> Instead of "location": "/mapr/cluster3", use "location": "/",
>
> "connection": "maprfs:///", already points to the default MapR cluster root.
>
> --Andries
>
>
>
> On 9/11/17, 2:23 AM, "Takeo Ogawara" <[email protected]> wrote:
>
> Dear all,
>
> I’m using PCAP storage plugin over MapR FS(5.2.0) with Drill(1.11.0)
> compiled as follows.
> $ mvn clean install -DskipTests -Pmapr
>
> Some queries caused errors as following.
> Does anyone know how to solve these errors?
>
> 1. Query error when cluster-name is not specified
> Storage “mfs” setting is this.
>
>> "type": "file",
>> "enabled": true,
>> "connection": "maprfs:///",
>> "config": null,
>> "workspaces": {
>> "root": {
>> "location": "/mapr/cluster3",
>> "writable": false,
>> "defaultInputFormat": null
>> }
>> }
>
>
> With this setting, the following query failed.
>> select * from mfs.`x.pcap` ;
>> Error: DATA_READ ERROR: /x.pcap (No such file or directory)
>>
>> File name: /x.pcap
>> Fragment 0:0
>>
>> [Error Id: 70b73062-c3ed-4a10-9a88-034b4e6d039a on node21:31010]
>> (state=,code=0)
>
> But these queries passed.
>> select * from mfs.root.`x.pcap` ;
>> select * from mfs.`x.csv`;
>> select * from mfs.root.`x.csv`;
>
> 2. Large PCAP file
> Query on very large PCAP file (larger than 100GB) failed with following
> error message.
>> Error: SYSTEM ERROR: IllegalStateException: Bad magic number = 0a0d0d0a
>>
>> Fragment 1:169
>>
>> [Error Id: 8882c359-c253-40c0-866c-417ef1ce5aa3 on node22:31010]
>> (state=,code=0)
>
> This happens even on Linux FS not MapR FS
>
> Thank you.
>
>
>
>
>
>
———————————————————————
<KDDI総合研究所 ビジョン>
Challenge for the future 豊かな未来への挑戦
———————————————————————
英雄だけの夏。
https://www.au.com/pr/cm/3taro/
———————————————————————
小河原 健生（Takeo Ogawara）
(株)KDDI総合研究所
コネクティッドカー1G
TEL:049-278-7495 / 070-3623-9914
