Re: Query Error on PCAP over MapR FS

[Takeo Ogawara]

I don’t specify the user name in sqlline command.
I meant I started Drill from Linux user “drill”.
[drill@node21 ~]$ ./apache-drill-1.11.0/bin/sqlline -u 
jdbc:drill:zk=node21:5181,node22:5181,node23:5181/drill/cluster3-drillbits
apache drill 1.11.0
"the only truly happy people are children, the creative minority and drill 
users"
0: jdbc:drill:zk=node21:5181,node22:5181,node> use dfs;
+---+--+
|  ok   | summary  |
+---+--+
| true  | Default schema changed to [dfs]  |
+---+--+
1 row selected (0.811 seconds)
0: jdbc:drill:zk=node21:5181,node22:5181,node> select * from `x.pcap`;
Error: DATA_READ ERROR: /x.pcap (No such file or directory)

File name: /x.pcap
Fragment 0:0

[Error Id: d6c1191a-ff79-4c39-96d3-0ae9e0be3ae9 on node25:31010] (state=,code=0)
0: jdbc:drill:zk=node21:5181,node22:5181,node> show files in  `x.pcap`;
+-+--+-+-+++--++-+
|  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  | 
  accessTime   |modificationTime |
+-+--+-+-+++--++-+
| x.pcap  | false| true| 6083| root   | root   | rw-r--r--| 
2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
+-+--+-+-+++--++-+
1 row selected (0.241 seconds)

Drillbits config is as follows.
drill.exec: {
  cluster-id: "cluster3-drillbits",
  zk.connect: "node21:5181,node22:5181,node23:5181”
}

Storage plugin has config for PCAP.
"pcap": {
  "type": "pcap"
},

Is it better to access via NFS to MapR FS?
I can access file:///mapr/cluster3/x.pcap in Drill sqlline.

Thank you.

> 2017/09/14 14:27、Robert Hou のメール:
> 
> You wrote:
> 
>   IӮm running drill as user ӡdrillӱ.
> 
> 
> How are you invoking sqllline?  Are you specifying a user "drill"?
> 
> 
> You should be able to query the file with two steps:
> 
> 
> 1) use mfs;
> 
> 
> this invokes the plugin
> 
> 
> 2) select * from `x.pcap`;
> 
> 
> Since x.pcap is in the root directory, you don't need to reference mfs again
> 
> 
> 
> Thanks.
> 
> --Robert
> 
> 
> From: Takeo Ogawara 
> Sent: Wednesday, September 13, 2017 9:17 PM
> To: user
> Subject: Re: Query Error on PCAP over MapR FS
> 
> I used storage plugin named ”°mfs”± with ”°maprfs:///”°.
> I modified plugin name from ”°mfs”± to ”°dfs”± and tested a query, but the 
> result was the same (No such file).
> 
> "Hadoop fs -ls / " can find x.pcap
> [drill@node21 log]$ hadoop fs -ls / | grep x.pcap
> -rw-r--r--   3 root root 6083 2017-09-13 16:14 /x.pcap
> 
> Show files in drill
> 0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
> +-+--+-+-+++--++-+
> |  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  
> |   accessTime   |modificationTime |
> +-+--+-+-+++--++-+
> | x.pcap  | false| true| 6083| root   | root   | rw-r--r--
> | 2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
> +-+--+-+-+++--++-+
> 1 row selected (0.328 seconds)
> 
> IӮm running drill as user ӡdrillӱ.
> Is there something wrong with file permissions?
> 
> 
>> For your testing you can just use the default root volume, but with MapR-FS 
>> it is a good idea to create volumes for different data/use cases and then 
>> mount these volumes on MapR-FS.
>> This allows for benefits like topology, quota & security management; also 
>> ease of use for enterprise features like mirroring, snapshots, etc in the 
>> future to name a few.
>> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
> Managing Data with 
> Volumes
> maprdocs.mapr.com
> MapR provides volumes as a way to organize data and manage cluster 
> performance. A volume is a logical unit that allows you to apply policies to 
> a set of files, directories, and sub-volumes. A ...
> 
> 
> 
> 
> Thank you for the information.
> IӮll separate the volume for PCAP from other services.
> 
> Thank you.
> 
>> 2017/09/13 23:48”¢Andries Engelbrecht ¤Ī„į©`„ė:
>> 
>> Drill is not seeing the file in the location you pointed it.
>> 
>> What did you name the storage plugin?
>> The default is normally 

Re: Query Error on PCAP over MapR FS

[Robert Hou]

This might work:


"pcap": {

 "type": "pcap"

}

Thanks.

--Robert



From: Arjun kr 
Sent: Wednesday, September 13, 2017 10:22 PM
To: user@drill.apache.org
Subject: Re: Query Error on PCAP over MapR FS


I have not used pcap storage format before. Doesn't it require specific format 
defined in storage plugin ( as psv format given below)?


"formats": {

"psv": {
  "type": "text",
  "extensions": [
"psv"
  ],
  "delimiter": "|"
},


Thanks,

Arjun



From: Takeo Ogawara 
Sent: Thursday, September 14, 2017 9:47 AM
To: user
Subject: Re: Query Error on PCAP over MapR FS

I used storage plugin named “mfs” with “maprfs:///“.
I modified plugin name from “mfs” to “dfs” and tested a query, but the result 
was the same (No such file).

"Hadoop fs -ls / " can find x.pcap
[drill@node21 log]$ hadoop fs -ls / | grep x.pcap
-rw-r--r--   3 root root 6083 2017-09-13 16:14 /x.pcap

Show files in drill
0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
+-+--+-+-+++--++-+
|  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  | 
  accessTime   |modificationTime |
+-+--+-+-+++--++-+
| x.pcap  | false| true| 6083| root   | root   | rw-r--r--| 
2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
+-+--+-+-+++--++-+
1 row selected (0.328 seconds)

I’m running drill as user “drill”.
Is there something wrong with file permissions?


> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...



Managing Data with 
Volumes
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...



maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...




Thank you for the information.
I’ll separate the volume for PCAP from other services.

Thank you.

> 2017/09/13 23:48、Andries Engelbrecht のメ�`ル:
>
> Drill is not seeing the file in the location you pointed it.
>
> What did you name the storage plugin?
> The default is normally dfs for the distributed filesystem.
>
> Also did you place the file in the root directory of the dfs?
> What do you get back if you run Hadoop fs �Cls /
>
> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...



Managing Data with 
Volumes
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 

Re: Query Error on PCAP over MapR FS

[Robert Hou]

You wrote:

   I’m running drill as user “drill”.


How are you invoking sqllline?  Are you specifying a user "drill"?


You should be able to query the file with two steps:


1) use mfs;


this invokes the plugin


2) select * from `x.pcap`;


Since x.pcap is in the root directory, you don't need to reference mfs again



Thanks.

--Robert


From: Takeo Ogawara 
Sent: Wednesday, September 13, 2017 9:17 PM
To: user
Subject: Re: Query Error on PCAP over MapR FS

I used storage plugin named “mfs” with “maprfs:///“.
I modified plugin name from “mfs” to “dfs” and tested a query, but the result 
was the same (No such file).

"Hadoop fs -ls / " can find x.pcap
[drill@node21 log]$ hadoop fs -ls / | grep x.pcap
-rw-r--r--   3 root root 6083 2017-09-13 16:14 /x.pcap

Show files in drill
0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
+-+--+-+-+++--++-+
|  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  | 
  accessTime   |modificationTime |
+-+--+-+-+++--++-+
| x.pcap  | false| true| 6083| root   | root   | rw-r--r--| 
2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
+-+--+-+-+++--++-+
1 row selected (0.328 seconds)

I’m running drill as user “drill”.
Is there something wrong with file permissions?


> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...




Thank you for the information.
I’ll separate the volume for PCAP from other services.

Thank you.

> 2017/09/13 23:48、Andries Engelbrecht のメ�`ル:
>
> Drill is not seeing the file in the location you pointed it.
>
> What did you name the storage plugin?
> The default is normally dfs for the distributed filesystem.
>
> Also did you place the file in the root directory of the dfs?
> What do you get back if you run Hadoop fs �Cls /
>
> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...



>
>
> --Andries
>
>
> On 9/13/17, 12:38 AM, "Takeo Ogawara"  wrote:
>
>Hi,
>
>I modified storage config like this.
>
>"type": "file",
> "enabled": true,
> "connection": "maprfs:///",
> "config": null,
> "workspaces": {
>   "root": {
> "location": "/",
> "writable": false,
> "defaultInputFormat": null
>   }
> }
>
>But query like “select * from mfs.`x.pcap`” failed.
>Is there any other configuration I should modify?
>
>This is drillbit.log and it seems java.io.FileInputStream is going to open 
> MapR FS file path.
>
>Thank you.
>
>2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 

Re: Query Error on PCAP over MapR FS

[Arjun kr]

I have not used pcap storage format before. Doesn't it require specific format 
defined in storage plugin ( as psv format given below)?


"formats": {

"psv": {
  "type": "text",
  "extensions": [
"psv"
  ],
  "delimiter": "|"
},


Thanks,

Arjun



From: Takeo Ogawara 
Sent: Thursday, September 14, 2017 9:47 AM
To: user
Subject: Re: Query Error on PCAP over MapR FS

I used storage plugin named “mfs” with “maprfs:///“.
I modified plugin name from “mfs” to “dfs” and tested a query, but the result 
was the same (No such file).

"Hadoop fs -ls / " can find x.pcap
[drill@node21 log]$ hadoop fs -ls / | grep x.pcap
-rw-r--r--   3 root root 6083 2017-09-13 16:14 /x.pcap

Show files in drill
0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
+-+--+-+-+++--++-+
|  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  | 
  accessTime   |modificationTime |
+-+--+-+-+++--++-+
| x.pcap  | false| true| 6083| root   | root   | rw-r--r--| 
2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
+-+--+-+-+++--++-+
1 row selected (0.328 seconds)

I’m running drill as user “drill”.
Is there something wrong with file permissions?


> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...




Thank you for the information.
I’ll separate the volume for PCAP from other services.

Thank you.

> 2017/09/13 23:48、Andries Engelbrecht のメ�`ル:
>
> Drill is not seeing the file in the location you pointed it.
>
> What did you name the storage plugin?
> The default is normally dfs for the distributed filesystem.
>
> Also did you place the file in the root directory of the dfs?
> What do you get back if you run Hadoop fs �Cls /
>
> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
Managing Data with 
Volumes
maprdocs.mapr.com
MapR provides volumes as a way to organize data and manage cluster performance. 
A volume is a logical unit that allows you to apply policies to a set of files, 
directories, and sub-volumes. A ...



>
>
> --Andries
>
>
> On 9/13/17, 12:38 AM, "Takeo Ogawara"  wrote:
>
>Hi,
>
>I modified storage config like this.
>
>"type": "file",
> "enabled": true,
> "connection": "maprfs:///",
> "config": null,
> "workspaces": {
>   "root": {
> "location": "/",
> "writable": false,
> "defaultInputFormat": null
>   }
> }
>
>But query like “select * from mfs.`x.pcap`” failed.
>Is there any other configuration I should modify?
>
>This is drillbit.log and it seems java.io.FileInputStream is going to open 
> MapR FS file path.
>
>Thank you.
>
>2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  

Re: How to query values in list

[Robert Hou]

Good to know!


Thanks.


--Robert



From: gus 
Sent: Wednesday, September 13, 2017 7:28 PM
To: user@drill.apache.org
Subject: Re: How to query values in list

Dear Robert, it worked!

I also needed to change this:

store.mongo.all_text_mode = true

thanks!,
gus

Em Thu, Sep 14, 2017 at 12:47:31AM +, Robert Hou escreveu:
> Try:
>
>
>select t.v11._ from dfs.``  t where t.v11._ = '0070';
>
>
> This works for json.  Try it with MongoDB.
>
>
> Thanks.
>
>
> --Robert
>
> 
> From: gus 
> Sent: Wednesday, September 13, 2017 11:26 AM
> To: user@drill.apache.org
> Subject: How to query values in list
>
> Hi, in this example[1], what's the syntax in apache drill to query "v11._" 
> with the value of "0070"?
> I'm using MongoDB.
>
> thanks!,
> gus
>
> [1] example:
> https://share.riseup.net/#PgfEnTO7jEXPbpA5QOgcRg
share.riseup.net
share.riseup.net
CLI - Contact - Up1 Upload is currently limited to 50mb and files are stored no 
longer than a week!



> share.riseup.net
share.riseup.net
share.riseup.net
CLI - Contact - Up1 Upload is currently limited to 50mb and files are stored no 
longer than a week!



> share.riseup.net
> CLI - Contact - Up1 Upload is currently limited to 50mb and files are stored 
> no longer than a week!
>
>
>

Re: Query Error on PCAP over MapR FS

[Takeo Ogawara]

I used storage plugin named “mfs” with “maprfs:///“.
I modified plugin name from “mfs” to “dfs” and tested a query, but the result 
was the same (No such file).

"Hadoop fs -ls / " can find x.pcap
[drill@node21 log]$ hadoop fs -ls / | grep x.pcap
-rw-r--r--   3 root root 6083 2017-09-13 16:14 /x.pcap

Show files in drill 
0: jdbc:drill:drillbit=localhost> show files in dfs.`x.pcap`;
+-+--+-+-+++--++-+
|  name   | isDirectory  | isFile  | length  | owner  | group  | permissions  | 
  accessTime   |modificationTime |
+-+--+-+-+++--++-+
| x.pcap  | false| true| 6083| root   | root   | rw-r--r--| 
2017-09-13 16:14:52.0  | 2017-09-13 16:14:52.24  |
+-+--+-+-+++--++-+
1 row selected (0.328 seconds)

I’m running drill as user “drill”.
Is there something wrong with file permissions?


> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html

Thank you for the information.
I’ll separate the volume for PCAP from other services.

Thank you.

> 2017/09/13 23:48、Andries Engelbrecht のメール:
> 
> Drill is not seeing the file in the location you pointed it.
> 
> What did you name the storage plugin?
> The default is normally dfs for the distributed filesystem.
> 
> Also did you place the file in the root directory of the dfs?
> What do you get back if you run Hadoop fs –ls / 
> 
> For your testing you can just use the default root volume, but with MapR-FS 
> it is a good idea to create volumes for different data/use cases and then 
> mount these volumes on MapR-FS.
> This allows for benefits like topology, quota & security management; also 
> ease of use for enterprise features like mirroring, snapshots, etc in the 
> future to name a few.
> https://maprdocs.mapr.com/home/AdministratorGuide/c_managing_data_with_volumes.html
> 
> 
> --Andries
> 
> 
> On 9/13/17, 12:38 AM, "Takeo Ogawara"  wrote:
> 
>Hi, 
> 
>I modified storage config like this.
> 
>"type": "file",
> "enabled": true,
> "connection": "maprfs:///",
> "config": null,
> "workspaces": {
>   "root": {
> "location": "/",
> "writable": false,
> "defaultInputFormat": null
>   }
> }
> 
>But query like “select * from mfs.`x.pcap`” failed.
>Is there any other configuration I should modify?
> 
>This is drillbit.log and it seems java.io.FileInputStream is going to open 
> MapR FS file path.
> 
>Thank you.
> 
>2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
> org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
>2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
> org.apache.drill.common.logical.FormatPluginConfig took 0ms
>2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 
> ms, numFiles: 1
>2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] 
> INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 
> ms, numFiles: 1
>2017-09-13 

Re: How to query values in list

[Robert Hou]

Try:


   select t.v11._ from dfs.``  t where t.v11._ = '0070';


This works for json.  Try it with MongoDB.


Thanks.


--Robert


From: gus 
Sent: Wednesday, September 13, 2017 11:26 AM
To: user@drill.apache.org
Subject: How to query values in list

Hi, in this example[1], what's the syntax in apache drill to query "v11._" with 
the value of "0070"?
I'm using MongoDB.

thanks!,
gus

[1] example:
https://share.riseup.net/#PgfEnTO7jEXPbpA5QOgcRg
share.riseup.net
share.riseup.net
CLI - Contact - Up1 Upload is currently limited to 50mb and files are stored no 
longer than a week!

How to query values in list

[gus]

Hi, in this example[1], what's the syntax in apache drill to query "v11._" with 
the value of "0070"?
I'm using MongoDB.

thanks!,
gus

[1] example:
https://share.riseup.net/#PgfEnTO7jEXPbpA5QOgcRg

Re: Workaround for drill queries during node failure

[John Omernik]

As long as the nodes are "up" during the planning phase they will be
included in the planning. If they go "down" after planning, (i.e. during
execution) and fragments are requested, they will not report, and will fail
the query.

So if you start off with 5 nodes, but node 4 is down for patches, and then
you submit a query, the remaining 4 nodes will execute the query with no
issues.

If you start out with 5 nodes, all up and running, you submit a query, and
during execution, you take down node 4, that query will fail.

There is work being done to help with "taking down" nodes sanely to help
with patching.

Resending work expected from a certain node to another node is "hard" if I
understand past email chains correctly. You can't just retry fragments on
other nodes once Node 4 goes down. It will take a Drill expert to explain
that better than I can though.

John


On Mon, Sep 11, 2017 at 12:30 PM, Padma Penumarthy 
wrote:

> Did you mean to say “we could not execute any queries” ?
>
> Need more details about configuration you have.
> When you say data is available on other nodes, is it because you
> have replication configured (assuming it is DFS) ?
>
> What exactly are you trying and what error you see when you try to
> execute the query ?
>
> Thanks,
> Padma
>
>
> On Sep 11, 2017, at 9:40 AM, Kshitija Shinde > wrote:
>
> Hi,
>
> We have installed drill in distributed mode. While testing drillbit we have
> observed that if one of node is done then we could execute any queries
> against the drill even if data is available on other nodes.
>
>
>
> Is there any workaround for this?
>
>
>
> Thanks,
>
> Kshitija
>
>

Re: Query Error on PCAP over MapR FS

[Takeo Ogawara]

Hi, 

I modified storage config like this.

"type": "file",
 "enabled": true,
 "connection": "maprfs:///",
 "config": null,
 "workspaces": {
   "root": {
 "location": "/",
 "writable": false,
 "defaultInputFormat": null
   }
 }

But query like “select * from mfs.`x.pcap`” failed.
Is there any other configuration I should modify?

This is drillbit.log and it seems java.io.FileInputStream is going to open MapR 
FS file path.

Thank you.

2017-09-13 16:20:06,123 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,124 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,125 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 9 classes for 
org.apache.drill.exec.store.dfs.FormatPlugin took 0ms
2017-09-13 16:20:06,145 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,146 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.c.s.persistence.ScanResult - loading 10 classes for 
org.apache.drill.common.logical.FormatPluginConfig took 0ms
2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-09-13 16:20:06,170 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-09-13 16:20:06,178 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, 
numFiles: 1
2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Time: 0ms total, 0.847323ms avg, 0ms max.
2017-09-13 16:20:06,179 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:foreman] INFO  
o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 
1 threads. Earliest start: 1.522000 μs, Latest start: 1.522000 μs, Average 
start: 1.522000 μs .
2017-09-13 16:20:06,199 [264723d8-bcba-6330-c9be-1c9c95dd2aa6:frag:0:0] INFO  
o.a.d.e.store.pcap.PcapRecordReader - User Error Occurred: /x.pcap (No such 
file or directory) (/x.pcap (No such file or directory))
org.apache.drill.common.exceptions.UserException: DATA_READ ERROR: /x.pcap (No 
such file or directory)

File name: /x.pcap

[Error Id: 48be766a-8706-407f-8dff-eb563271a4a3 ]
at 
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
 ~[drill-common-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.store.pcap.PcapRecordReader.setup(PcapRecordReader.java:109)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ScanBatch.(ScanBatch.java:104) 
[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin.getReaderBatch(EasyFormatPlugin.java:166)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:35)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.store.dfs.easy.EasyReaderBatchCreator.getBatch(EasyReaderBatchCreator.java:28)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:156)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch(ImplCreator.java:136)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getChildren(ImplCreator.java:179)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getRootExec(ImplCreator.java:109)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.physical.impl.ImplCreator.getExec(ImplCreator.java:87) 
[drill-java-exec-1.11.0.jar:1.11.0]
at 
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:207)
 [drill-java-exec-1.11.0.jar:1.11.0]
at 

Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS

[Robert Hou]

Great, thank you.


--Robert



From: Takeo Ogawara 
Sent: Tuesday, September 12, 2017 6:59 PM
To: user@drill.apache.org
Cc: j...@apache.org
Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS

Hi,

I opened the issue at DRILL-5785.
Please check if that contains enough information.

>>> Also, it appears that you have tried to query your data across many small
>>> files rather than one large file.  This is the preferred approach, and it
>>> seems that this approach works for you.  Please let me know if you think
>>> otherwise, that you need to access your data in one large PCAP file.
There is no problem with analyzing across many small files.
But, I want to know how large file is acceptable or small in Drill.

> So then there should be one fragment reading the data.  Takeo, if you file a 
> Jira, perhaps you can include the Drill profile, and we can look at how many 
> fragments are used by the query.  The profiles can be found via a web URL

I attached query profile to the issue.


Thank you.


> 2017/09/13 8:50、Robert Hou のメ�`ル:
>
> So then there should be one fragment reading the data.  Takeo, if you file a 
> Jira, perhaps you can include the Drill profile, and we can look at how many 
> fragments are used by the query.  The profiles can be found via a web URL
>
>
>   http://:8047
>
>
> Thanks.
>
>
> --Robert
>
> 
> From: Ted Dunning 
> Sent: Tuesday, September 12, 2017 4:30 PM
> To: user
> Cc: j...@apache.org
> Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS
>
> PCAP is a binary format that cannot easily be split.
>
>
>
> On Wed, Sep 13, 2017 at 1:15 AM, Robert Hou  wrote:
>
>> Hi Ted,
>>
>>
>> My understanding is, by default, Drill should not have multiple threads
>> reading the PCAP file in parallel.  But if the records in a PCAP file can
>> be read in parallel (e.g. rows are restricted to one line each), then the
>> plugin can be designed to read the file in parallel.
>>
>>
>> Are PCAP records single-line records?
>>
>>
>> Thanks.
>>
>>
>> --Robert
>>
>> 
>> From: Ted Dunning 
>> Sent: Tuesday, September 12, 2017 3:54 PM
>> To: user
>> Cc: j...@apache.org
>> Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS
>>
>> Robert,
>>
>> Thanks for looking at this. I think that this is a bug in the way that we
>> wrote the format plugin in that it isn't telling Drill to not split the
>> file.
>>
>>
>>
>> On Wed, Sep 13, 2017 at 12:52 AM, Robert Hou  wrote:
>>
>>> I asked a couple of Drill developers.  We don't have much experience with
>>> PCAP yet.  Takeo, can you file a Jira for this, and include the
>> information
>>> below?  The error message mentions a bad magic number, which Drill
>>> sometimes uses to help determine the file format.
>>>
>>>
>>> Also, it appears that you have tried to query your data across many small
>>> files rather than one large file.  This is the preferred approach, and it
>>> seems that this approach works for you.  Please let me know if you think
>>> otherwise, that you need to access your data in one large PCAP file.
>>>
>>>
>>> Thanks.
>>>
>>>
>>> --Robert
>>>
>>>
>>> 
>>> From: Ted Dunning 
>>> Sent: Monday, September 11, 2017 8:15 PM
>>> To: user; j...@apache.org
>>> Subject: Re: ***UNCHECKED*** Re: Query Error on PCAP over MapR FS
>>>
>>> This stack trace makes it clear that this is a bug in the PCAP decoder
>>> caused by a misunderstanding of how to force large files to be read in
>> one
>>> batch on a single drillBit.
>>>
>>> Are there some real Drill experts out there who can provide hints about
>> how
>>> to avoid this?
>>>
>>>
>>>
>>> On Tue, Sep 12, 2017 at 5:03 AM, Takeo Ogawara <
>>> ta-ogaw...@kddi-research.jp>
>>> wrote:
>>>
 Sorry

 I paste plain texts.

> 2017-09-11 15:06:52,390 [BitServer-2] WARN  o.a.d.exec.rpc.control.
>>> WorkEventBus
 - A fragment message arrived but there was no registered listener for
>>> that
 message: profile {
>  state: FAILED
>  error {
>error_id: "bbf284b6-9da4-4869-ac20-fa100eed11b9"
>endpoint {
>  address: "node22"
>  user_port: 31010
>  control_port: 31011
>  data_port: 31012
>  version: "1.11.0"
>}
>error_type: SYSTEM
>message: "SYSTEM ERROR: IllegalStateException: Bad magic number =
 0a0d0d0a\n\nFragment 1:200\n\n[Error Id: bbf284b6-9da4-4869-ac20-
>>> fa100eed11b9
 on node22:31010]"
>exception {
>  exception_class: "java.lang.IllegalStateException"
>  message: "Bad magic number = 0a0d0d0a"
>  stack_trace {
>class_name: "com.google.common.base.Preconditions"
>file_name: "Preconditions.java"
>line_number: 173
>