Hi Tilak,
Drill's format matching mechanism uses the file suffix (only). Drill has no way
to match on any other part of the name. So, in order to get Drill to read your
file, you either need a file suffix that matches a format config, or define the
default config to be the one you want. That is, where you have
"defaultInputFormat": null,
Try changing it to
"defaultInputFormat": "csv",
Caveat: I've not tested this just now, but I believe I've used this trick in
the past...
Thanks,
- Paul
On Tuesday, May 8, 2018, 6:06:26 AM PDT, Surneni Tilak
<[email protected]> wrote:
Hi Team,
I am trying to query the part files which are of type text(Encoding format
ASCII ) but with no file extension. Those part files are generated using
Sparkbench suite.
Please find the below error message:
Query Failed: An Error Occurred
org.apache.drill.common.exceptions.UserRemoteException: VALIDATION ERROR: From
line 1, column 15 to line 1, column 17: Table
'dfs./mapr/mycluster/NSC-TEST/large_inputs01/PageRank/Input328141MB/part-00219'
not found SQL Query null [Error Id: b15a3b3b-3a32-493c-a2b7-1490e1f00d3c on
node1:31010]
My workspace details:
{
"type": "file",
"enabled": true,
"connection": "file:///",
"config": null,
"workspaces": {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"httpd": {
"type": "httpd",
"logFormat": "%h %t \"%r\" %>s %b \"%{Referer}i\""
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json"
]
},
"pcap": {
"type": "pcap"
},
"avro": {
"type": "avro"
},
"sequencefile": {
"type": "sequencefile",
"extensions": [
"seq"
]
},
"csvh": {
"type": "text",
"extensions": [
"csvh"
],
"extractHeader": true,
"delimiter": ","
}
}
}
Best regards,
_________________________
Tilak
-----Original Message-----
From: Parth Chandra [mailto:[email protected]]
Sent: Monday, May 7, 2018 6:55 PM
To: [email protected]
Subject: Re: Not Able to Query Part files Using Drill
What part files are these? Can you share the workspace settings? Also what is
the detailed error message you're getting.
On Mon, May 7, 2018 at 3:37 AM, Surneni Tilak <[email protected]>
wrote:
> Hi Team,
>
> I am trying to run drill query on part files present in the local file
> system. But Drill is throwing Table not found error , I think it is
> not able to identify the file format and giving that error. I have
> tried to include default input format option mentioned in the Drill
> website in my workspace even then same error is repeating. Please help me on
> this.
>
> Best regards,
> _________________________
> Tilak
>
>
>
