[
https://issues.apache.org/jira/browse/DRILL-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953257#comment-14953257
]
Michael England commented on DRILL-1131:
----------------------------------------
Has this been fixed in Drill 1.2? The related JIRA DRILL-2424 was closed as a
duplicate, but surely this shouldn't be limited to parquet files? If you stream
files into Hadoop with flume, it renames .tmp files once it completes writing
to them. If you run a drill query on a folder containing these .tmp files and
it renames it during the query, the query fails - making it useless for
querying hot data.
> Drill should ignore files in starting with . _
> ----------------------------------------------
>
> Key: DRILL-1131
> URL: https://issues.apache.org/jira/browse/DRILL-1131
> Project: Apache Drill
> Issue Type: New Feature
> Components: Storage - Parquet
> Reporter: Ramana Inukonda Nagaraj
> Fix For: Future
>
>
> Files containing . and _ as the first characters are ignored by hive and
> others are these are typically logs and status files written out by tools
> like mapreduce. Drill should not read them when querying a directory
> containing a list of parquet files.
> Currently it fails with the error:
> message: "Failure while setting up Foreman. < AssertionError:[ Internal
> error: Error while applying rule DrillPushProjIntoScan, args
> [rel#78:ProjectRel.NONE.ANY([]).[](child=rel#15:Subset#1.ENUMERABLE.ANY([]).[],p_partkey=$1,p_type=$2),
> rel#8:EnumerableTableAccessRel.ENUMERABLE.ANY([]).[](table=[dfs,
> drillTestDirDencTpchSF100, part])] ] < DrillRuntimeException:[
> java.io.IOException: Could not read footer: java.io.IOException: Could not
> read footer for file com.mapr.fs.MapRFileStatus@99c9d45e ] < IOException:[
> Could not read footer: java.io.IOException: Could not read footer for file
> com.mapr.fs.MapRFileStatus@99c9d45e ] < IOException:[ Could not read footer
> for file com.mapr.fs.MapRFileStatus@99c9d45e ] < IOException:[ Open failed
> for file: /drill/testdata/dencSF100/part/.impala_insert_staging, error:
> Invalid argument (22) ]"
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
