[ 
https://issues.apache.org/jira/browse/HIVE-30248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

László Bodor updated HIVE-30248:
--------------------------------
    Description: 
{{ParquetColumnVectorProducer.createReadPipeline}} turns down a split whenever 
the footer read throws an {{IOException}}. It returns {{null}}, and 
{{LlapInputFormat.fallback()}} then passes the split to the stock 
{{VectorizedParquetRecordReader}} (HIVE-30059 design doc, §6.1 and §7: "Footer 
read throws IOException → producer logs at WARN and returns null").

That is correct when the file itself fails to read, but not when the 
{{IOException}} comes from an interrupt, for example when the fragment is being 
killed or the daemon is shutting down. In that case the fallback starts the 
same footer read again on the stock reader, on a thread that should be 
unwinding. It also logs a WARN with a stack trace saying the split "lost its 
vectored reads", which hides what actually happened.

> LLAP IO native Parquet cache should not fall back to the normal reader on an 
> interrupt
> --------------------------------------------------------------------------------------
>
>                 Key: HIVE-30248
>                 URL: https://issues.apache.org/jira/browse/HIVE-30248
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>
> {{ParquetColumnVectorProducer.createReadPipeline}} turns down a split 
> whenever the footer read throws an {{IOException}}. It returns {{null}}, and 
> {{LlapInputFormat.fallback()}} then passes the split to the stock 
> {{VectorizedParquetRecordReader}} (HIVE-30059 design doc, §6.1 and §7: 
> "Footer read throws IOException → producer logs at WARN and returns null").
> That is correct when the file itself fails to read, but not when the 
> {{IOException}} comes from an interrupt, for example when the fragment is 
> being killed or the daemon is shutting down. In that case the fallback starts 
> the same footer read again on the stock reader, on a thread that should be 
> unwinding. It also logs a WARN with a stack trace saying the split "lost its 
> vectored reads", which hides what actually happened.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to