[
https://issues.apache.org/jira/browse/HIVE-23824?focusedWorklogId=457970&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-457970
]
ASF GitHub Bot logged work on HIVE-23824:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Jul/20 10:34
Start Date: 13/Jul/20 10:34
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1238:
URL: https://github.com/apache/hive/pull/1238#discussion_r453556055
##########
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
##########
@@ -575,31 +626,8 @@ private OrcFileMetadata getFileFooterFromCacheOrDisk()
throws IOException {
tailBuffers = metadataCache.getFileMetadata(fileKey);
if (tailBuffers != null) {
try {
- MemoryBuffer tailBuffer = tailBuffers.getSingleBuffer();
- ByteBuffer bb = null;
- if (tailBuffer != null) {
- bb = tailBuffer.getByteBufferDup();
- // TODO: remove the copy after ORC-158 and ORC-197
- // if (bb.isDirect()) {
- ByteBuffer dupBb = tailBuffer.getByteBufferDup(); // Don't mess
with the cached object.
- bb = ByteBuffer.allocate(dupBb.remaining());
- bb.put(dupBb);
- bb.flip();
- // }
- } else {
- // TODO: add the ability to extractFileTail to read from multiple
buffers?
- MemoryBuffer[] tailBufferArray = tailBuffers.getMultipleBuffers();
- int totalSize = 0;
- for (MemoryBuffer buf : tailBufferArray) {
- totalSize += buf.getByteBufferRaw().remaining();
- }
- bb = ByteBuffer.allocate(totalSize);
- for (MemoryBuffer buf : tailBufferArray) {
- bb.put(buf.getByteBufferDup());
- }
- bb.flip();
- }
- OrcTail orcTail = ReaderImpl.extractFileTail(bb);
+ OrcTail orcTail = getOrcTailFromLlapBuffers(tailBuffers);
Review comment:
Oh, I see now... it is decRefBuffer...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 457970)
Time Spent: 40m (was: 0.5h)
> LLAP - add API to look up ORC metadata for certain Path
> -------------------------------------------------------
>
> Key: HIVE-23824
> URL: https://issues.apache.org/jira/browse/HIVE-23824
> Project: Hive
> Issue Type: Improvement
> Reporter: Ádám Szita
> Assignee: Ádám Szita
> Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> LLAP IO supports caching but currently this is only done via LlapRecordReader
> / using splits, aka good old mapreduce way.
> At certain times it would worth to leverage the caching of files on certain
> paths, that are not necessarily associated with a record reader directly. An
> example of this could be the caching of ACID delete delta files, as they are
> currently being read without caching.
> With this patch we'd extend the LLAP API and offer another entry point for
> retrieving metadata of ORC files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
