[
https://issues.apache.org/jira/browse/HIVE-23890?focusedWorklogId=473610&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-473610
]
ASF GitHub Bot logged work on HIVE-23890:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 23/Aug/20 06:27
Start Date: 23/Aug/20 06:27
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #1330:
URL: https://github.com/apache/hive/pull/1330#discussion_r475177713
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
##########
@@ -5685,6 +5706,67 @@ private void alter_table_core(String catName, String
dbname, String name, Table
}
}
+ @Override
+ public GetFileListResponse get_file_list(GetFileListRequest req) throws
MetaException {
+ String catName = req.isSetCatName() ? req.getCatName() :
getDefaultCatalog(conf);
+ String dbName = req.getDbName();
+ String tblName = req.getTableName();
+ List<String> partitions = req.getPartVals();
+ // Will be used later, when cache is introduced
+ String validWriteIdList = req.getValidWriteIdList();
+
+ startFunction("get_file_list", ": " + TableName.getQualified(catName,
dbName, tblName)
+ + ", partitions: " + partitions.toString());
+
+
+ GetFileListResponse response = new GetFileListResponse();
+
+ boolean success = false;
+ Exception ex = null;
+ try {
+ Partition p = getMS().getPartition(catName, dbName, tblName,
partitions);
+ Path path = new Path(p.getSd().getLocation());
+
+ FileSystem fs = path.getFileSystem(conf);
+ RemoteIterator<LocatedFileStatus> itr = fs.listFiles(path, true);
+ while (itr.hasNext()) {
+ FileStatus fStatus = itr.next();
+ Reader reader = OrcFile.createReader(fStatus.getPath(),
OrcFile.readerOptions(fs.getConf()));
+ boolean isRawFormat =
!CollectionUtils.isEqualCollection(reader.getSchema().getFieldNames(),
ALL_ACID_ROW_NAMES);
+ int fileFormat = isRawFormat ? 0 : 2;
Review comment:
Please use enum on the Java side, but on thrift side stick to
strings/ints whatever. Again, when upgrading HMS API enum in the thrift can
cause issues
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 473610)
Time Spent: 2h 40m (was: 2.5h)
> Create HMS endpoint for querying file lists using FlatBuffers as serialization
> ------------------------------------------------------------------------------
>
> Key: HIVE-23890
> URL: https://issues.apache.org/jira/browse/HIVE-23890
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Barnabas Maidics
> Assignee: Barnabas Maidics
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> New thrift objects would be:
> {code:java}
> struct GetFileListRequest {
> 1: optional string catName,
> 2: required string dbName,
> 3: required string tableName,
> 4: required list<string> partVals,
> 6: optional string validWriteIdList
> }
> struct GetFileListResponse {
> 1: required binary fileListData
> }
> {code}
> Where GetFileListResponse contains a binary field, which would be a
> FlatBuffer object
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
