[
https://issues.apache.org/jira/browse/DRILL-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965974#comment-14965974
]
ASF GitHub Bot commented on DRILL-3893:
---------------------------------------
GitHub user vkorukanti opened a pull request:
https://github.com/apache/drill/pull/212
DRILL-3893: Change Hive metadata cache invalidation policy to "1 min after
last write"
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vkorukanti/drill DRILL-3893
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/212.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #212
----
commit 21c4325abcb25ce22846217e61eee23816f73571
Author: vkorukanti <[email protected]>
Date: 2015-10-19T18:35:09Z
DRILL-3938: Support reading from Hive tables that have schema altered after
the creation
Also:
+ Remove "redoRecord" logic which is not needed after "automatic
reallocation" (DRILL-1960) changes.
+ Remove HiveTestRecordReader. This is incomplete in implementation and not
used anywhere. It is currently just
a burden to maintain with changes in its superclass HiveRecordReader
commit 7994a2cd6e343870f7ded4e00c7dd493e60f1ef8
Author: vkorukanti <[email protected]>
Date: 2015-10-20T23:21:09Z
DRILL-3893: Change Hive metadata cache invalidation policy to "1 min after
last write".
----
> Issue with Drill after Hive Alters the Table
> ---------------------------------------------
>
> Key: DRILL-3893
> URL: https://issues.apache.org/jira/browse/DRILL-3893
> Project: Apache Drill
> Issue Type: Bug
> Components: Functions - Hive, Storage - Hive
> Affects Versions: 1.0.0, 1.1.0
> Environment: DEV
> Reporter: arnab chatterjee
>
> I reproduced this again on another partitioned table with existing data.
> Providing some more details. I have enabled the version mode for errors.
> Drill is unable to fetch the new column name that was introduced.This most
> likely to me seems to me that it’s still picking up the stale metadata of
> hive.
> if (!tableColumns.contains(columnName)) {
> if (partitionNames.contains(columnName)) {
> selectedPartitionNames.add(columnName);
> } else {
> throw new ExecutionSetupException(String.format("Column %s does
> not exist", columnName));
> }
> }
> select testdata from testtable;
> Error: SYSTEM ERROR: ExecutionSetupException: Column testdata does not exist
> Fragment 0:0
> [Error Id: be5cccba-97f6-4cc4-94e8-c11a4c53c8f4 on x.x.com:9999]
> (org.apache.drill.common.exceptions.ExecutionSetupException) Failure while
> initializing HiveRecordReader: Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():241
> org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745
> Caused By (org.apache.drill.common.exceptions.ExecutionSetupException)
> Column testdata does not exist
> org.apache.drill.exec.store.hive.HiveRecordReader.init():206
> org.apache.drill.exec.store.hive.HiveRecordReader.<init>():138
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():58
> org.apache.drill.exec.store.hive.HiveScanBatchCreator.getBatch():34
> org.apache.drill.exec.physical.impl.ImplCreator.getRecordBatch():150
> org.apache.drill.exec.physical.impl.ImplCreator.getChildren():173
> org.apache.drill.exec.physical.impl.ImplCreator.getRootExec():106
> org.apache.drill.exec.physical.impl.ImplCreator.getExec():81
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():235
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> #########################################################
> Please note that this is a partitioned table with existing data.
> Does Drill Cache the Meta somewhere and hence it’s not getting reflected
> immediately ?
> DRILL CLI
> > select x from xx;
> Error: SYSTEM ERROR: ExecutionSetupException: Column x does not exist
> Fragment 0:0
> [Error Id: 62086e22-1341-459e-87ce-430a24cc5119 on x.x.com:999]
> (state=,code=0)
> HIVE CLI
> hive> describe formatted x;
> OK
> # col_name data_type comment
> <columns>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
