Sorabh Hamirwasia created DRILL-7125:
----------------------------------------
Summary: REFRESH TABLE METADATA fails after upgrade from Drill
1.13.0 to Drill 1.15.0
Key: DRILL-7125
URL: https://issues.apache.org/jira/browse/DRILL-7125
Project: Apache Drill
Issue Type: Bug
Components: Metadata
Affects Versions: 1.15.0, 1.14.0
Reporter: Sorabh Hamirwasia
Assignee: Sorabh Hamirwasia
Fix For: 1.16.0
REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however
after upgrade Drill 1.15.0 there are errors sometime.
{code:java}
In sqlline logging in as regular user "alice" or Drill process user "admin"
gives the same error (permission denied)
If this helps, here's also what I am seeing on sqlline
Error message contains random but valid user's names other than the user
(Alice) that logged in to refresh the metadata. Looks like during the refresh
metadata drillbits seems to incorrectly try the metadata generation as some
random user which obviously does not have write access
2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0:
State change requested RUNNING --> FINISHED
2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO
o.a.d.e.w.f.FragmentStatusReporter - 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0:
State to report: FINISHED
2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO
o.a.drill.exec.work.foreman.Foreman - Query text for query with id
2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA
dfs.root.`/user/alice/logs/hive/warehouse/detail`
2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR
o.a.d.e.s.parquet.metadata.Metadata - Failed to read
'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories'
metadata file
java.io.IOException: 2879.5854742.1036302960
/user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata
(Permission denied)
at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.flushPages(Inode.java:505)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
~[maprfs-6.1.0-mapr.jar:na]
at
com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
~[maprfs-6.1.0-mapr.jar:na]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726)
~[jackson-core-2.9.5.jar:2.9.5]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950)
~[jackson-databind-2.9.5.jar:2.9.5]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:605)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.readMetadataDirs(Metadata.java:179)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:251)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:615)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:407)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:554)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83)
[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at
org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:288)
[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at
org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTable(CalciteSchema.java:648)
[calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2]
at
org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:77)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_152]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
2019-03-12 15:27:23,446 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR
o.a.d.e.p.s.h.RefreshMetadataHandler - Failed to update metadata for table
'/user/alice/logs/hive/warehouse/detail'
java.io.IOException: 2879.5854742.1036302960
/user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata
(Permission denied)
at com.mapr.fs.Inode.throwIfFailed(Inode.java:390)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.flushPages(Inode.java:505)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.Inode.releaseDirty(Inode.java:583)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73)
~[maprfs-6.1.0-mapr.jar:na]
at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85)
~[maprfs-6.1.0-mapr.jar:na]
at
com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39)
~[maprfs-6.1.0-mapr.jar:na]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381)
~[jackson-core-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726)
~[jackson-core-2.9.5.jar:2.9.5]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120)
~[jackson-databind-2.9.5.jar:2.9.5]
at
com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950)
~[jackson-databind-2.9.5.jar:2.9.5]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.store.parquet.metadata.Metadata.createMeta(Metadata.java:111)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:124)
~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272)
[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_152]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
2019-03-12 15:27:23,459 [2377cdb3-86cc-438d-8ada-787d2a84df9a:frag:0:0] INFO
o.a.d.e.w.fragment.FragmentExecutor - 2377cdb3-86cc-438d-8ada-787d2a84df9a:0:0:
State change requested AWAITING_ALLOCATION --> RUNNING
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
