[ https://issues.apache.org/jira/browse/DRILL-7125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807281#comment-16807281 ]
Kunal Khatua commented on DRILL-7125: ------------------------------------- [~shamirwasia] does this require any Documentation? (cc: [~bbevens]) > REFRESH TABLE METADATA fails after upgrade from Drill 1.13.0 to Drill 1.15.0 > ---------------------------------------------------------------------------- > > Key: DRILL-7125 > URL: https://issues.apache.org/jira/browse/DRILL-7125 > Project: Apache Drill > Issue Type: Bug > Components: Metadata > Affects Versions: 1.14.0, 1.15.0 > Reporter: Sorabh Hamirwasia > Assignee: Sorabh Hamirwasia > Priority: Major > Labels: ready-to-commit > Fix For: 1.16.0 > > > REFRESH TABLE METADATA command worked successfully on Drill 1.13.0, however > after upgrade Drill 1.15.0 there are errors sometime. > {code:java} > In sqlline logging in as regular user "alice" or Drill process user "admin" > gives the same error (permission denied) > If this helps, here's also what I am seeing on sqlline > Error message contains random but valid user's names other than the user > (Alice) that logged in to refresh the metadata. Looks like during the refresh > metadata drillbits seems to incorrectly try the metadata generation as some > random user which obviously does not have write access > 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State change requested RUNNING --> > FINISHED > 2019-03-12 15:27:20,564 [2377cdd9-dd6e-d213-de1a-70b50d3641d7:frag:0:0] INFO > o.a.d.e.w.f.FragmentStatusReporter - > 2377cdd9-dd6e-d213-de1a-70b50d3641d7:0:0: State to report: FINISHED > 2019-03-12 15:27:23,032 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] INFO > o.a.drill.exec.work.foreman.Foreman - Query text for query with id > 2377cdb3-86cc-438d-8ada-787d2a84df9a issued by alice: REFRESH TABLE METADATA > dfs.root.`/user/alice/logs/hive/warehouse/detail` > 2019-03-12 15:27:23,350 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR > o.a.d.e.s.parquet.metadata.Metadata - Failed to read > 'file://user/alice/logs/hive/warehouse/detail/.drill.parquet_metadata_directories' > metadata file > java.io.IOException: 2879.5854742.1036302960 > /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata > (Permission denied) > at com.mapr.fs.Inode.throwIfFailed(Inode.java:390) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.Inode.flushPages(Inode.java:505) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.Inode.releaseDirty(Inode.java:583) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726) > ~[jackson-core-2.9.5.jar:2.9.5] > at > org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.readBlockMeta(Metadata.java:605) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.readMetadataDirs(Metadata.java:179) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:251) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:615) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.create(WorkspaceSchemaFactory.java:407) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:96) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.get(ExpandingConcurrentMap.java:90) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory$WorkspaceSchema.getTable(WorkspaceSchemaFactory.java:554) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.calcite.jdbc.SimpleCalciteSchema.getImplicitTable(SimpleCalciteSchema.java:83) > [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2] > at > org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:288) > [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2] > at > org.apache.calcite.jdbc.CalciteSchema$SchemaPlusImpl.getTable(CalciteSchema.java:648) > [calcite-core-1.17.0-drill-r2.jar:1.17.0-drill-r2] > at > org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:77) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_152] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_152] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152] > 2019-03-12 15:27:23,446 [2377cdb3-86cc-438d-8ada-787d2a84df9a:foreman] ERROR > o.a.d.e.p.s.h.RefreshMetadataHandler - Failed to update metadata for table > '/user/alice/logs/hive/warehouse/detail' > java.io.IOException: 2879.5854742.1036302960 > /user/alice/logs/hive/warehouse/detail/file1/.drill.parquet_metadata > (Permission denied) > at com.mapr.fs.Inode.throwIfFailed(Inode.java:390) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.Inode.flushPages(Inode.java:505) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.Inode.releaseDirty(Inode.java:583) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.mapr.fs.MapRFsOutStream.dropCurrentPage(MapRFsOutStream.java:73) > ~[maprfs-6.1.0-mapr.jar:na] > at com.mapr.fs.MapRFsOutStream.write(MapRFsOutStream.java:85) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.mapr.fs.MapRFsDataOutputStream.write(MapRFsDataOutputStream.java:39) > ~[maprfs-6.1.0-mapr.jar:na] > at > com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2085) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1097) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2645) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:381) > ~[jackson-core-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1726) > ~[jackson-core-2.9.5.jar:2.9.5] > at > org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:448) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata_V3$ColumnMetadata_v3$Serializer.serialize(Metadata_V3.java:417) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:604) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:1120) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:950) > ~[jackson-databind-2.9.5.jar:2.9.5] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.writeFile(Metadata.java:548) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:259) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.createMetaFilesRecursively(Metadata.java:225) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.store.parquet.metadata.Metadata.createMeta(Metadata.java:111) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.handlers.RefreshMetadataHandler.getPlan(RefreshMetadataHandler.java:124) > ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:184) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:110) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:584) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:272) > [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [na:1.8.0_152] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [na:1.8.0_152] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152] > 2019-03-12 15:27:23,459 [2377cdb3-86cc-438d-8ada-787d2a84df9a:frag:0:0] INFO > o.a.d.e.w.fragment.FragmentExecutor - > 2377cdb3-86cc-438d-8ada-787d2a84df9a:0:0: State change requested > AWAITING_ALLOCATION --> RUNNING > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)