[
https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101370#comment-16101370
]
ASF GitHub Bot commented on DRILL-5660:
---------------------------------------
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/877#discussion_r129104084
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java
---
@@ -132,23 +135,57 @@ public static ParquetTableMetadata_v3
getParquetTableMetadata(FileSystem fs,
}
/**
- * Get the parquet metadata for a directory by reading the metadata file
+ * Get the parquet metadata for the table by reading the metadata file
*
- * @param fs
+ * @param fs current file system
* @param path The path to the metadata file, located in the directory
that contains the parquet files
- * @return
- * @throws IOException
+ * @param metaContext metadata context
+ * @param formatConfig parquet format plugin configs
+ * @return parquet table metadata
+ * @throws IOException if metadata file can't be read or updated
*/
- public static ParquetTableMetadataBase readBlockMeta(FileSystem fs,
String path, MetadataContext metaContext, ParquetFormatConfig formatConfig)
throws IOException {
- Metadata metadata = new Metadata(fs, formatConfig);
- metadata.readBlockMeta(path, false, metaContext);
- return metadata.parquetTableMetadata;
+ public static @Nullable ParquetTableMetadataBase
readBlockMeta(FileSystem fs, Path path, MetadataContext metaContext,
+ ParquetFormatConfig formatConfig) {
+ if (metaContext.isMetaCacheFileCorrect) {
+ Metadata metadata = new Metadata(fs, formatConfig);
+ try {
+ metadata.readBlockMeta(path, false, metaContext);
+ return metadata.parquetTableMetadata;
+ } catch (IOException e) {
+ logger.error(e.toString());
+ metaContext.isMetaCacheFileCorrect = false;
+ }
+ }
+ logger.warn("Ignoring unsupported or corrupted metadata file version.
Query performance may be slow. Make sure " +
+ "the cache file is up-to-date by running the REFRESH TABLE
METADATA command");
+ return null;
}
- public static ParquetTableMetadataDirs readMetadataDirs(FileSystem fs,
String path, MetadataContext metaContext, ParquetFormatConfig formatConfig)
throws IOException {
- Metadata metadata = new Metadata(fs, formatConfig);
- metadata.readBlockMeta(path, true, metaContext);
- return metadata.parquetTableMetadataDirs;
+ /**
+ * Get the parquet metadata for all subdirectories by reading the
metadata file
+ *
+ * @param fs current file system
+ * @param path The path to the metadata file, located in the directory
that contains the parquet files
+ * @param metaContext metadata context
+ * @param formatConfig parquet format plugin configs
+ * @return parquet metadata for a directory
+ * @throws IOException if metadata file can't be read or updated
+ */
+ public static @Nullable ParquetTableMetadataDirs
readMetadataDirs(FileSystem fs, Path path,
+ MetadataContext metaContext, ParquetFormatConfig formatConfig) {
+ if (metaContext.isMetaDirsCacheFileCorrect) {
+ Metadata metadata = new Metadata(fs, formatConfig);
+ try {
+ metadata.readBlockMeta(path, true, metaContext);
+ return metadata.parquetTableMetadataDirs;
+ } catch (IOException e) {
+ logger.error(e.toString());
--- End diff --
Done
> Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867
> ----------------------------------------------------------------------------
>
> Key: DRILL-5660
> URL: https://issues.apache.org/jira/browse/DRILL-5660
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Vitalii Diravka
> Labels: doc-impacting
>
> Drill recently accepted a PR for the following bug:
> DRILL-3867: Store relative paths in metadata file
> This PR turned out to have a flaw which affects version compatibility.
> The DRILL-3867 PR changed the format of Parquet metadata files to store
> relative paths. All Drill servers after that PR create files with relative
> paths. But, the version number of the file is unchanged, so that older
> Drillbits don't know that they can't understand the file.
> Instead, if an older Drill tries to read the file, queries fail left and
> right. Drill will resolve the paths, but does so relative to the user's HDFS
> home directory, which is wrong.
> What should have happened is that we should have bumped the parquet metadata
> file version number so that older Drillbits can’t read the file. This ticket
> requests that we do that.
> Now, one could argue that the lack of version number change is fine. Once a
> user upgrades Drill, they won't use an old Drillbit. But, things are not that
> simple:
> * A developer tests a branch based on a pre-DRILL-3867 build on a cluster in
> which metadata files have been created by a post-DRILL-3867 build. (This has
> already occurred multiple times in our shop.)
> * A user tries to upgrade to Drill 1.11, finds an issue, and needs to roll
> back to Drill 1.10. Doing so will cause queries to fail due to
> seemingly-corrupt metadata files.
> * A user tries to do a rolling upgrade: running 1.11 on some servers, 1.10 on
> others. Once a 1.11 server is installed, the metadata is updated ("corrupted"
> from the perspective of 1.10) and queries fail.
> Standard practice in this scenario is to:
> * Bump the file version number when the file format changes, and
> * Software refuses to read files with a version newer than the software was
> designed for.
> Of course, it is highly desirable that newer servers read old files, but that
> is not the issue here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
