[ https://issues.apache.org/jira/browse/DRILL-5660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102945#comment-16102945 ]
ASF GitHub Bot commented on DRILL-5660: --------------------------------------- Github user arina-ielchiieva commented on a diff in the pull request: https://github.com/apache/drill/pull/877#discussion_r129783208 --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/MetadataVersion.java --- @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * <p/> + * http://www.apache.org/licenses/LICENSE-2.0 + * <p/> + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.drill.exec.store.parquet; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ComparisonChain; +import com.google.common.collect.Lists; +import org.apache.drill.common.exceptions.DrillRuntimeException; + +import java.util.List; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + + +class MetadataVersion implements Comparable<MetadataVersion> { + /** + * String version starts from 'v' letter<p> + * First group is major metadata version (any number of digits, except a single zero digit)<p> + * Next character is optional '.' (if minor version is specified)<p> + * Next group is optional, minor metadata version (any number of digits, except a single zero digit)<p> + * Examples of correct metadata versions: v1, v10, v4.13 + */ + private static final String FORMAT = "v((?!0)\\d+)\\.?((?!0)\\d+)?"; + private static final Pattern PATTERN = Pattern.compile(FORMAT); + + private final int major; + private final int minor; + + public MetadataVersion(int major, int minor) { + this.major = major; + this.minor = minor; + } + + public MetadataVersion(String metadataVersion) { + Matcher matcher = PATTERN.matcher(metadataVersion); + if (!matcher.matches()) { + DrillRuntimeException.format("Could not parse metadata version '%s' using format '%s'", metadataVersion, FORMAT); + } + this.major = Integer.parseInt(matcher.group(1)); + this.minor = matcher.group(2) != null ? Integer.parseInt(matcher.group(2)) : 0; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (!(o instanceof MetadataVersion)) { + return false; + } + MetadataVersion that = (MetadataVersion) o; + return this.major == that.major + && this.minor == that.minor; + } + + @Override + public int hashCode() { + int result = major; + result = 31 * result + minor; + return result; + } + + @Override --- End diff -- `StringBuilder` will look much cleaner here: ``` StringBuilder builder = new StringBuilder(); builder.append("v").append(major); if (minor != 0) { builder.append(".").append(minor); } return builder.toString(); ``` > Drill 1.10 queries fail due to Parquet Metadata "corruption" from DRILL-3867 > ---------------------------------------------------------------------------- > > Key: DRILL-5660 > URL: https://issues.apache.org/jira/browse/DRILL-5660 > Project: Apache Drill > Issue Type: Bug > Affects Versions: 1.11.0 > Reporter: Paul Rogers > Assignee: Vitalii Diravka > Labels: doc-impacting > > Drill recently accepted a PR for the following bug: > DRILL-3867: Store relative paths in metadata file > This PR turned out to have a flaw which affects version compatibility. > The DRILL-3867 PR changed the format of Parquet metadata files to store > relative paths. All Drill servers after that PR create files with relative > paths. But, the version number of the file is unchanged, so that older > Drillbits don't know that they can't understand the file. > Instead, if an older Drill tries to read the file, queries fail left and > right. Drill will resolve the paths, but does so relative to the user's HDFS > home directory, which is wrong. > What should have happened is that we should have bumped the parquet metadata > file version number so that older Drillbits can’t read the file. This ticket > requests that we do that. > Now, one could argue that the lack of version number change is fine. Once a > user upgrades Drill, they won't use an old Drillbit. But, things are not that > simple: > * A developer tests a branch based on a pre-DRILL-3867 build on a cluster in > which metadata files have been created by a post-DRILL-3867 build. (This has > already occurred multiple times in our shop.) > * A user tries to upgrade to Drill 1.11, finds an issue, and needs to roll > back to Drill 1.10. Doing so will cause queries to fail due to > seemingly-corrupt metadata files. > * A user tries to do a rolling upgrade: running 1.11 on some servers, 1.10 on > others. Once a 1.11 server is installed, the metadata is updated ("corrupted" > from the perspective of 1.10) and queries fail. > Standard practice in this scenario is to: > * Bump the file version number when the file format changes, and > * Software refuses to read files with a version newer than the software was > designed for. > Of course, it is highly desirable that newer servers read old files, but that > is not the issue here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)