[
https://issues.apache.org/jira/browse/DRILL-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548315#comment-16548315
]
ASF GitHub Bot commented on DRILL-5365:
---------------------------------------
arina-ielchiieva commented on a change in pull request #1296: DRILL-5365:
Prevent plugin config from changing default fs. Make DrillFileSystem Immutable.
URL: https://github.com/apache/drill/pull/1296#discussion_r203505132
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/DrillFileSystem.java
##########
@@ -65,46 +62,105 @@
import com.google.common.collect.Maps;
/**
- * DrillFileSystem is the wrapper around the actual FileSystem implementation.
+ * DrillFileSystem is the wrapper around the actual FileSystem implementation.
The {@link DrillFileSystem} is
+ * immutable.
*
* If {@link org.apache.drill.exec.ops.OperatorStats} are provided it returns
an instrumented FSDataInputStream to
* measure IO wait time and tracking file open/close operations.
*/
public class DrillFileSystem extends FileSystem implements OpenFileTracker {
static final org.slf4j.Logger logger =
org.slf4j.LoggerFactory.getLogger(DrillFileSystem.class);
private final static boolean TRACKING_ENABLED =
AssertionUtil.isAssertionsEnabled();
+ private final static DrillFileSystemCache CACHE = new DrillFileSystemCache();
+ public static final String FS_DEFAULT_NAME = "fs.default.name";
public static final String UNDERSCORE_PREFIX = "_";
public static final String DOT_PREFIX = ".";
private final ConcurrentMap<DrillFSDataInputStream, DebugStackTrace>
openedFiles = Maps.newConcurrentMap();
+ private final Configuration fsConf;
private final FileSystem underlyingFs;
private final OperatorStats operatorStats;
private final CompressionCodecFactory codecFactory;
+ private boolean initialized = false;
+
public DrillFileSystem(Configuration fsConf) throws IOException {
this(fsConf, null);
}
public DrillFileSystem(Configuration fsConf, OperatorStats operatorStats)
throws IOException {
- this.underlyingFs = FileSystem.get(fsConf);
+ // Configuration objects are mutable, and the underlying FileSystem object
may directly use a passed in Configuration.
+ // In order to avoid scenarios where a Configuration can change after a
DrillFileSystem is created, we make a copy
+ // of the Configuration.
+ this.fsConf = new Configuration(fsConf);
+ normalize(fsConf);
+
+ this.underlyingFs = CACHE.get(fsConf);
this.codecFactory = new CompressionCodecFactory(fsConf);
this.operatorStats = operatorStats;
+ this.initialized = true;
Review comment:
@ilooner if this flag is set in constructor to true do we need it at all?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> FileNotFoundException when reading a parquet file
> -------------------------------------------------
>
> Key: DRILL-5365
> URL: https://issues.apache.org/jira/browse/DRILL-5365
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Hive
> Affects Versions: 1.10.0
> Reporter: Chun Chang
> Assignee: Timothy Farkas
> Priority: Major
> Fix For: 1.14.0
>
>
> The parquet file is generated through the following CTAS.
> To reproduce the issue: 1) two or more nodes cluster; 2) enable
> impersonation; 3) set "fs.default.name": "file:///" in hive storage plugin;
> 4) restart drillbits; 5) as a regular user, on node A, drop the table/file;
> 6) ctas from a large enough hive table as source to recreate the table/file;
> 7) query the table from node A should work; 8) query from node B as same user
> should reproduce the issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
