[
https://issues.apache.org/jira/browse/DRILL-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Rogers updated DRILL-6520:
-------------------------------
Description:
In a Hadoop HDFS environment, the docs. say to set the HDFS configuration in
the storage plugin config.
However, when Drill runs on an existing HDFS cluster, one must often provide
more than the simple HDFS URL. Particularly in a secure cluster, other
configuration settings are also needed. At present, these must be copied out of
the HDFS config files into the Drill storage plugin config, and the two must be
updated in tandem. Clearly less than ideal.
In experimenting with better integration, I added my HDFS {{core-site.xml}}
file to the class path, hoping Drill would pick up the settings from the Hadoop
config files that I had placed on the class path. Instead, Drill failed to
start.
{noformat}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:218)
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
{noformat}
Modify the {{core-site.xml}} file to comment out the {{fs.defaultFS}} property.
{noformat}
<configuration>
<!--
<property>
<name>fs.defaultFS</name>
<value>hdfs://myNameNode:8020</value>
</property>
-->
</configuration>
{noformat}
Now Drill starts fine.
The "myNameNode" is not the real name. But, the actual value here is identical
to the value set in the dfs "connection" property, which works fine by itself
without {{core-site.xml}}.
Expected that Drill would allow the common Hadoop pattern of setting
{{fs.defaultFS}} in {{core-site.xml}}. At the very least, Drill should not
crash when this (or any other) property is set in {{core-site.xml}}.
Strangely, I have many other properties set in {{core-site.xml}} and they all
work just fine. It is just the most important one that causes Drill to crash.
was:
In a Hadoop HDFS environment, the docs. say to set the HDFS configuration in
the storage plugin config.
However, when Drill runs on an existing HDFS cluster, one must often provide
more than the simple HDFS URL. Particularly in a secure cluster, other
configuration settings are also needed. At present, these must be copied out of
the HDFS config files into the Drill storage plugin config, and the two must be
updated in tandem. Clearly less than ideal.
In experimenting with better integration, I added my HDFS {{core-site.xml}}
file to the class path, hoping Drill would pick up the settings from the Hadoop
config files that I had placed on the class path. Instead, Drill failed to
start.
{noformat}
Exception in thread "main" java.lang.NullPointerException
at
org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:218)
at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
{noformat}
Modify the {{core-site.xml}} file to comment out the {{fs.defaultFS}} property.
{noformat}
<configuration>
<!--
<property>
<name>fs.defaultFS</name>
<value>hdfs://myNameNode:8020</value>
</property>
-->
</configuration>
{noformat}
Now Drill starts fine.
The "myNameNode" is not the real name. But, the actual value here is identical
to the value set in the dfs "connection" property, which works fine by itself
without {{core-site.xml}}.
Expected that Drill would allow the common Hadoop pattern of setting
{{fs.defaultFS}} in {{core-site.xml}}. At the very least, Drill should not
crash when this (or any other) property is set in {{core-site.xml}}.
> Setting fs.defaultFS in core-site.xml causes startup crash
> ----------------------------------------------------------
>
> Key: DRILL-6520
> URL: https://issues.apache.org/jira/browse/DRILL-6520
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.13.0
> Reporter: Paul Rogers
> Priority: Major
>
> In a Hadoop HDFS environment, the docs. say to set the HDFS configuration in
> the storage plugin config.
> However, when Drill runs on an existing HDFS cluster, one must often provide
> more than the simple HDFS URL. Particularly in a secure cluster, other
> configuration settings are also needed. At present, these must be copied out
> of the HDFS config files into the Drill storage plugin config, and the two
> must be updated in tandem. Clearly less than ideal.
> In experimenting with better integration, I added my HDFS {{core-site.xml}}
> file to the class path, hoping Drill would pick up the settings from the
> Hadoop config files that I had placed on the class path. Instead, Drill
> failed to start.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.drill.exec.coord.zk.ZKClusterCoordinator.update(ZKClusterCoordinator.java:218)
> at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:228)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:401)
> at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:372)
> at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:368)
> {noformat}
> Modify the {{core-site.xml}} file to comment out the {{fs.defaultFS}}
> property.
> {noformat}
> <configuration>
> <!--
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://myNameNode:8020</value>
> </property>
> -->
> </configuration>
> {noformat}
> Now Drill starts fine.
> The "myNameNode" is not the real name. But, the actual value here is
> identical to the value set in the dfs "connection" property, which works fine
> by itself without {{core-site.xml}}.
> Expected that Drill would allow the common Hadoop pattern of setting
> {{fs.defaultFS}} in {{core-site.xml}}. At the very least, Drill should not
> crash when this (or any other) property is set in {{core-site.xml}}.
> Strangely, I have many other properties set in {{core-site.xml}} and they all
> work just fine. It is just the most important one that causes Drill to crash.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
