[
https://issues.apache.org/jira/browse/DRILL-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pritesh Maker updated DRILL-6263:
---------------------------------
Fix Version/s: (was: 1.14.0)
> Improvements to DoY initial experience
> --------------------------------------
>
> Key: DRILL-6263
> URL: https://issues.apache.org/jira/browse/DRILL-6263
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.13.0
> Reporter: Paul Rogers
> Priority: Minor
>
> As part of the Drill 1.13 release process, I tested out DoY after a year of
> not having used it. That time gap pointed out some improvements for
> first-time users.
> * Copy the
> [USAGE.md|https://github.com/apache/drill/blob/master/drill-yarn/USAGE.md]
> file into the Drill home directory with the name "DRILL_YARN_USAGE.md.".
> * Change the {{drill-on-yarn-example.conf}} file to be a valid file for the
> default Drill and YARN configurations.
> {noformat}
> heap: "2G"
> max-direct-memory: "2G"
> memory-mb: 5125
> {noformat}
> * Change the {{drill-on-yarn-example.conf}} to disable SSL by default. Just
> comment out the following line:
> {noformat}
> #ssl-enabled: true
> {noformat}
> * Change the {{drill-on-yarn-example.conf}} to disable authorization by
> default. That is, comment out the following line:
> {noformat}
> #auth-type: "drill"
> {noformat}
> * Change the {{drill-on-yarn-example.conf}} to use no AM node labels by
> default. That is, comment out the following line:
> {noformat}
> #node-label-expr: "drill-am"
> {noformat}
> Failure to comment out this line results in the following error:
> {noformat}
> Failed to start Drill application master
> Caused by: Submit application failed
> Caused by: Invalid resource request, node label not enabled but request
> contains label expression
> {noformat}
> Also, add this to the Troubleshooting section in {{USAGE.md}}.
> * Change {{DrillOnYarnConfig.findSuffix}}, to allow the {{.tar}} suffix. This
> is what one ends up with it ht Mac does its automatic extract. A tar file is
> larger than the compressed version, but no reason it should not be allowed
> (assuming YARN supports it.)
> * Otherwise, change {{DrillOnYarnConfig.getRemoteDrillHome()}}, where we emit
> the error "does not name a valid archive" to differentiate between no
> sufficient and an unsupported suffix. (I got the following error and had to
> look at the source to figure out what I'd done wrong):
> {noformat}
> drill.yarn.drill-install.client-path does not name a valid archive:
> /Users/paulrogers/bin/apache-drill-1.13.0.tar
> {noformat}
> * Change the newly-added error reporting code in {{DrillOnYarn.displayError}}
> to omit displaying the exception cause it if just repeats the main error
> message. Here is the full error message from above, the second line is
> redundant:
> {noformat}
> drill.yarn.drill-install.client-path does not name a valid archive:
> /Users/paulrogers/bin/apache-drill-1.13.0.tar
> Caused by: drill.yarn.drill-install.client-path does not name a valid
> archive: /Users/paulrogers/bin/apache-drill-1.13.0.tar
> {noformat}
> * Add to
> [USAGE.md|https://github.com/apache/drill/blob/master/drill-yarn/USAGE.md]
> pointers for how to set up a basic HDFS, ZK and YARN configuration. Mostly
> just state what is to be done and point to the [relevant Hadoop
> docs|https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html],
> "Pseudo-Distributed Operation". In particular, we want to create an actual
> HDFS file system, not use the default of local file system.
> * Add to {{USAGE.md}} a description of the supported YARN (actually Hadoop)
> versions. Feature was developed with 2.7.1. Currently verifying with 2.9.0.
> Probably needs to be rechecked on the 3.x series.
> * Add to {{USAGE.md}} the fact that Drill is built with, and includes the
> jars for, Hadoop 2.7.1. It is not clear what version compatibility Hadoop
> has; are these jars compatible with the latest 2.x series Hadoop? With Hadoop
> 3.x?
> * Until DRILL-6268 is fixed, explain that the HDFS configuration *must* use
> port 8020. Also, add this to the Troubleshooting section in {{USAGE.md}}.
> * Add to {{USAGE.md}}, Troubleshooting: if configuration issues cause Drill
> to fail to start, then Drill-on-YARN will blacklist each node after several
> tries. Unfortunately, the YARN UI appears to not provide access to the logs
> for failed application containers. So, to track down the failure, look for
> the container logs in YARN. In the default single-node install, they are in
> {{$HADOOP_HOME/logs/userlogs/application_xxx/container_xx_00000y}} where y=1
> is the AM, y>1 are the Drillbit containers.
> * Change {{USAGE.md}} to change the following line:
> {noformat}
> cp $DRILL_HOME/conf/drill-override-example.conf
> $DRILL_SITE/drill-override.conf
> {noformat}
> To the following:
> {noformat}
> cp $DRILL_HOME/conf/drill-override.conf $DRILL_SITE
> {noformat}
> Without this change, Drill will fail to start and you'll see the following in
> the YARN container log directory, {{drillbit.log}} file:
> {noformat}
> 2018-03-17 16:11:25,293 [main] ERROR o.a.d.e.r.u.s.PamUserAuthenticator -
> Problem in finding the native library of JPAM (Pluggable Authenticator Module
> API). Make sure to set Drillbit JVM option 'java.library.path' to point to
> the directory where the native JPAM exists.
> java.lang.UnsatisfiedLinkError: no jpam in java.library.path
> {noformat}
> None of these are show stoppers, each is instead just a bit of sand in the
> gears that makes progress a bit slower than it need be.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
