[
https://issues.apache.org/jira/browse/BIGTOP-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Weiser updated BIGTOP-1634:
-----------------------------------
Attachment: 0001-BIGTOP-1634-puppet-Replace-extlookup-with-hiera-use-.patch
{quote}{noformat}Error: suffix(): expected first argument to be an Array, got
"/data/1" at
/bigtop-home/bigtop-deploy/puppet/modules/hadoop/manifests/init.pp:149 on node
bigtop1.docker{noformat}{quote}
What I really meant to say was:
{noformat}$journalnode_edits_dir =
"${hadoop::hadoop_storage_dirs[0]}/journalnode",{noformat}
meaning "put journalnode edits dir in first hadoop storage dir by default". No
idea if that's a good default because we're using a different directory layout
in our setup and everything is overridden directly from hiera. No idea why that
error didn't crop up in my tests with the default setup either. Sorry again.
Now it did. New patch attached.
How are you testing the journalnode stuff? I haven't put any of the necessary
glue code into cluster.pp because we're not using cluster.pp and the
journalnode stuff is bleeding edge on our end as well. I guess it's not much
use to pop a component "journalnode" in there because the whole concept of
master and standby namenode doesn't apply in that case, does it?
{quote}However, if I do not specify components in config, hence deploying
all{quote}
That doesn't work for me either, but for a different reason: The package names
in hcatalog are wrong, they're missing a leading hive-, I think. Tachyon also
fails later on missing its init script. And finally solr doesn't start for
whatever reason.
{noformat}Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o
DPkg::Options::=--force-confold install hcatalog-server' returned 100: Reading
package lists...
Error: /Stage[main]/Tachyon::Master/Service[tachyon-master]: Could not
evaluate: Could not find init script for 'tachyon-master'
{noformat}
I have reproduced those with the stock puppet recipes. So as far as I can tell
they're not regressions. Unfortunately we're using neither component so I've
got no expertise to track down what's going on with them.
With all but those three components enabled, puppet agent -t runs all the way
through for me though.
{quote}{noformat}Error: Could not find resource 'Service[tachyon-master]' for
relationship on 'Service[tachyon-worker]' on node
bigtop1.docker{noformat}{quote}
*sigh* It must be the third time, I've fixed that stupid scoping bug in the
tachyon module. It seems to go missing all the time somehow. It's fixed in the
attached patch.
{quote}Also probably useful to mention if there is any version requirement on
stdlib{quote}
stdlib already was required. It came in with commit
82922353d2ee80bd08a4e9edad552b44fe0e38be by [~jayunit100] (BIGTOP-1553). It
introduced any2array to split the components list. That's still used. The only
function I use is suffix to replace append_each from bigtop_util. Both
any2array and suffix seem to have been introduced with stdlib 4.0.0. Long story
short: The modules had and have a dependency on stdlib >= 4.0.0. Updated the
README in the attached patch accordingly.
Another thing to look out for: I'm testing all this stuff in a puppet
master/agent setup, not with puppet apply. I only noticed that discrepancy when
updating the README - which obviously I did last. ;-/ Also, I have changed file
imports to puppet:/// URIs which should work with apply but I haven't tested
it. Is it working for you so far? (It's only used for the ssh keys, I think.)
> Puppet class parameter and hiera conversion
> -------------------------------------------
>
> Key: BIGTOP-1634
> URL: https://issues.apache.org/jira/browse/BIGTOP-1634
> Project: Bigtop
> Issue Type: Improvement
> Components: deployment
> Affects Versions: backlog
> Reporter: Michael Weiser
> Labels: patch
> Attachments:
> 0001-BIGTOP-1634-puppet-Replace-extlookup-with-hiera-use-.patch,
> 0001-BIGTOP-1634-puppet-Replace-extlookup-with-hiera-use-.patch,
> 0002-BIGTOP-1634-Add-enhanced-configuration.patch
>
>
> As discussed on the DEV list:
> Update the puppet code to use self-contained, parametrised classes and proper
> scoping. Replace all extlookup calls bei either explicit or automatic hiera
> parameter lookups. Implement HA/non-HA alternative via hiera lookup
> hierarchy. Replace append_each from bigtop_util by suffix from stdlib. Do
> file imports via puppet:/// scheme. Remove bigtop_util because remaining
> function get_settings is not needed any more.
> Additionally: Add additional configuration options for zookeeper and yarn as
> well as a new class for journalnode configuration.
> I've separated it into two patches of ease of review:
> 0001: actual hiera/class conversion
> 0002: functional enhancements including journalnode configuration on top of
> that. Mainly meant as an example for ease of further expansion and
> containment of changes to single modules.
> This JIRA is meant for the stuff contained in 0001, the actual hiera
> conversion. I can resubmit 0002 as a separate JIRA if desired. Also it should
> be possible to backport 0002 to the current puppet code base without much
> fuss.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
