Thanks again Konstantinos for all your help. I made some changes and am now able to pass the 01-giraph-test.py: https://pastebin.ubuntu.com/23526882/
I have some more questions though: 1. I am using two environment variables ($HADOOP_CLASSPATH and $LIBJARS) that I set in my smoke-test. However, I believe that it would be more appropriate if these variables were set in .bashrc as they are necessary for submitting giraph jobs to hadoop. Is this possible? 2. I am quite sure that the first time I deployed the giraph charm that I am building, there was more content in the /usr/share/doc/giraph/ directory. As the installation of giraph is pretty much automated through bigtop (if I am not missing something) my question is where should I look to find out what gets installed and where. Any ideas? 3. Is the documentation on the charm proof tool up-to-date? ( https://jujucharms.com/docs/2.0/authors-charm-writing#run-the-charm-proof-tool ) Thank you, Panagiotis 2016-11-24 9:12 GMT+02:00 Konstantinos Tsakalozos <[email protected]>: > As it turned out this error was due to a version misalignment between the > juju-deployer and amulet. Its been fixed now. Please, let us know if you > face any more obstacles. > > Thanks, > Konstantinos > > On Wed, Nov 23, 2016 at 4:00 PM, Panagiotis Liakos <[email protected]> > wrote: >> >> Thanks a lot for the clarifications Konstantinos! >> >> I opted to use the mahout interface and managed to overcome the previous >> issue. >> >> Then I bumped into this bug: >> https://bugs.launchpad.net/juju-deployer/+bug/1575863 >> >> I followed the suggested workaround (mkdir ~/.juju) and now I get an >> "error getting env api endpoints". Any ideas? >> >> Panagiotis >> >> 2016-11-23 15:54:36 Starting deployment of lxd:admin/default >> 2016-11-23 15:54:36 Error getting env api endpoints, env bootstrapped? >> 2016-11-23 15:54:36 Command (juju api-endpoints -e lxd:admin/default) >> Output: >> >> >> 2016-11-23 15:54:36 Deployment stopped. run time: 0.15 >> E >> ====================================================================== >> ERROR: setUpClass (__main__.TestDeploy) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "./tests/01-giraph-test.py", line 44, in setUpClass >> cls.d.setup(timeout=3600) >> File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 704, in >> setup >> subprocess.check_call(shlex.split(cmd)) >> File "/usr/lib/python3.5/subprocess.py", line 581, in check_call >> raise CalledProcessError(retcode, cmd) >> subprocess.CalledProcessError: Command '['juju-deployer', '-W', '-c', >> '/tmp/amulet-juju-deployer-o1zyf3xd/deployer-schema.json', '-e', >> 'lxd:admin/default', '-t', '3700', 'lxd:admin/default']' returned >> non-zero exit status 1 >> >> ---------------------------------------------------------------------- >> Ran 0 tests in 4.738s >> >> FAILED (errors=1) >> >> >> >> 2016-11-22 16:31 GMT+02:00 Konstantinos Tsakalozos >> <[email protected]>: >> > Cool! The error that you are getting now says that the giraph charm has >> > no >> > giraph relation [0]. >> > >> > Looking at the metadata.yaml, you are using the mahout interface to >> > relate >> > to the hadoop client. So the relation call should look like: >> > cls.d.relate('giraph:mahout', 'client:mahout') . This will work for now >> > since both Mahout and Giraph just want to add jars on the hadoop-client >> > charm. However, on the long run we should refactor charms such as >> > hadoop-client, spark, pig, etc, so that they have an interface exactly >> > for >> > that purpose (adding jars to the file system). For now you have two >> > options >> > either keep using the mahout interface or use the "juju-info" [1] >> > interface >> > that is present in all charms. Note, however, that if you do use >> > "juju-info" >> > your charm would be a subordinate to any other charm even if that does >> > not >> > make sense. >> > >> > [0] >> > >> > http://pythonhosted.org/amulet/amulet.html?highlight=relate#amulet.deployer.Deployment.relate >> > [1] https://jujucharms.com/docs/2.0/authors-implicit-relations >> > >> > On Tue, Nov 22, 2016 at 4:02 PM, Panagiotis Liakos <[email protected]> >> > wrote: >> >> >> >> Thanks Konstantinos, you were right. Now the script progresses a little >> >> further: >> >> >> >> $ python3 ./tests/01-giraph-test.py >> >> E >> >> ====================================================================== >> >> ERROR: setUpClass (__main__.TestDeploy) >> >> ---------------------------------------------------------------------- >> >> Traceback (most recent call last): >> >> File "./tests/01-giraph-test.py", line 42, in setUpClass >> >> cls.d.relate('giraph:giraph', 'client:mahout') >> >> File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 431, >> >> in >> >> relate >> >> raise ValueError('%s does not exist for %s' % (rel, srv)) >> >> ValueError: giraph does not exist for giraph >> >> >> >> ---------------------------------------------------------------------- >> >> Ran 0 tests in 2.285s >> >> >> >> FAILED (errors=1) >> >> >> >> >> >> Is this a namespace issue? Any ideas? >> >> >> >> Panagiotis >> >> >> >> 2016-11-22 14:16 GMT+02:00 Konstantinos Tsakalozos >> >> <[email protected]>: >> >> > Hi Panagiotis, >> >> > >> >> > The 404 error seems to be because the Giraph charm is not yet >> >> > promulgated >> >> > (your charm currently lives in cs:~panagiotisl/giraph). Could you try >> >> > the >> >> > same test only this time with a charm revision you have pushed under >> >> > your >> >> > namespace in line >> >> > >> >> > >> >> > https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/giraph/layer-giraph/tests/01-giraph-test.py#L29 >> >> > ? You can also use a local path pointing to where your charm build >> >> > output >> >> > is. >> >> > >> >> > Glad the multiple SLF4J bindings are not causing any problem. >> >> > >> >> > Thanks, >> >> > Konstantinos >> >> > >> >> > On Tue, Nov 22, 2016 at 1:15 PM, Panagiotis Liakos >> >> > <[email protected]> >> >> > wrote: >> >> >> >> >> >> Dear Merlijn and Konstantinos, >> >> >> >> >> >> Thanks a lot for the very valuable information. I will have another >> >> >> look at the documentation for writing tests and debugging and I'll >> >> >> try >> >> >> to add some logs in my test. >> >> >> >> >> >> When I execute my smoke-test (through ssh to giraph) I get the >> >> >> following (which is exactly what I expect): >> >> >> https://pastebin.ubuntu.com/23516317/ >> >> >> >> >> >> When I execute the 01-giraph-test.py (and after I installed amulet) >> >> >> I >> >> >> get the following: https://pastebin.ubuntu.com/23516415/ >> >> >> >> >> >> The URL mentioned in the HTTPError exception is valid for mahout but >> >> >> not for giraph so I guess I am requesting something that is not >> >> >> there >> >> >> (yet). >> >> >> >> >> >> Thanks again, I'll keep you updated. >> >> >> >> >> >> Panagiotis >> >> >> >> >> >> >> >> >> 2016-11-22 12:08 GMT+02:00 Konstantinos Tsakalozos >> >> >> <[email protected]>: >> >> >> > Hi Panagiotis, >> >> >> > >> >> >> > Merlijn is right, the output of the smoke-test script is available >> >> >> > in >> >> >> > the >> >> >> > juju debug logs (juju debug-log). Here is what I got when running >> >> >> > your >> >> >> > smoke-test: http://pastebin.ubuntu.com/23516167/ (multiple SLF4J >> >> >> > bindings) >> >> >> > It seems the slf4j-log4j12-1.7.5.jar jar from Giraph is slightly >> >> >> > older >> >> >> > that >> >> >> > the slf4j-log4j12-1.7.10.jar from Hadoop. >> >> >> > >> >> >> > In order to add some output to the smoke-test action you could >> >> >> > call >> >> >> > "action >> >> >> > set" and/or "action fail" as we do here >> >> >> > >> >> >> > >> >> >> > >> >> >> > https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/hbase/layer-hbase/actions/smoke-test >> >> >> > >> >> >> > Thank you for your work, >> >> >> > Konstantinos >> >> >> > >> >> >> > >> >> >> > On Tue, Nov 22, 2016 at 11:06 AM, Merlijn Sebrechts >> >> >> > <[email protected]> wrote: >> >> >> >> >> >> >> >> Hi Panagiotis Liakos >> >> >> >> >> >> >> >> >> >> >> >> Cool that you're charming Giraph! Feel free to let me know when >> >> >> >> you >> >> >> >> have a >> >> >> >> working prototype, some of my colleagues are interested in using >> >> >> >> Giraph. >> >> >> >> >> >> >> >> Now for your question; tests and actions are two different >> >> >> >> things. >> >> >> >> >> >> >> >> Actions are similar to hooks; they run on the charm unit itself. >> >> >> >> The >> >> >> >> mahout smoke-test is an action. You run them using `run-action`, >> >> >> >> you >> >> >> >> can >> >> >> >> debug them using `juju debug-hooks` and all output gets logged by >> >> >> >> juju >> >> >> >> (you >> >> >> >> can see the output in `juju debug-log`). More info on actions: >> >> >> >> https://jujucharms.com/docs/stable/actions >> >> >> >> Tests are scripts that run on your own machine that deploy and >> >> >> >> test >> >> >> >> charms. You run them using bundletester. More info on tests: >> >> >> >> https://jujucharms.com/docs/2.0/developer-testing >> >> >> >> >> >> >> >> So if you want to alter the `smoke-test` action; you should alter >> >> >> >> this >> >> >> >> file: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test >> >> >> >> >> >> >> >> The actual tests of the Mahout Charm use the smoke-test action to >> >> >> >> verify >> >> >> >> the deployment. That is what you see at line 52: >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py#L52 >> >> >> >> `self.mahout.run_action('smoke-test')` calls `juju run-action` in >> >> >> >> the >> >> >> >> background. >> >> >> >> >> >> >> >> Does that help you a bit? >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Kind regards >> >> >> >> Merlijn >> >> >> >> >> >> >> >> 2016-11-22 9:42 GMT+01:00 Panagiotis Liakos <[email protected]>: >> >> >> >>> >> >> >> >>> Hi all, >> >> >> >>> >> >> >> >>> I am trying to build a charm for Apache Giraph. I have used the >> >> >> >>> mahout >> >> >> >>> charm ( >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/mahout >> >> >> >>> ) and made the necessary changes to deploy giraph instead of >> >> >> >>> mahout. >> >> >> >>> >> >> >> >>> Deployment seems to work fine and I am able to submit giraph >> >> >> >>> jobs >> >> >> >>> through an ssh connection. >> >> >> >>> >> >> >> >>> Now I am trying to create a test similar to the smoke-test of >> >> >> >>> the >> >> >> >>> mahout charm ( >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/actions/smoke-test >> >> >> >>> ) >> >> >> >>> >> >> >> >>> I have successfully run the script that I have included in my >> >> >> >>> smoke-test through ssh and I >> >> >> >>> would expect that the test would also execute without errors. >> >> >> >>> However, I am not at all familiar with the 'run-action' process >> >> >> >>> of >> >> >> >>> juju and I have no idea how I should alter the following file: >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/mahout/layer-mahout/tests/01-mahout-test.py >> >> >> >>> >> >> >> >>> In particular, I am puzzled by the way the result status is set >> >> >> >>> in >> >> >> >>> this example. Simply changing the references of mahout to giraph >> >> >> >>> does >> >> >> >>> not seem to work. You can see my file here: >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> https://github.com/panagiotisl/bigtop/blob/master/bigtop-packages/src/charm/giraph/layer-giraph/tests/01-giraph-test.py >> >> >> >>> >> >> >> >>> When I execute show-action-output I receive very limited >> >> >> >>> information: >> >> >> >>> message: exit status 1 >> >> >> >>> status: failed >> >> >> >>> timing: >> >> >> >>> completed: 2016-11-21 15:48:01 +0000 UTC >> >> >> >>> enqueued: 2016-11-21 15:47:38 +0000 UTC >> >> >> >>> started: 2016-11-21 15:47:40 +0000 UTC >> >> >> >>> >> >> >> >>> Is there a way I can view the full output of the 'smoke-test' >> >> >> >>> execution? >> >> >> >>> >> >> >> >>> Thank you, >> >> >> >>> Panagiotis Liakos >> >> >> >>> >> >> >> >>> -- >> >> >> >>> Juju mailing list >> >> >> >>> [email protected] >> >> >> >>> Modify settings or unsubscribe at: >> >> >> >>> https://lists.ubuntu.com/mailman/listinfo/juju >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Juju mailing list >> >> >> >> [email protected] >> >> >> >> Modify settings or unsubscribe at: >> >> >> >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> >> >> >> >> >> > >> >> > >> >> > >> > >> > > > -- Juju mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
