On Tue, Oct 15, 2013 at 6:51 AM, Pablo Fernandez <[email protected]>wrote:
> Hehe, sorry for the auto-reply... but I made it work, and I wanted to > post it for reference. > > The key was to add also the edges: it can't be an empty list. > > For the record, doing: > > # curl -vv -G -H "Accept: application/json" ' > http://localhost:8080/v2/commands' --data-urlencode 'payload={ > > "command": "replace catalog", > "version": 2, > "payload": "{ > \"metadata\": { > \"api_version\": 1 > }, > \"data\": { > \"name\": \"myhost.blabla.bla\", > \"version\": \"1111\", > \"edges\": [ > {\"source\": {\"type\": \"Service\", \"title\": \"iptables\"}, > \"target\": {\"type\": \"Service\", \"title\": \"iptables\"}, > \"relationship\": \"before\"} > ], > \"resources\": [ > > {\"type\": \"Service\", > \"title\": \"iptables\", > \"exported\": false, > \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\", > \"line\": 121, > \"tags\": [\"node\", \"puppetdb\"], > \"parameters\": {\"hasrestart\": true, \"ensure\" : \"running\"} > } > ] > } > }" > }' > > Works like a charm, and add a new host into the PuppetDB out of the blue. > Next: automatically serialize those escaped quotes and try to see if it > exported resources work as they should. > > Thanks, and sorry again for the noise, > BR/Pablo > Hi Pablo, I'm glad you got this working and I apologize it took so long. A few comments: 1) You don't need to formulate the "payload" as a pre-serialized JSON string (with all the requisite escaping and whatnot) anymore. As of "replace catalog" version 2, you can just inline the raw JSON object itself: "payload": {"data": ..., "metadata": ..., ...} The entire command is a single JSON object that way, and the payload is just a normal value in that hash. 2) You are correct that the edges key is required, and should contain every relationship specified in the supplied catalog: http://docs.puppetlabs.com/puppetdb/1.5/api/wire_format/catalog_format.html#main-data-type-catalog I do agree that the error message isn't helpful at all, though, and we should fix that. It's likely that we aren't doing proper validation on that and letting things get to the storage code, which throws an opaque, low-level error that is difficult to debug. That said, as there aren't any relationships specified in your above set of resource (no require/subscribe params, for example) I'm curious why the empty list of edges fails. That smells like a bug to me; would you mind filing a ticket against PuppetDB with the above command included? We can take a look and see what's up. 3) With curl, you can supply an argument of "--data-urlencode [email protected]". Then you can just put the payload in a mycatalog.json file, and curl should automatically load the content and urlencode it for you. See http://curl.haxx.se/docs/manpage.html#--data-urlencode deepak > > > > On 10/15/2013 11:25 AM, Pablo Fernandez wrote: > > If I try to add a fictitious resource: > > {\"type\": \"Service\", > \"title\": \"iptables\", > \"exported\": false, > \"file\": \"/etc/puppet/modules/puppetdb/manifests/server.pp\", > \"line\": 121, > \"tags\": [\"node\", \"puppetdb\"], > \"parameters\": { > \"ensure\" : \"running\"} > } > > I get: > > 2013-10-15 11:20:43,042 ERROR [command-proc-45] [puppetdb.command] > [15abcd55-b7c6-4acc-8839-94b96f85ba68] [replace catalog] Retrying after > attempt 8, due to: org.postgresql.util.PSQLException: No value specified > for parameter 1. > > > This way of issuing commands is quite nasty... > > BR/Pablo > > > On 10/15/2013 11:00 AM, Pablo Fernandez wrote: > > Ok, so I finally opted for inserting a full node from one DB into another, > since getting the resources should be quite easy with a GET. > > But then I run into a problem when running the replace catalog command > (with an empty catalog): > > curl -vv -G -H "Accept: application/json" ' > http://localhost:8080/v2/commands' --data-urlencode > 'payload={"command":"replace catalog","version": 2,"payload":"{ > \"metadata\": {\"api_version\": 1}, \"data\": {\"name\": \" > puppet09.cscs.ch\",\"version\": \"1111\", \"edges\": [], \"resources\": > [] } }"}' > > Which returns ok. The puppetdb.log shows: > > 2013-10-15 10:42:02,060 ERROR [command-proc-45] [puppetdb.command] > [43c41752-e7d4-4725-b062-7b3a9b1ac8ec] [replace catalog] Retrying after > attempt 8, due to: org.postgresql.util.PSQLException: ERROR: syntax error > at or near ")" > Position: 66 > > I have looked again and again into the data, and syntax seems to be > correct. > > Anybody tried this? Hints? > > Thanks a lot, > Pablo Fernandez > > > > > On 10/10/2013 05:23 PM, Throwe, Jesse wrote: > > That would really depend on the approach ultimately taken for the > multi-master. If the puppet master (or agent in masterless mode) forwards > the same fact/report/node/etc data to multiple puppetdb instances then yes > the scale breaks horribly and does not meet part of this request. If there > is some kind of store-and-forward mechanism in puppetdb itself, it would be > possible to extend the forward to say 'only if it matches these rules'. > This could definitely be useful in a variety of ways beyond simple HA > purposes. > > > > On Thu, Oct 10, 2013 at 11:07 AM, Pablo Fernandez <[email protected] > > wrote: > >> Yes, I saw that... this may be a solution for replicating full PuppetDBs >> (if you tackle the backlog during a long shortage), but that does not scale >> very well if you have 5-10 PuppetDBs all being replicated to a single, >> bigger one. >> >> Besides, it would not cover cases when you want to have only some nodes >> replicated, or even only some resources. >> >> >> >> On 10/10/2013 04:35 PM, Throwe, Jesse wrote: >> >> It sounds like a shade of #19321 - >> https://projects.puppetlabs.com/issues/19321 >> >> >> On Thu, Oct 10, 2013 at 10:31 AM, Wolf Noble <[email protected]> wrote: >> >>> I'd be interested in accomplishing a similar thing. I want to have a >>> central reporting puppetdb which I can (outside of puppet) query for >>> multisite information without tying isolated environments together via a >>> single postgres db, or using multi-master replication. >>> >>> my needs do not involve making each puppet site aware of each other's >>> data, but it would be nice to have one db which has all the data for my >>> digging pleasure. >>> >>> from my understanding this is not currently supported. >>> >>> I was thinking something could be done to ask puppetdb at site A (what >>> updates have you had in the past N minutes/ since the last time I checked >>> in with you/ since timestamp(N)), and then iterate through the collected >>> data and feed it into a reporting puppetdb instance, however I've not gone >>> beyond thinking about it. >>> >>> >>> >>> >>> >>> On Thu, Oct 10, 2013 at 7:12 AM, Pablo Fernandez < >>> [email protected]> wrote: >>> >>>> Thanks Ken, >>>> >>>> I took a look at the Command API and seems to be quite interesting, but >>>> it triggers a few more questions. >>>> >>>> As a reminder (also for others), what I wanted to do was to add exported >>>> resources from Node_A (managed by PuppetMaster_A that has its own >>>> PuppetDB_A) into a different PuppetDB_B, where Node_A is unknown, in >>>> order for other Nodes_B (managed by PuppetMaster_B) be aware and >>>> possibly realize those exported resources. >>>> >>>> Taking a look the Commands API seem to capabilities to change a whole >>>> catalog, or facts. Do they need to be existing catalogs, or could I add >>>> new ones? >>>> I am thinking: If I want to add Node_A's exported resource(s) to >>>> PuppetDB_B, I could "artificially" add its whole catalog to >>>> PuppetDB_B... but the Command API is "replace catalog", no "add >>>> catalog". Would it be rejected? How does the first catalog of a node get >>>> introduced into the DB? And moreover, how do I query a whole catalog? >>>> >>>> Another possibility would be to include the resource inside an existing >>>> Node_B catalog on the PuppetDB_B (after all, exported resources don't >>>> really mater what node do they belong, as long as they have the right >>>> attributes, right?), but the Catalog wire format seems to require the >>>> whole set of resources, so, if I define the resources of Node_B with >>>> some exported resources from Node_A, I believe I would lose the original >>>> Node_B's resources. The only possibility would be to query Node_B full >>>> catalog, and *add also* those resources from Node_A before sending them >>>> to the DB... and this seems to me like a bit dirty :) >>>> >>>> Am I missing anything? Or is there another way to implemen what I need >>>> with Puppet 3.2? >>>> >>>> Thanks again, >>>> BR/Pablo >>>> >>>> >>>> On 10/10/2013 09:16 AM, Ken Barber wrote: >>>> >>> Is there a mechanism to pull and push some exported resources from >>>> one >>>> >>> PuppetDB to another? I guess this could be done with the API, but >>>> has >>>> >>> anybody tried it, or will it work at all? >>>> > Not provided by us, yet. We've had a lot of discussions internally >>>> > about this kind of thing. How about you outline your requirements in >>>> > the form of a feature request here: >>>> > >>>> > http://projects.puppetlabs.com/projects/puppetdb/issues >>>> > >>>> >> Did anybody try to push data to a PuppetDB by hand? The API does not >>>> >> seem to have the PUT or POST methods. >>>> > The client obviously pushes uses POST :-). The documentation for the >>>> > mechanism is here: >>>> > >>>> > http://docs.puppetlabs.com/puppetdb/1.5/api/commands.html >>>> > >>>> > ken. >>>> > >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/puppet-users. >>>> For more options, visit https://groups.google.com/groups/opt_out. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/puppet-users. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/puppet-users. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/puppet-users. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.
