Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Patrick Carlisle updated an issue Puppet / PUP-8601 agent should support rich data content negotiation Change By: Patrick Carlisle Release Notes Summary: The `rich_data` setting is now enabled by default. Catalog requests have two new content-types, `application/vnd.puppet.rich+json` and `application/vnd.puppet.rich+msgpack`, that are used when both master and agent have this enabled (and depending on whether `preferred_serialization_format` is `json` or `msgpack`). Release Notes: New Feature Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Kenn Hussey commented on PUP-8601 Re: agent should support rich data content negotiation Patrick Carlisle please provide release notes for this issue if needed, thanks! Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Kenn Hussey updated an issue Puppet / PUP-8601 agent should support rich data content negotiation Change By: Kenn Hussey Flagged: Impediment Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Kenn Hussey updated an issue Puppet / PUP-8601 agent should support rich data content negotiation Change By: Kenn Hussey Flagged: Impediment Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Maggie Dreyer updated an issue Puppet / PUP-8601 agent should support rich data content negotiation Change By: Maggie Dreyer Fix Version/s: PUP 6.0.0 Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Henrik Lindberg commented on PUP-8601 Re: agent should support rich data content negotiation We can clearly leave the options for later - at the moment they cannot really be used for the catalog anyway. Regarding MIME type - earlier when discussing this for XPP we came up with these: mime type meaning application/vnd.puppet.pcore+json pcore serialization in json application/vnd.puppet.pcore+msgpack pcore serialization in msgpack application/vnd.puppet.pcore+yaml pcore serialization in yaml At that point we did not have the generic data format - so maybe use: mime type meaning application/vnd.puppet.rich+json generic rich data in json application/vnd.puppet.rich+msgpack generic rich data in msgpack application/vnd.puppet.rich+yaml generic rich data in yaml
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Patrick Carlisle commented on PUP-8601 Re: agent should support rich data content negotiation For now I've just threaded the rich_data setting through the context instead of settings and actually it is working, so I seem to have the basics going here with one mimetype that's negotiated. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Patrick Carlisle commented on PUP-8601 Re: agent should support rich data content negotiation How little of this can we do in the next two weeks without making life hard later? Can we define one mimetype that does the current default rich_data format and leave options to later? Maybe include them in the string but not allow changing them for now. I've started to look at threading this through from the request and it seems less trivial than I thought (checks global or environment level settings right now), but I'm not really familiar at all with how rich data encoding works in the actual puppet codebase. I think I'll need help with that side of things to get this done in time. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Henrik Lindberg commented on PUP-8601 Re: agent should support rich data content negotiation Another aspect of content negotiation is that we need to make this available to the compiler to allow users to handle update of agents more gracefully; i.e. one code path for agents that does not accept "rich data" (and therefore cannot handle Deferred etc), and another path for those that can (that is, if users want to do this with the same set of classes; which is probably the easiest). Thus, the result of content negotiation should be available and it cannot really be among the facter facts, but could possibly be placed in $facts even if it is a binding that takes place at time of communication rather than time of sending the facts. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Henrik Lindberg commented on PUP-8601 Re: agent should support rich data content negotiation Patrick Carlisle There is now a PR against specifications with information about rich data representation: PUP-9070 (https://github.com/puppetlabs/puppet-specifications/pull/121). To summarize - we have: Generic Data Representation (what we use for --rich_data and Catalog) Pcore Serialization (not used in any user facing way yet) The Generic Data Representation means that rich data is transformed to a Data compliant value and thus something that can be serialized with compliant JSON (or something else). The Pcore Serialization supports JSON and MsgPack (not yet merged). This form requires an implementation of our format at both ends even though the underlying data stream (JSON or MsgPack) can be read with only the respective technology. These protocols are more efficient than Generic Data Representation, but because of the need to have logic to comprehend the stream it was deemed too much work for PDB to us this. I think that in the future there will be a performance advantage for users to use the Pcore + MsgPack in production (pending benchmarking etc). To take advantage of this possibility in the future we should have a content negotiation that is capable of handling all of the options. The options are: Current Lossy Format (not rich data, some data as strings), Generic Data Format or Pcore Serialization On the wire format: JSON, MsgPack Options wrt to deduplification and type references By using dedup=true all complex values will be decomposed and common parts are only serialized once. This has the advantage that the deserialized content ends up taking less memory By using types_by_reference = true, only the "names" of data types are used for custom types and the recipient must know what they are. When set to false, the type itself is serialized the first time it is encountered. (This is the Pcore definition of the type, if an implementation of behavior is required it must still be present on the receiving side). The current --rich_data format is GenericData + JSON + dedup=false, types_by_reference=true. More work is required to benefit from the dedup=true and type_by_reference=false options. I think we should express this using a MIME type with three parts: SEMANTIC + WIRE + OPTIONS. Will dig up what we came up with earlier when discussing use of MIME types in a follow up comment.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Patrick Carlisle commented on PUP-8601 Re: agent should support rich data content negotiation Henrik Lindberg You mentioned this week that you were going to update this ticket with some more about the possibilities of the rich data format I think? Maybe just the permutations or options that could occur. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Henrik Lindberg commented on PUP-8601 Re: agent should support rich data content negotiation I am going to make a spike and see how hard it is to change the runtime representation of `Binary` to be an 8-bit-ascii `String`. This because I rather do that rather than combining PSON with rich-data. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Patrick Carlisle assigned an issue to Patrick Carlisle Puppet / PUP-8601 agent should support rich data content negotiation Change By: Patrick Carlisle Assignee: Patrick Carlisle Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Josh Cooper commented on PUP-8601 Re: agent should support rich data content negotiation FWIW, the agent will also downgrade to PSON if the facts it's submitting as part of the catalog request contain binary, or if the report contains binary. Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Eric Thompson commented on PUP-8601 Re: agent should support rich data content negotiation Henrik Lindberg the predocs says: Consider altering deserialization (or post processing of catalog) such that all Binary instances are changed to Ruby String with ASCII-8-bit encoding. This would make transport of Binary values backwards compatible. Current state means that all types/providers that expect binary data need to be aware of the Binary data type and get the ascii-8-bit buffer from the wrapping Binary instance. is that still being considered? it seems like it would fit in with option2 above (Raise an error and tell user to use a Binary instead of an ASCII-8bit). Add Comment This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title
Eric Thompson updated an issue
Puppet / PUP-8601
agent should support rich data content negotiation
Change By:
Eric Thompson
The 6.0 agent should request a catalog using a rich-data encoding by default.The request rather than the current {{\--rich-data}} setting should control the format of the catalog. (The setting is still needed but it should default to true for command line execution).An agent < 6.0 when talking to a server >= 6.0.0 will not know how to request rich data format (now it assumes rich data if it has {{\--rich-data}} turned on.I think it is safe to assume that we can simply ignore the {{\--rich-data}} flag on the agent side and state that in order to use rich-data you need to use a 6.y agent.Further - in the current protocol, we change the encoding from JSON (the default) to PSON if we detect ASCII-8bit String data. With rich-data encoding we should not do this. There are options:# (Magically) transform the ASCII-8bit String to a Binary - the agent will need to be able to deal with an instance of Binary (our File resource will be able to do so).# Raise an error and tell user to use a Binary instead of an ASCII-8bit# (Magically) deserialize a Binary to ASCII-8bit on the agent side to allow types and providers that are unaware of the Binary type work as they did before. (This has bad side effects further downstream in reports etc).Number 2 is the preferred (non magical) solution.
Add Comment
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Jira (PUP-8601) agent should support rich data content negotiation
Title: Message Title Henrik Lindberg created an issue Puppet / PUP-8601 agent should support rich data content negotiation Issue Type: Improvement Assignee: Unassigned Created: 2018/03/23 7:56 AM Priority: Normal Reporter: Henrik Lindberg The 6.0 agent should request a catalog using a rich-data encoding by default. The request rather than the current --rich-data setting should control the format of the catalog. (The setting is still needed but it should default to true for command line execution). An agent < 6.0 when talking to a server >= 6.0.0 not know how to request rich data format (now it assumes rich data if it has --rich-data turned on. I think it is safe to assume that we can simply ignore the --rich-data flag on the agent side and state that in order to use rich-data you need to use a 6.y agent. Further - in the current protocol, we change the encoding from JSON (the default) to PSON if we detect ASCII-8bit String data. With rich-data encoding we should not do this. There are options: (Magically) transform the ASCII-8bit String to a Binary - the agent will need to be able to deal with an instance of Binary (our File resource will be able to do so). Raise an error and tell user to use a Binary instead of an ASCII-8bit (Magically) deserialize a Binary to ASCII-8bit on the agent side to allow types and providers that are unaware of the Binary type work as they did before. (This has bad side effects further downstream in reports etc). Number 2 is the preferred (non magical) solution.
