[
https://issues.apache.org/jira/browse/UNOMI-571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevan Jahanshahi updated UNOMI-571:
-----------------------------------
Description:
h4. The goal here is to re-introduce the extension system we originally planned
to implement to have a better integration with JSON Schema using the power of
_[schema
composition|https://json-schema.org/understanding-json-schema/reference/combining.html]_
_(allOf{-}, anyOf, oneOf{-})_ keyword.
The idea is:
* to store extensions as full JSON schemas
* update the original extended JSON schemas in RAM with *allOf* -OR *anyOf* OR
*oneOf*- referencing the extensions (only one default cardinality will be
handle at first: {*}allOf{*})
h1. Why do we need extensions system for json schema ?
because Unomi is going to close coimpletely the structure of events and
incoming data for security purpose, any unknow property not mapped in a json
schema will end up by rejecting the event.
Due to this we are loosing the power of the events data structure that was free
until Unomi 2.
To counter this limitation we want the basic Object and schema provided by
Unomi by default to be extensible so new properties and custom properties could
still be used in events that would use Unomi based object.
h1. Necessary changes, code and implem:
h3. Create the extension it self:
The extension that was a piece of JSON schema in first implem should now be a
full JSON Schema definition that will only contains the new props to be
validated on the same instance (object, event).
impacts:
* -remove the endpoints and custom storage of the JSON Schema extensions-
*(Already done)*
* use existing endpoints of JSON Schema to create them
** (we will probably have to introduce a new param or schema property to
identify those schemas as extensions)
h4. Before (first implem that have been reverted during the refacto):
{code:java}
{
"id": "extension-test-event-1",
"schemaId": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"metadata": {
"description": "Description of the extension",
"id": "extension-test-event-1",
"name": "The name of the extension"
},
"priority": "10",
"extension": {
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
}
],
"properties": {
"properties": {
"properties": {
"floatProperty": {
"type": "number",
"maximum": 100,
"description": "Extension of float property"
},
"stringProperty": {
"type": "string",
"maxLength": 100,
"description": "Updated description"
},
"newProperty": {
"type": "string",
"description": "A new element"
}
}
}
}
}
}
{code}
h4. After (expected implem):
{code:java}
{
"$id":
"https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEventExtension1",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
}
],
"properties": {
"properties": {
"properties": {
"floatProperty": {
"type": "number",
"maximum": 100,
"description": "Extension of float property"
},
"stringProperty": {
"type": "string",
"maxLength": 100,
"description": "Updated description"
},
"newProperty": {
"type": "string",
"description": "A new element"
}
}
}
}
}
{code}
* Some additional informations may be required on JSON Schema that are
extensions, like:
** {*}-priority-{*}: NOT necessary anymore, there is no notion of priority in
the allOf, anyOf, oneOf
** {-}*cardinality*{-}: NOT necessary in current implem, only "{*}allOf{*}"
will be used and by default. (we will see if it's needed in next steps)
** {*}originalSchemaID{*}: the id of the original schema to extend.
** This infos may be store in the self part of the JSON Schema that will be
extension (final structure could be discussed with the other dev of the team),
here is an example:
{code:java}
"self": {
"vendor": "org.apache.unomi",
"name": "testEventExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0",
"unomiExtends": {
"schema": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";
}
}
{code}
h3. The merge:
The merge was originally design to merge both the original JSON shema and the
extension schema in a final JSON schema merged and stored in RAM.
This was a bit complex to implement and could be handle in a more easy and
fashioned way using the allOf, anyOf capabilities.
Now that that the extensions are. real schemas we can just reference them using
this notation (reusing above exemple)
h4. original JSON Schema to be extended:
{code:java}
{
"$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventForExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEvent",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
}
],
"properties": {
"properties": {
"type": "object",
"properties": {
"floatProperty": {
"type": "number"
},
"stringProperty": {
"type": "string",
"maxLength": 50,
"description": "Initial description"
}
}
}
}
}{code}
h4. Original JSON Schema extended:
{code:java}
{
"$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventForExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEvent",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
},
{
"$ref":
"https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";
}
],
"properties": {
"properties": {
"type": "object",
"properties": {
"floatProperty": {
"type": "number"
},
"stringProperty": {
"type": "string",
"maxLength": 50,
"description": "Initial description"
}
}
}
}
}
{code}
As you can see we still need a merge operation that will store the results in
RAM, but the only part to update in the original schema is the "{*}allOf{*}"
property, by adding the extension schema URIs.
h3. Changes on the original schema:
In first implem we wanted to use the {*}additionalProperties: false{*}. but
this is not possible because this keyword doesn't work with allOf, anyOf or
other sub schemas combination.
So we will have to use: *unevaluatedProperties: false* insteads.
Read:
*
[https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties]
*
[https://json-schema.org/understanding-json-schema/reference/object.html#additional-properties]
NOTE that *unevaluatedProperties: false* should only be set on all our original
schemas.
Also we have to consider refactoring all our original schemas to avoid having
nested objects in schemas.
I attached 3 schemas to the ticket as an example of schema and extension for
the form event.
[^form.json] is the Original JSON Schema for the form event it self
is the Original JSON Schema for the form properties
[form.contact.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48993/48993_form.contact.json]is
the extension for a given form, providing fields for a contact form and
extending the
[form.properties.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48995/48995_form.properties.json]schema.
This does represent the final structure goal of schemas and extensions.
was:
h4. The goal here is to re-introduce the extension system we originally planned
to implement to have a better integration with JSON Schema using the power of
_[schema
composition|https://json-schema.org/understanding-json-schema/reference/combining.html]_
_(allOf{-}, anyOf, oneOf{-})_ keyword.
The idea is:
* to store extensions as full JSON schemas
* update the original extended JSON schemas in RAM with *allOf* -OR *anyOf* OR
*oneOf*- referencing the extensions (only one default cardinality will be
handle at first: {*}allOf{*})
h1. Why do we need extensions system for json schema ?
because Unomi is going to close coimpletely the structure of events and
incoming data for security purpose, any unknow property not mapped in a json
schema will end up by rejecting the event.
Due to this we are loosing the power of the events data structure that was free
until Unomi 2.
To counter this limitation we want the basic Object and schema provided by
Unomi by default to be extensible so new properties and custom properties could
still be used in events that would use Unomi based object.
h1. Necessary changes, code and implem:
h3. Create the extension it self:
The extension that was a piece of JSON schema in first implem should now be a
full JSON Schema definition that will only contains the new props to be
validated on the same instance (object, event).
impacts:
* -remove the endpoints and custom storage of the JSON Schema extensions-
*(Already done)*
* use existing endpoints of JSON Schema to create them
** (we will probably have to introduce a new param or schema property to
identify those schemas as extensions)
h4. Before (first implem that have been reverted during the refacto):
{code:java}
{
"id": "extension-test-event-1",
"schemaId": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"metadata": {
"description": "Description of the extension",
"id": "extension-test-event-1",
"name": "The name of the extension"
},
"priority": "10",
"extension": {
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
}
],
"properties": {
"properties": {
"properties": {
"floatProperty": {
"type": "number",
"maximum": 100,
"description": "Extension of float property"
},
"stringProperty": {
"type": "string",
"maxLength": 100,
"description": "Updated description"
},
"newProperty": {
"type": "string",
"description": "A new element"
}
}
}
}
}
}
{code}
h4. After (expected implem):
{code:java}
{
"$id":
"https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEventExtension1",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
}
],
"properties": {
"properties": {
"properties": {
"floatProperty": {
"type": "number",
"maximum": 100,
"description": "Extension of float property"
},
"stringProperty": {
"type": "string",
"maxLength": 100,
"description": "Updated description"
},
"newProperty": {
"type": "string",
"description": "A new element"
}
}
}
}
}
{code}
* Some additional informations may be required on JSON Schema that are
extensions, like:
** {*}-priority-{*}: NOT necessary anymore, there is no notion of priority in
the allOf, anyOf, oneOf
** {-}*cardinality*{-}: NOT necessary in current implem, only "{*}allOf{*}"
will be used and by default. (we will see if it's needed in next steps)
** {*}originalSchemaID{*}: the id of the original schema to extend.
** This infos may be store in the self part of the JSON Schema that will be
extension (final structure could be discussed with the other dev of the team),
here is an example:
{code:java}
"self": {
"vendor": "org.apache.unomi",
"name": "testEventExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0",
"unomiExtends": {
"schema": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";
}
}
{code}
h3. The merge:
The merge was originally design to merge both the original JSON shema and the
extension schema in a final JSON schema merged and stored in RAM.
This was a bit complex to implement and could be handle in a more easy and
fashioned way using the allOf, anyOf capabilities.
Now that that the extensions are. real schemas we can just reference them using
this notation (reusing above exemple)
h4. original JSON Schema to be extended:
{code:java}
{
"$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventForExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEvent",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
}
],
"properties": {
"properties": {
"type": "object",
"properties": {
"floatProperty": {
"type": "number"
},
"stringProperty": {
"type": "string",
"maxLength": 50,
"description": "Initial description"
}
}
}
}
}{code}
h4. Original JSON Schema extended:
{code:java}
{
"$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
"$schema": "https://json-schema.org/draft/2019-09/schema";,
"self": {
"vendor": "org.apache.unomi",
"name": "testEventForExtension",
"format": "jsonschema",
"target": "events",
"version": "1-0-0"
},
"title": "TestEvent",
"type": "object",
"allOf": [
{
"$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
},
{
"$ref":
"https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";
}
],
"properties": {
"properties": {
"type": "object",
"properties": {
"floatProperty": {
"type": "number"
},
"stringProperty": {
"type": "string",
"maxLength": 50,
"description": "Initial description"
}
}
}
}
}
{code}
As you can see we still need a merge operation that will store the results in
RAM, but the only part to update in the original schema is the "{*}allOf{*}"
property, by adding the extension schema URIs.
h3. Changes on the original schema:
In first implem we wanted to use the {*}additionalProperties: false{*}. but
this is not possible because this keyword doesn't work with allOf, anyOf or
other sub schemas combination.
So we will have to use: *unevaluatedProperties: false* insteads.
Read:
*
[https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties]
*
[https://json-schema.org/understanding-json-schema/reference/object.html#additional-properties]
NOTE that *unevaluatedProperties: false* should only be set on all our original
schemas.
Also we have to consider refactoring all our original schemas to avoid having
nested objects in schemas.
I attached 3 schemas to the ticket as an example of schema and extension for
the form event.
[form.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48994/48994_form.json]is
the Original JSON Schema for the form event it self
[form.properties.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48995/48995_form.properties.json]is
the Original JSON Schema for the form properties
[form.contact.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48993/48993_form.contact.json]is
the extension for a given form, providing fields for a contact form and
extending the
[form.properties.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48995/48995_form.properties.json]schema.
This does represent the final structure goal of schemas and extensions.
> JSON Schema extension system
> ----------------------------
>
> Key: UNOMI-571
> URL: https://issues.apache.org/jira/browse/UNOMI-571
> Project: Apache Unomi
> Issue Type: New Feature
> Reporter: Kevan Jahanshahi
> Priority: Major
> Attachments: form.contact.json, form.json, form.properties.json
>
>
> h4. The goal here is to re-introduce the extension system we originally
> planned to implement to have a better integration with JSON Schema using the
> power of _[schema
> composition|https://json-schema.org/understanding-json-schema/reference/combining.html]_
> _(allOf{-}, anyOf, oneOf{-})_ keyword.
> The idea is:
> * to store extensions as full JSON schemas
> * update the original extended JSON schemas in RAM with *allOf* -OR *anyOf*
> OR *oneOf*- referencing the extensions (only one default cardinality will be
> handle at first: {*}allOf{*})
> h1. Why do we need extensions system for json schema ?
> because Unomi is going to close coimpletely the structure of events and
> incoming data for security purpose, any unknow property not mapped in a json
> schema will end up by rejecting the event.
> Due to this we are loosing the power of the events data structure that was
> free until Unomi 2.
> To counter this limitation we want the basic Object and schema provided by
> Unomi by default to be extensible so new properties and custom properties
> could still be used in events that would use Unomi based object.
> h1. Necessary changes, code and implem:
> h3. Create the extension it self:
> The extension that was a piece of JSON schema in first implem should now be a
> full JSON Schema definition that will only contains the new props to be
> validated on the same instance (object, event).
> impacts:
> * -remove the endpoints and custom storage of the JSON Schema extensions-
> *(Already done)*
> * use existing endpoints of JSON Schema to create them
> ** (we will probably have to introduce a new param or schema property to
> identify those schemas as extensions)
> h4. Before (first implem that have been reverted during the refacto):
> {code:java}
> {
> "id": "extension-test-event-1",
> "schemaId": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
> "metadata": {
> "description": "Description of the extension",
> "id": "extension-test-event-1",
> "name": "The name of the extension"
> },
> "priority": "10",
> "extension": {
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
> }
> ],
> "properties": {
> "properties": {
> "properties": {
> "floatProperty": {
> "type": "number",
> "maximum": 100,
> "description": "Extension of float property"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 100,
> "description": "Updated description"
> },
> "newProperty": {
> "type": "string",
> "description": "A new element"
> }
> }
> }
> }
> }
> }
> {code}
> h4. After (expected implem):
> {code:java}
> {
> "$id":
> "https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";,
> "$schema": "https://json-schema.org/draft/2019-09/schema";,
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEventExtension1",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/customitem/1-0-0";
> }
> ],
> "properties": {
> "properties": {
> "properties": {
> "floatProperty": {
> "type": "number",
> "maximum": 100,
> "description": "Extension of float property"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 100,
> "description": "Updated description"
> },
> "newProperty": {
> "type": "string",
> "description": "A new element"
> }
> }
> }
> }
> }
> {code}
> * Some additional informations may be required on JSON Schema that are
> extensions, like:
> ** {*}-priority-{*}: NOT necessary anymore, there is no notion of priority
> in the allOf, anyOf, oneOf
> ** {-}*cardinality*{-}: NOT necessary in current implem, only "{*}allOf{*}"
> will be used and by default. (we will see if it's needed in next steps)
> ** {*}originalSchemaID{*}: the id of the original schema to extend.
> ** This infos may be store in the self part of the JSON Schema that will be
> extension (final structure could be discussed with the other dev of the
> team), here is an example:
>
> {code:java}
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0",
> "unomiExtends": {
> "schema": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";
> }
> }
> {code}
>
> h3. The merge:
> The merge was originally design to merge both the original JSON shema and the
> extension schema in a final JSON schema merged and stored in RAM.
> This was a bit complex to implement and could be handle in a more easy and
> fashioned way using the allOf, anyOf capabilities.
> Now that that the extensions are. real schemas we can just reference them
> using this notation (reusing above exemple)
> h4. original JSON Schema to be extended:
> {code:java}
> {
> "$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
> "$schema": "https://json-schema.org/draft/2019-09/schema";,
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventForExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEvent",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
> }
> ],
> "properties": {
> "properties": {
> "type": "object",
> "properties": {
> "floatProperty": {
> "type": "number"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 50,
> "description": "Initial description"
> }
> }
> }
> }
> }{code}
> h4. Original JSON Schema extended:
> {code:java}
> {
> "$id": "https://unomi.apache.org/schemas/json/events/test-event/1-0-0";,
> "$schema": "https://json-schema.org/draft/2019-09/schema";,
> "self": {
> "vendor": "org.apache.unomi",
> "name": "testEventForExtension",
> "format": "jsonschema",
> "target": "events",
> "version": "1-0-0"
> },
> "title": "TestEvent",
> "type": "object",
> "allOf": [
> {
> "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0";
> },
> {
> "$ref":
> "https://unomi.apache.org/schemas/json/events/test-event-extension-1/1-0-0";
> }
> ],
> "properties": {
> "properties": {
> "type": "object",
> "properties": {
> "floatProperty": {
> "type": "number"
> },
> "stringProperty": {
> "type": "string",
> "maxLength": 50,
> "description": "Initial description"
> }
> }
> }
> }
> }
> {code}
> As you can see we still need a merge operation that will store the results in
> RAM, but the only part to update in the original schema is the "{*}allOf{*}"
> property, by adding the extension schema URIs.
> h3. Changes on the original schema:
> In first implem we wanted to use the {*}additionalProperties: false{*}. but
> this is not possible because this keyword doesn't work with allOf, anyOf or
> other sub schemas combination.
> So we will have to use: *unevaluatedProperties: false* insteads.
> Read:
> *
> [https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties]
> *
> [https://json-schema.org/understanding-json-schema/reference/object.html#additional-properties]
> NOTE that *unevaluatedProperties: false* should only be set on all our
> original schemas.
> Also we have to consider refactoring all our original schemas to avoid having
> nested objects in schemas.
> I attached 3 schemas to the ticket as an example of schema and extension for
> the form event.
> [^form.json] is the Original JSON Schema for the form event it self
> is the Original JSON Schema for the form properties
> [form.contact.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48993/48993_form.contact.json]is
> the extension for a given form, providing fields for a contact form and
> extending the
> [form.properties.json{^}!https://jira.jahia.org/images/icons/link_attachment_7.gif|width=7,height=7!{^}|https://jira.jahia.org/secure/attachment/48995/48995_form.properties.json]schema.
> This does represent the final structure goal of schemas and extensions.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
