[jira] [Commented] (FELIX-5332) Serializer

2016-09-25 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521550#comment-15521550
 ] 

David Leangen commented on FELIX-5332:
--

Thanks, [~bosschaert]!

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-24 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519445#comment-15519445
 ] 

David Bosschaert commented on FELIX-5332:
-

Hi David,

Thanks - I'll plan to merge soon. 
BTW I would still consider this module to be part of the general 'converter' 
subproject, just a different module of that subproject :)

I'm sure we can improve going forward and try to reduce duplication etc.

Cheers,

David

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-21 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511473#comment-15511473
 ] 

David Leangen commented on FELIX-5332:
--

Hey [~bosschaert], it was not really my intent to own a new project 
(unfortunately, I don't really have the time or energy right now), so I just 
want to stop and think a bit about what we are doing before I dig myself in any 
deeper. :-)

The code I wrote is a very simplistic example, and needs a lot of work. It 
"works", in that I am able to do serialization with it, and I am now actually 
using it internally, but it's nowhere near ready for general use, I think.

The main point I was trying to make is that OSGi is creating a "Serialization" 
spec, does not seem to be giving consideration for what, I believe, is a 
critical element of serialization: the schema. Without a schema, complex data 
cannot be deserialized. The only reason this can work without a Schema object 
is because during deserialization, the application already knows which DTO 
object (including help from TypeReference) needs to be used. If that context is 
not there, or if the object is too "complex", then boom!

There are a few schema types out there. Here are a few:

 * xsd
 * [protobuf|https://developers.google.com/protocol-buffers/]
 * [Cap'n Proto|https://capnproto.org/]

I am sure there are others. (I think the last one is really neat, as it also 
explicitly deals with generics.)

The DTO in OSGi *is* the data schema. If it can not be properly serialized / 
deserialized, then there is a serious problem, IMHO. I was trying to point out 
this problem, but it is turning out to be much more involved than I had 
originally imagined.

The above listed solutions are "complete", but they also drag in a log of 
baggage. It would seem that the last 2 even require a native compiler to be 
installed. They all seem to work "offline" or during the build, rather than at 
runtime. It would be awesome to have a language agnostic schema, which could be 
converted into any of these formats. The DTO has that potential. It would be 
extremely powerful, combined with the Converter and "Serializer".

I think my little experiment here can be considered both a success and a 
failure. It is a success in that I was able to accomplish the job of 
serializing somewhat complex data (complex enough that it could not be done 
without a schema) by way of this simple prototype, but it was a failure in that 
we have not yet found a good way to integrate it, and I don't think I've yet 
convinced anybody of its necessity.

I think we need to sit back and think of something more convincing, perhaps, 
before continuing to pursue my experiment.

wdyt?


BTW, I am having a bit of trouble with naming it a "Serializer" if it cannot 
completely do the job... Codec actually seemed more appropriate. Just sayin'. 
:-)

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-21 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509547#comment-15509547
 ] 

David Bosschaert commented on FELIX-5332:
-

Hi [~dleangen], there is no rush at all - as a separate module this can evolve 
using its own timeline :)

The idea of 'withContext(Object obj)' is to use that in place of what you 
proposed initially as with(Schema x) but then a little more generic. This would 
just provide a way to provide the (de)serializer with extra context, which 
could be a schema, but might also be something other implementation specific. 
It would not tie the Serializer API to whatever the context is, and 
implementations can choose to supply whatever they want, including a Schema 
object.

As a suggested starting point I have added the withContext() to the Serializer 
and Deserializer APIs. By default they do nothing (as not all serializers need 
this context), but implementations can use these to get the Schema... Do you 
think this might work? It's just an idea...


> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-20 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507367#comment-15507367
 ] 

David Leangen commented on FELIX-5332:
--

Hi [~bosschaert], yeah, it does make the code much less light-weight. It would 
appear to roughly double the size, so I think you are right that it should be 
separated out into a separate API/module. It would appear that I was completely 
mistaken when I originally assumed that it would overlap with the same code. It 
could, but it would make the code much more complex, which is not desirable. To 
keep it separate means that the codebase doubles in size, which is also not 
desirable. Sigh. :-(

I can make this change, but probably can't start until about next week or so. 
Is that ok?

I'm not entirely sure what you mean by withContext(Object obj), so I may need 
your help.

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-20 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15506873#comment-15506873
 ] 

David Bosschaert commented on FELIX-5332:
-

Hi [~dleangen], this is quite a substantial amount of functionality! :)

What I think might be a good idea is to isolate it out into its own module. 
Alongside converter and serializer, maybe something called 
'org.apache.felix.schematizer'. I think for the API we should probably use API 
in the org.apache.felix namespace for the moment. Then we can maybe add a 
withContext(Object obj) to the Serializer interface (or maybe to the 
Serializing/Deserializing ones). This could be a generic mechanism to provide 
implementation-specific context to the serializer, which could be the Schema in 
this case.

Reason for such an approach is that the converter is currently super-light. It 
would be nice to keep it light IMHO. Having it as a separate module means that 
only those who need this advanced feature get the complexity.

WDYT?

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>Assignee: David Bosschaert
> Attachments: patch.diff
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-14 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489715#comment-15489715
 ] 

David Leangen commented on FELIX-5332:
--

I have made another attempt. Here are a few additional learnings, and a bit of 
an explanation about what I've done. This patch (which I will submit right 
away) is up-to-date with the current version HEAD in the felix git repo.

In addition to a few API updates and corresponding implementation classes, I 
have submitted several classes in the test cases. The goal here is to mock what 
I would consider to be an epitomic use case. Here, I created a mock-up of 
Prevayler. Prevayler needs to serialize objects (normally to a file on disk), 
then deserialize the objects. Problem is, there is no way to know which object 
is being deserialized, and no knowledge about the object, so without some help, 
deserialization will not work.

To give a bit more context: each "transaction" is wrapped in an object 
prescribed by Prevayler. I created an arbitrary means of wrapping up the 
transaction as a DTO. Probably, this part needs much more thought. 
Specifically: what should go into the OSGi Serializer, and what should not. I 
think some of the stuff going on outside should actually be brought inside, 
since it is stuff that will *always* have to be done (such as providing an 
identity for a Schema to avoid having to also serialize the entire schema along 
with the payload).

I think that the Schematizer could eventually be much "smarter" and require 
fewer hints via rules, but I am running out of time. Working with rules was 
quicker for me, because it allowed me to dumb down the Schematizer.

If there is anything unclear in the code, please ask. I can also submit 
comments.

The code helps in two ways:

 #  Creates a 
[Schema|https://github.com/dleangen/felix/blob/serializer/converter/converter/src/main/java/org/osgi/service/converter/Schema.java]
 of the object to serialize, which can provide all the hints necessary to 
properly deserialize later
 # Serializes the Schema by means of an alias, so as to not take up disk space 
unnecessarily

I'm sure there is a more intelligent way to serialize the alias, but whatever. 
This works. :-)

I'm sure others will find lots and lots of improvements to make, but as I say, 
I hope at least that this will start a serious discussion.

Some limitations:

 * Only works with DTOs for now (maybe that is how it should remain, I don't 
know).
 * Code could be cleaner (but is a good starting point for discussion).
 * Introduces new objects into the API (so adoption / agreement may not be 
easy).

BTW, I find it much easier working with git than submitting patches via svn. 
[Here is the branch on my git 
clone|https://github.com/dleangen/felix/tree/serializer/converter]. Would it be 
possible to make a pull request next time instead of submitting a patch?

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-08 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473494#comment-15473494
 ] 

David Leangen commented on FELIX-5332:
--

Ok, [~bosschaert], I have simplified and integrated into the code base 
according to my understanding of your suggestions. Please take a look.

There are still (at least) two things missing:
 * Better client integration (will get back to this later)
 * Handing of Collections (will try to do this very soon)

I have attached the patch file. Hope this works, Instead of working via svn, I 
am now using git, so I created the patch from there. Feel free to go directly 
to my github clone if that's easier. :-)

Look forward to your comments.

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-06 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467168#comment-15467168
 ] 

David Leangen commented on FELIX-5332:
--

[~bosschaert], in your example above, you suggest:

{code}
mySpecialCodec.decode(sometextfile).withSchema(mySchema).to(MyDTO.class)
{code}

Do you think we should make Schema a public interface, so mySchema is a Schema 
object? Or should Schema be "hidden" in the implementation, so mySchema is the 
name of the Schema, and the Schema object is private to the implementation?

If Schema is public, we would need to update the API to include both the Schema 
and Node interfaces. Is that ok? I think that would be clearer, and therefore 
preferable, but I don't know how much flexibility you have with regards to API 
changes.

BTW, I believe that the above would actually be:

{code}
mySpecialCodec.decode(sometextfile).to(mySchema)
{code}

or maybe

{code}
mySpecialCodec.decode(sometextfile).with(mySchema)
{code}

The reason is because including both the Schema *and* the target DTO is 
redundant, and could cause problems if there is a conflict between the two 
(i.e. the target DTO does not match the schema).

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-06 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466950#comment-15466950
 ] 

David Bosschaert commented on FELIX-5332:
-

[~dleangen], yes, ideally it would be a patch on the existing codebase :)

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-05 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465895#comment-15465895
 ] 

David Leangen commented on FELIX-5332:
--

[~bosschaert], I will try to update according to your comments. First, though, 
I am going to try to get Prevayler working so we can have an "outside" view 
with "working" code.

Please let me know how I should provide these changes. Just continue on this 
branch in my fork, and you do the merge??

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-05 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464837#comment-15464837
 ] 

David Leangen commented on FELIX-5332:
--

Thank you, [~bosschaert].

{quote}
To me, the serializer/schematizer sounds a bit like a special Codec to me. For 
example if you used it with XML it could produce an XML schema as part of the 
serialization and then do the deserialization together with the schema. 
{quote}

Yes, I agree! I did not want to complicate the existing code, so I put 
everything into a separate class, service. What I realised when writing this 
code, though, is that my original assumption ("we are parsing the object 
anyway, so handing the schema should be easy and we should therefore take 
advantage of this") was a bit off. True that we are doing parsing, but parsing 
for the Schema turns out to be different enough that either (1) it should be in 
a different class, or (2) the algorithms will be significantly complexified. 
That said, if the code is well structured, I'm sure it could be done in a nice 
way. My experience tells me (and probably you've noticed the same thing) that 
working with reflection code on this level can quickly turn into a big mess. 
That is my only worry.

{quote}
It would be nice if the codec (or other API) could be such that it would 
support such implementations.
{quote}

Yes, I agree!

{quote}
To produce a schema, this might just be done as a by-product of the encoding 
process? Maybe additionally configured via ConfigAdmin configuration factories? 
Do we need anything extra in the codec API for this?
{quote}

Again, I agree with you. This can, and probably should, be handled behind the 
scenes, at least for serialization/deserialization. For deep object 
introspection and manipulation, we would actually need access to the Schema 
object, which would require updating the API. Perhaps that could be offered as 
a "non-standard" service as a pilot to see if anybody would actually use it.

Back to the serialization-only part, though: I am using the [Prevayler 
Serializer|https://github.com/jsampson/prevayler/blob/master/core/src/main/java/org/prevayler/foundation/serialization/Serializer.java]
 as my test case. I just noticed that the current impl is lacking.

To accomplish what you suggest (i.e. keeping the Schema hidden internally), 
what we would need to do is provide a name/alias for each schema, and be able 
to save it and invoke it (internally should be ok) by name. Perhaps something 
like this in the Adapter:

{code}
Adapter schema(String name, TypeReference type);  // No path means the 
top level
Adapter schema(String name, String path, TypeReference type);  // Path 
indicates the object somewhere in the graph
{code}

When "saving" the schema, only the name would be serialized. If we don't do 
this, I found that, especially for small objects, there is way too much 
output/noise. Using the "alias" as the value to serialize works very nicely.

Also, the above rule would allow for multiple schemas. I just noticed a bug in 
my impl: right now you can only save one single schema, which is not useful.

{quote}
For decoding the story is probably different, as you need to be able to pass in 
the schema as context of the decode operation.
{quote}

My solution was to ensure that you use the same Adapter, which is already 
configured, for deserialisation. The Schema is already saved under an alias. 
Example:

{code}
{"schema":"Schema1", "payload":{ ... }}
{code}

The additional chars serialized is quite minimal, and the input is easy to 
parse. Since there is a Map in the Adapter that looks up the schema by name (in 
this case "Schema1"), then deserialization is quite easy. At least, that is 
what I tested, and it seems to work very well.

{quote}
This could potentially be done via configadmin too, but that would be awkward I 
think. Maybe if we add a method to the Decoding interface to provide 
context/schema it might be useful, so you could do 
mySpecialCodec.decode(sometextfile).withSchema(mySchema).to(MyDTO.class)
{quote}

Yes, that would also work!

However, if we work under the assumption that deserialization does not happen 
without serialization, and since we (must??) also control the serialization, 
then this stuff can be hidden internally, I think.

{quote}
Just an idea. I think it would be great if we could make the API such that 
special implementations like your schema-based one work within the general 
API...
{quote}

Great ideas! Thanks for supporting this. I personally think it is not only very 
important stuff, but *should* be part of the Converter/Codec.

I'll continue working a bit more with Prevayler so I can show you what I come 
up with in terms of a client-facing interface. So far, it is turning out to be 
quite elegant, actually. (At least from the outside!) :-)

> Serializer
> --
>
> Key: FELIX-5332
> URL: 

[jira] [Commented] (FELIX-5332) Serializer

2016-09-05 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464738#comment-15464738
 ] 

David Bosschaert commented on FELIX-5332:
-

Hi [~dleangen], thanks for this. I can't say I fully digested your code just 
yet but I was wondering... To me, the serializer/schematizer sounds a bit like 
a special Codec to me. For example if you used it with XML it could produce an 
XML schema as part of the serialization and then do the deserialization 
together with the schema. 
It would be nice if the codec (or other API) could be such that it would 
support such implementations. 

To produce a schema, this might just be done as a by-product of the encoding 
process? Maybe additionally configured via ConfigAdmin configuration factories? 
Do we need anything extra in the codec API for this?

For decoding the story is probably different, as you need to be able to pass in 
the schema as context of the decode operation. This could potentially be done 
via configadmin too, but that would be awkward I think. Maybe if we add a 
method to the Decoding interface to provide context/schema it might be useful, 
so you could do 
  mySpecialCodec.decode(sometextfile).withSchema(mySchema).to(MyDTO.class)

Just an idea. I think it would be great if we could make the API such that 
special implementations like your schema-based one work within the general 
API...

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-09-04 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462397#comment-15462397
 ] 

David Leangen commented on FELIX-5332:
--

I have done some experimenting. Overall conclusion: possible, but much more 
"complicated" than I had expected.

Related code is 
[here|https://github.com/dleangen/felix/tree/serializer/converter].

I did not want to interfere with the current code base, so I created a few 
additional services that work in conjunction with Converter and Codec. In order 
to maintain the same structure and style, I literally copied the Converter and 
Codec to Schematizer and Serializer, respectively, and refactored from there. 
In the end, I'm not sure if that was a great idea. The code is not as elegant 
as I’d like. But it’s good enough for my current needs, and also as a proof of 
concept to better communicate my thoughts.

Here are a few of the insights I gained while going through this discovery 
process.

In a nutshell, I found that there are a few different roles.

 - Converter: converts an object of one type into an object of another type

 - Codec: encodes an object to some form (i.e. String), and decodes back into 
the object

 - 
[Schematizer|https://github.com/dleangen/felix/tree/serializer/converter/converter/src/main/java/org/apache/felix/schematizer/impl]:
 creates the schema based on the static structure of the DTO

 - 
[Serializer|https://github.com/dleangen/felix/tree/serializer/converter/codec/src/main/java/org/apache/felix/serializer/impl/json]:
 serializes an object (i.e. to String), and using the schema deserializes back 
to object (mostly the same as Codec, but uses the schema for proper typing)

The Schematizer can schematize on its own, but in order for it to have any 
meaning, it needs to be used together with the Serializer, and the Serializer 
needs to do conversion. So in the end, the Converter, Codec, Schematizer, and 
Serializer all need to come together. For that reason, keeping them separated 
as they are now may not be ideal. But integrating them would likely require 
updates to the API, and I don't know if that's possible. There may be a smart 
way to somehow integrate the functionality without a major API update. That 
would be great, but I'm not seeing it right now. Maybe after a few days away 
from the code I could look at it again with fresh eyes...

One observation I had: the Converter acts kind of like an interpretation engine 
for a DTO, while the Schematizer acts more like a compiler. The Converter does 
a “real time” conversion, while the Schematizer reads the static structure and 
keeps a record for later. The advantage of the Converter (relating to this 
aspect only) is that you do not need to keep anything in memory. The advantage 
of the Schematizer is that it can be used for deserialisation, which depending 
on the object structure may not be (easily) possible without a memory of the 
original object graph.

The Converter is much more complex. It can act on DTOs, Maps, and interfaces. I 
limited the Schematizer to act only on DTOs. It can provide a Map 
representation, but that’s it. If ever this becomes interesting to somebody, I 
suppose that flexibility could always be added later.

The Schematizer can produce a serialisable version of the DTO schema. I was 
considering making a quick and dirty prototype for [JSON 
Schema|http://json-schema.org/], but there is a bit more complexity in there 
than I am willing to handle right now, so I just did a simple “custom” JSON 
version of a schema. It merely reflects the state of the Schema object output 
by the Schematizer. I suppose that it could be possible to implement with 
JXPath instead, but that seemed a bit heavy handed to me. The purpose here is 
to specialise for DTOs, and to try to keep it lightweight and free of 
dependencies. That said, the code is more complex than I think it needs to be. 
I'm sure it could be simplified quite a bit, but this is quite a complex puzzle 
with many moving parts.

Limitations + things to consider:

 * Only works with objects that extend DTO
 * Not thoroughly tested
 * Not well integrated with Converter / Codec
 * Perhaps Adapter is not necessary (move everything to Schematizer directly)
 * Probably much more :-)

The Schematizer could eventually be used to output JSON Schema, DTDs, or XML 
Schema etc. I did not implement that.

It could also be used to introspect a DTO, for instance if an object needs to 
be split up or joined when working with a persistence framework. I have not yet 
implemented that, either.


Look forward to hearing your thoughts about which direction to take with this.

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
>   

[jira] [Commented] (FELIX-5332) Serializer

2016-08-24 Thread David Bosschaert (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435136#comment-15435136
 ] 

David Bosschaert commented on FELIX-5332:
-

Hi [~dleangen], I just reopened the issue since you are thinking of 
experimenting more :)

Couple of thoughts from my side...

Regarding configuration of these things. I personally still think a Codec may 
play a role here (but am happy to be proved wrong). On the configuration side 
of it, maybe we can use a factory configuration with ConfigAdmin where you can 
specify alternative instances of the codec and look them up via a custom 
mime-type, e.g. for your one with extra metadata you could define your own mime 
type: x-application/json+metadata or something like that. Then you'd have 
multiple service instances available at runtime and you'd use the one that you 
want where you want it, so no single global configuration... I will play a bit 
with this idea and try to work it out.

Just another thought on looking up the classes on the deserialization side. 
Class.forName("classname") is probably not what you want to do because that 
will not work if this class is provided by a bundle in your system. It might be 
better to include the Bundle SymbolicName and version in the metadata, then you 
can look up this bundle when reconstructing the object and load the required 
class directly from it...

Looking forward to your experiments :)

David

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-08-24 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435109#comment-15435109
 ] 

David Leangen commented on FELIX-5332:
--

Hmmm. Maybe I should not have acted so quickly.

I am finding that this is more difficult than I'd thought because of generic 
types. If there is a generic type nested deep in the structure, it gets 
deserialised as a Map, which causes a ClassCastException in the client code. 
This leads me to believe that the _metadata information may be necessary for 
all generic types.

I'll experiment a bit more.

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (FELIX-5332) Serializer

2016-08-24 Thread David Leangen (JIRA)

[ 
https://issues.apache.org/jira/browse/FELIX-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435023#comment-15435023
 ] 

David Leangen commented on FELIX-5332:
--

On second thought, probably this is not a common enough use case to warrant the 
complication. On top of that, it does not seem all that difficult to do. So I 
don't think it's worth including this after all.

Sorry for the noise.

Will close this issue.

> Serializer
> --
>
> Key: FELIX-5332
> URL: https://issues.apache.org/jira/browse/FELIX-5332
> Project: Felix
>  Issue Type: New Feature
>  Components: Converter
>Reporter: David Leangen
> Attachments: diff-serializer.txt
>
>
> Test case and a bit of code to show how a Serializer could work.
> To work as a Serializer, the type information needs to be embedded into the 
> output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)