Re: Adding type awareness in Camel route

[Claus Ibsen]

Hi

So the PR from Tomo was a bit overwhealming.
I have talked with Tomo to do this in smaller steps so we can all
follow where this is taking Camel.

At first step is something like

- type awareness to route level (similar to rest-dsl binding)
- example in xml
- example in java
- this feature is totally opt-on
- allow to define input and output as just "xml" "json" or "java" etc
to just indicate its one of these kind, but maybe without reference to
a xml schema / namespace etc / ditto for json. But being able to just
tell the Camel developer that this is a route that receives XML and
returns JSon is a good win. This allows developers to easier maintain
this code in the future to be able to see this.

And then step 2 can be
- can we reuse/align this with rest-dsl binding
- feedback from community, rinse and repeat
- jmx and some other apis to expose this for tooling / management

And for following steps
- should we have type awareness on processor level also? And if so we
need some non invasive general hook to perform this, so we dont have
to scatter code around. However this is a bridge to cross in the
future.





On Mon, Nov 7, 2016 at 4:06 AM, Tomohisa Igarashi  wrote:
> Hi,
>
> I filed a JIRA for this one:
> https://issues.apache.org/jira/browse/CAMEL-10447
>
> and added a camel-example-transformer with some bug fixes:
> https://github.com/igarashitm/camel/tree/CAMEL-10447
> https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-02f2b5457a9643a58d973ef7e971d405
> https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-b64bb94ed2382215eef0598fc79aa9bf
>
> I'll submit a PR once I finish the document bits. I haven't integrated with
> existing data type awareness on REST DSL yet, but I believe that won't be
> too complex. Anyway that is a next thing to do.
>
> Thanks,
> Tomo
>
>
> On 10/26/2016 10:52 PM, Tomohisa Igarashi wrote:
>>
>> Hi Claus,
>>
>> Thanks for the feedback!
>>
>> On 10/26/2016 09:39 PM, Claus Ibsen wrote:
>>>
>>> On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
>>>  wrote:

 Hi,

 I'd like to resume the discussion about this. I still think 3.0 would be
 the
 best target to get this feature fully supported, but to achieve it in
 better
 shape, I'd like to have it in 2.19 as well as an experimental, ask
 feedback
 and then reflect those for 3.0 full support. Fortunately this is purely
 an
 addition to existing features, i.e. is not breaking any existing API.
 What
 do you think?

>>>
>>> Having community feedback would be great.
>>>
>>> I wonder if you would mind creating a little example in the examples
>>> that uses this new feature. Then its easier for users to look at and
>>> better understand what its about.
>>
>>
>> OK sure, I'll add an example for this feature.
>>
>>> Also when we do DSL changes/additions we tend to write unit tests in
>>> camel-core, and then reuse those tests in camel-spring where we extend
>>> the unit test from camel-core-test and then provide the routes in XML
>>> DSL instead of java code. Then the unit test is the same and ensures
>>> the DSL works in both worlds (Java and XML). I am not sure if you have
>>> this in your commits?
>>>
>>> You can take a look at the existing tests in camel-spring to find some
>>> examples of this practice.
>>
>>
>> Yep, actually SpringTransformerTest inherits test methods from
>> TransformerTest with that practice :)
>> I'll add more unit tests in addition to that.
>>
>>> Down the road we do have some kind of typeawareness with the rest-dsl
>>> where you can also specify in and output types which the swagger
>>> api-doc uses. And it does some automatic binding to/from
>>> pojo/xml/json. So we may have some overlap here, and may ponder if
>>> there is something we can eventually make use the new typeware stuff
>>> or meet in the middle or something. However maybe its better to take a
>>> look at that after the other stuff is more ready.
>>
>>
>> That sounds like a thing I need to look into, at least need to understand
>> what is available right now. Hopefully there's a good meeting point with it.
>>
>>> Also in Java DSL you may want to allow to specify a java type using
>>> the Class as type instead of having to type the FQN in a String. But
>>> mind in XML DSL this is not possible where you specify a String
>>> attribute for the type.
>>
>>
>> Good idea, I'll add that too!
>>
 Thanks,
 Tomo


 On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
>
>
> Hi Claus,
>
> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to
> be
> targeted than 2.x as this introduces some schema updates.
>
> Including this one, I'm always looking for the chance to make any
> contribution to Camel. If there's anything I can help please let me
> know.
>
> Thanks,
> Tomo
>
> On 

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi,

I filed a JIRA for this one:
https://issues.apache.org/jira/browse/CAMEL-10447

and added a camel-example-transformer with some bug fixes:
https://github.com/igarashitm/camel/tree/CAMEL-10447
https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-02f2b5457a9643a58d973ef7e971d405
https://github.com/igarashitm/camel/commit/1c9bb6d8f23e0d96e942161f8ffc821845d99c59#diff-b64bb94ed2382215eef0598fc79aa9bf

I'll submit a PR once I finish the document bits. I haven't integrated with 
existing data type awareness on REST DSL yet, but I believe that won't be too 
complex. Anyway that is a next thing to do.

Thanks,
Tomo

On 10/26/2016 10:52 PM, Tomohisa Igarashi wrote:

Hi Claus,

Thanks for the feedback!

On 10/26/2016 09:39 PM, Claus Ibsen wrote:

On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
 wrote:

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the
best target to get this feature fully supported, but to achieve it in better
shape, I'd like to have it in 2.19 as well as an experimental, ask feedback
and then reflect those for 3.0 full support. Fortunately this is purely an
addition to existing features, i.e. is not breaking any existing API. What
do you think?



Having community feedback would be great.

I wonder if you would mind creating a little example in the examples
that uses this new feature. Then its easier for users to look at and
better understand what its about.


OK sure, I'll add an example for this feature.


Also when we do DSL changes/additions we tend to write unit tests in
camel-core, and then reuse those tests in camel-spring where we extend
the unit test from camel-core-test and then provide the routes in XML
DSL instead of java code. Then the unit test is the same and ensures
the DSL works in both worlds (Java and XML). I am not sure if you have
this in your commits?

You can take a look at the existing tests in camel-spring to find some
examples of this practice.


Yep, actually SpringTransformerTest inherits test methods from TransformerTest 
with that practice :)
I'll add more unit tests in addition to that.


Down the road we do have some kind of typeawareness with the rest-dsl
where you can also specify in and output types which the swagger
api-doc uses. And it does some automatic binding to/from
pojo/xml/json. So we may have some overlap here, and may ponder if
there is something we can eventually make use the new typeware stuff
or meet in the middle or something. However maybe its better to take a
look at that after the other stuff is more ready.


That sounds like a thing I need to look into, at least need to understand what 
is available right now. Hopefully there's a good meeting point with it.


Also in Java DSL you may want to allow to specify a java type using
the Class as type instead of having to type the FQN in a String. But
mind in XML DSL this is not possible where you specify a String
attribute for the type.


Good idea, I'll add that too!


Thanks,
Tomo


On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:


Hi Claus,

Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be
targeted than 2.x as this introduces some schema updates.

Including this one, I'm always looking for the chance to make any
contribution to Camel. If there's anything I can help please let me know.

Thanks,
Tomo

On 09/17/2016 06:33 PM, Claus Ibsen wrote:


Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
 wrote:


Hi Camel developers,

I'd like to propose an enhancement on handling data types of Camel
message
contents. To start a smooth discussion I implemented the idea first:
https://github.com/igarashitm/camel/tree/contract-based-type-awareness

And these testcases demonstrates the declarative transformer usage
according
to the declared data types:
[Java DSL]

https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
[Spring DSL]

https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534

This adds input/output content type declaration on from and all other
processors. It also introduces well-known Exchange properties,
INPUT_TYPE
and OUTPUT_TYPE which are used to specify the current message content
type.
The data type is URN like string starts with scheme, like
java:org.example.ItemA or xml:{org.example.xml}ItemA.

If the content type is declared via inputType/outputType/contract, the
ContractProcessor wraps the actual processor and process

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi Claus,

Thanks for the feedback!

On 10/26/2016 09:39 PM, Claus Ibsen wrote:

On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
 wrote:

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the
best target to get this feature fully supported, but to achieve it in better
shape, I'd like to have it in 2.19 as well as an experimental, ask feedback
and then reflect those for 3.0 full support. Fortunately this is purely an
addition to existing features, i.e. is not breaking any existing API. What
do you think?



Having community feedback would be great.

I wonder if you would mind creating a little example in the examples
that uses this new feature. Then its easier for users to look at and
better understand what its about.


OK sure, I'll add an example for this feature.


Also when we do DSL changes/additions we tend to write unit tests in
camel-core, and then reuse those tests in camel-spring where we extend
the unit test from camel-core-test and then provide the routes in XML
DSL instead of java code. Then the unit test is the same and ensures
the DSL works in both worlds (Java and XML). I am not sure if you have
this in your commits?

You can take a look at the existing tests in camel-spring to find some
examples of this practice.


Yep, actually SpringTransformerTest inherits test methods from TransformerTest 
with that practice :)
I'll add more unit tests in addition to that.


Down the road we do have some kind of typeawareness with the rest-dsl
where you can also specify in and output types which the swagger
api-doc uses. And it does some automatic binding to/from
pojo/xml/json. So we may have some overlap here, and may ponder if
there is something we can eventually make use the new typeware stuff
or meet in the middle or something. However maybe its better to take a
look at that after the other stuff is more ready.


That sounds like a thing I need to look into, at least need to understand what 
is available right now. Hopefully there's a good meeting point with it.


Also in Java DSL you may want to allow to specify a java type using
the Class as type instead of having to type the FQN in a String. But
mind in XML DSL this is not possible where you specify a String
attribute for the type.


Good idea, I'll add that too!


Thanks,
Tomo


On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:


Hi Claus,

Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be
targeted than 2.x as this introduces some schema updates.

Including this one, I'm always looking for the chance to make any
contribution to Camel. If there's anything I can help please let me know.

Thanks,
Tomo

On 09/17/2016 06:33 PM, Claus Ibsen wrote:


Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
 wrote:


Hi Camel developers,

I'd like to propose an enhancement on handling data types of Camel
message
contents. To start a smooth discussion I implemented the idea first:
https://github.com/igarashitm/camel/tree/contract-based-type-awareness

And these testcases demonstrates the declarative transformer usage
according
to the declared data types:
[Java DSL]

https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
[Spring DSL]

https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534

This adds input/output content type declaration on from and all other
processors. It also introduces well-known Exchange properties,
INPUT_TYPE
and OUTPUT_TYPE which are used to specify the current message content
type.
The data type is URN like string starts with scheme, like
java:org.example.ItemA or xml:{org.example.xml}ItemA.

If the content type is declared via inputType/outputType/contract, the
ContractProcessor wraps the actual processor and process
transformation/validation according to the type, say if INPUT_TYPE
exchange
property has xml:{org.example.xml}ItemA and the declared inputType is
xml:{org.example.xml}ItemB, then it transforms
xml:{org.example.xml}ItemA
content into xml:{org.example.xml}ItemB. The  element
which is
introduced right under the  is the one to declare the
mappings
between transformer implementation and those from/to data type. I
implemented only transformer first, but validator would be brought in in
a
same way. This way allows users to make data types visible in the route
definition and keep the transformation/validation apart from route
definition itself.

The most important thing is that the ContractProcessor is involved only
when

Re: Adding type awareness in Camel route

[Claus Ibsen]

On Mon, Oct 17, 2016 at 4:35 AM, Tomohisa Igarashi
 wrote:
> Hi,
>
> I'd like to resume the discussion about this. I still think 3.0 would be the
> best target to get this feature fully supported, but to achieve it in better
> shape, I'd like to have it in 2.19 as well as an experimental, ask feedback
> and then reflect those for 3.0 full support. Fortunately this is purely an
> addition to existing features, i.e. is not breaking any existing API. What
> do you think?
>

Having community feedback would be great.

I wonder if you would mind creating a little example in the examples
that uses this new feature. Then its easier for users to look at and
better understand what its about.

Also when we do DSL changes/additions we tend to write unit tests in
camel-core, and then reuse those tests in camel-spring where we extend
the unit test from camel-core-test and then provide the routes in XML
DSL instead of java code. Then the unit test is the same and ensures
the DSL works in both worlds (Java and XML). I am not sure if you have
this in your commits?

You can take a look at the existing tests in camel-spring to find some
examples of this practice.

Down the road we do have some kind of typeawareness with the rest-dsl
where you can also specify in and output types which the swagger
api-doc uses. And it does some automatic binding to/from
pojo/xml/json. So we may have some overlap here, and may ponder if
there is something we can eventually make use the new typeware stuff
or meet in the middle or something. However maybe its better to take a
look at that after the other stuff is more ready.

Also in Java DSL you may want to allow to specify a java type using
the Class as type instead of having to type the FQN in a String. But
mind in XML DSL this is not possible where you specify a String
attribute for the type.









> Thanks,
> Tomo
>
>
> On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
>>
>> Hi Claus,
>>
>> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be
>> targeted than 2.x as this introduces some schema updates.
>>
>> Including this one, I'm always looking for the chance to make any
>> contribution to Camel. If there's anything I can help please let me know.
>>
>> Thanks,
>> Tomo
>>
>> On 09/17/2016 06:33 PM, Claus Ibsen wrote:
>>>
>>> Hey
>>>
>>> Can we take this discussion post Camel 2.18 release.
>>>
>>> We are working on the last details to get it ready, and its our main
>>> focus to get this new release out.
>>>
>>> After this release we will pickup talks about the next releases
>>> whether that is 2.19 or 3.0, and for the latter what the broad goals
>>> of that is. What you talk about seems more of a 3.0 candidate to me,
>>> than on 2.x.
>>>
>>>
>>>
>>> On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
>>>  wrote:

 Hi Camel developers,

 I'd like to propose an enhancement on handling data types of Camel
 message
 contents. To start a smooth discussion I implemented the idea first:
 https://github.com/igarashitm/camel/tree/contract-based-type-awareness

 And these testcases demonstrates the declarative transformer usage
 according
 to the declared data types:
 [Java DSL]

 https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
 [Spring DSL]

 https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534

 This adds input/output content type declaration on from and all other
 processors. It also introduces well-known Exchange properties,
 INPUT_TYPE
 and OUTPUT_TYPE which are used to specify the current message content
 type.
 The data type is URN like string starts with scheme, like
 java:org.example.ItemA or xml:{org.example.xml}ItemA.

 If the content type is declared via inputType/outputType/contract, the
 ContractProcessor wraps the actual processor and process
 transformation/validation according to the type, say if INPUT_TYPE
 exchange
 property has xml:{org.example.xml}ItemA and the declared inputType is
 xml:{org.example.xml}ItemB, then it transforms
 xml:{org.example.xml}ItemA
 content into xml:{org.example.xml}ItemB. The  element
 which is
 introduced right under the  is the one to declare the
 mappings
 between transformer implementation and those from/to data type. I
 implemented only transformer first, but validator would be brought in in
 a
 same way. This way allows users to make data types visible in the route
 definition and keep the transformation/validation apart from route
 definition itself.

 The most important thing is that the ContractProcessor is involved only
 when
 content type is explicitly declared in a route definition, so that it
 never
 breaks existing camel routes. Ofcourse 

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi,

On 10/18/2016 10:31 PM, Siano, Stephan wrote:

Hi,

Well, you aim quite high.

So the idea is to annotate each endpoint with an input and an output data 
format and then do the mapping between these types automatically via the type 
converter registry. Did I get this right?


Yes that direction, but not only TypeConverter as it just performs a Java->Java 
transformation. With leveraging transformation endpoints and POJO as well, it enables 
Any->Any transformation. That's why I needed to introduce  into 
CamelContext. These 2 testcases demonstrate it.
[Java DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
[Spring DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534


In this case it might be a useful feature, but I would guess it is probably 
very hard to implement.


Yes I went back and forth again and again in camel-core in depth. It eventually 
worked :)

Thanks,
Tomo


Best regards
Stephan

-Original Message-
From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com]
Sent: Montag, 17. Oktober 2016 16:51
To: dev@camel.apache.org
Subject: Re: Adding type awareness in Camel route

Hi,

On 10/17/2016 03:32 PM, Siano, Stephan wrote:

Hi,

I am not sure if I really understand the concept behind this. Is there any text 
about the general ideas behind these changes?


Good question, here is the original proposal my colleague made. "1) Extend the 
DSL" is what I implemented.
https://github.com/kcbabo/sandbox/blob/master/camel-metadata.md


If I got this right, you propose that endpoints declare a kind of data type 
that can be verified as input and output types (like a specific XML schema) and 
the camel route will automatically validate this constraint. Did I get this 
right?


Validation is also one of the things type declaration enables, I don't yet implement 
though. I implemented declarative transformation first. The type declaration specifies 
"how content should look like at this point", and if a transformer which 
transforms from current data type to that declared data type exists, it is triggered. The 
current data type is transfered as an Exchange property, if it's not set then Java type 
of IN message body is used.

The type awareness benefits all of 3 endpoint types you categorized:


Concerning this payload classification we have actually three kinds of 
endpoints in camel:
1. Endpoints that do not care about data content at all (e.g. a file)


Let's say if there's from("file:dir").inputType("xml:{ws.example.com}aaaInput").to("cxf://ws.example.com/bbb") and 
a transformer "byte[] -> xml:{ws.example.com}aaa" is declared, that transformer will be picked up and triggered. Not yet implemented 
though, cxf endpoint would be able to declare its input/output automatically from WSDL so that the "xml:{ws.example.com}aaaInput -> 
xml:{ws.example.com/bbbInput}" transformer is triggered. Maybe we can even introduce a kind of "type parser" which user can plugin 
on file consumer so that the file content type is recognized at first place?


2. Endpoints that do care about data content, however the specific content type 
definition may be part of the endpoint configuration. An example for this might 
be a CXF web service endpoint with a WSDL. The endpoint as such only demands 
that the payload is some kind of XML but the WSDL may (or may not e.g. if the 
message definition is xsd:any) put some additional restriction on the payload 
data.


This is a good example actually, CXF consumer would be able to set current 
input type and expected output type automatically from WSDL. Including producer 
side I mentioned in #1, that would be the 2nd step to improve this type 
awareness stuff.


3. Endpoints that enforce some kind of (usually technical) format on the 
endpoint (data types are usually of that kind. If you are trying a zipfile 
unmarshal, you'd better have a zip file as a payload...).


Just an idea yet though, type name "zip:xml:{ws.example.com}aaaInput" may imply the 
content is zip compressed single text file, which contains "xml:{ws.example.com}aaaInput" 
type of XML.


Are you focusing any of these kinds of endpoint types or are you just focusing 
some special processors that operate on very specific data?


The answer is all of them, plus not only for endpoint, but for arbitrary 
processors!

Does it make sense?

Thanks,
Tomo



Best regards
Stephan

-Original Message-
From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com]
Sent: Montag, 17. Oktober 2016 04:36
To: dev@camel.apache.org
Subject: Re: Adding type awareness in Camel route

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the 
best target to get this feature fully supported, but to achieve it in better 
shape, I'd like to ha

RE: Adding type awareness in Camel route

[Siano, Stephan]

Hi,

Well, you aim quite high. 

So the idea is to annotate each endpoint with an input and an output data 
format and then do the mapping between these types automatically via the type 
converter registry. Did I get this right?

In this case it might be a useful feature, but I would guess it is probably 
very hard to implement.

Best regards
Stephan

-Original Message-
From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com] 
Sent: Montag, 17. Oktober 2016 16:51
To: dev@camel.apache.org
Subject: Re: Adding type awareness in Camel route

Hi,

On 10/17/2016 03:32 PM, Siano, Stephan wrote:
> Hi,
>
> I am not sure if I really understand the concept behind this. Is there any 
> text about the general ideas behind these changes?

Good question, here is the original proposal my colleague made. "1) Extend the 
DSL" is what I implemented.
https://github.com/kcbabo/sandbox/blob/master/camel-metadata.md

> If I got this right, you propose that endpoints declare a kind of data type 
> that can be verified as input and output types (like a specific XML schema) 
> and the camel route will automatically validate this constraint. Did I get 
> this right?

Validation is also one of the things type declaration enables, I don't yet 
implement though. I implemented declarative transformation first. The type 
declaration specifies "how content should look like at this point", and if a 
transformer which transforms from current data type to that declared data type 
exists, it is triggered. The current data type is transfered as an Exchange 
property, if it's not set then Java type of IN message body is used.

The type awareness benefits all of 3 endpoint types you categorized:

> Concerning this payload classification we have actually three kinds of 
> endpoints in camel:
> 1. Endpoints that do not care about data content at all (e.g. a file)

Let's say if there's 
from("file:dir").inputType("xml:{ws.example.com}aaaInput").to("cxf://ws.example.com/bbb")
 and a transformer "byte[] -> xml:{ws.example.com}aaa" is declared, that 
transformer will be picked up and triggered. Not yet implemented though, cxf 
endpoint would be able to declare its input/output automatically from WSDL so 
that the "xml:{ws.example.com}aaaInput -> xml:{ws.example.com/bbbInput}" 
transformer is triggered. Maybe we can even introduce a kind of "type parser" 
which user can plugin on file consumer so that the file content type is 
recognized at first place?

> 2. Endpoints that do care about data content, however the specific content 
> type definition may be part of the endpoint configuration. An example for 
> this might be a CXF web service endpoint with a WSDL. The endpoint as such 
> only demands that the payload is some kind of XML but the WSDL may (or may 
> not e.g. if the message definition is xsd:any) put some additional 
> restriction on the payload data.

This is a good example actually, CXF consumer would be able to set current 
input type and expected output type automatically from WSDL. Including producer 
side I mentioned in #1, that would be the 2nd step to improve this type 
awareness stuff.

> 3. Endpoints that enforce some kind of (usually technical) format on the 
> endpoint (data types are usually of that kind. If you are trying a zipfile 
> unmarshal, you'd better have a zip file as a payload...).

Just an idea yet though, type name "zip:xml:{ws.example.com}aaaInput" may imply 
the content is zip compressed single text file, which contains 
"xml:{ws.example.com}aaaInput" type of XML.

> Are you focusing any of these kinds of endpoint types or are you just 
> focusing some special processors that operate on very specific data?

The answer is all of them, plus not only for endpoint, but for arbitrary 
processors!

Does it make sense?

Thanks,
Tomo


> Best regards
> Stephan
>
> -----Original Message-
> From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com]
> Sent: Montag, 17. Oktober 2016 04:36
> To: dev@camel.apache.org
> Subject: Re: Adding type awareness in Camel route
>
> Hi,
>
> I'd like to resume the discussion about this. I still think 3.0 would be the 
> best target to get this feature fully supported, but to achieve it in better 
> shape, I'd like to have it in 2.19 as well as an experimental, ask feedback 
> and then reflect those for 3.0 full support. Fortunately this is purely an 
> addition to existing features, i.e. is not breaking any existing API. What do 
> you think?
>
> Thanks,
> Tomo
>
> On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
>> Hi Claus,
>>
>> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be 
>> targeted than 2.x as this introduces some schema updates.
>>
>> Including this one, I'm always l

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi,

On 10/17/2016 03:32 PM, Siano, Stephan wrote:

Hi,

I am not sure if I really understand the concept behind this. Is there any text 
about the general ideas behind these changes?


Good question, here is the original proposal my colleague made. "1) Extend the 
DSL" is what I implemented.
https://github.com/kcbabo/sandbox/blob/master/camel-metadata.md


If I got this right, you propose that endpoints declare a kind of data type 
that can be verified as input and output types (like a specific XML schema) and 
the camel route will automatically validate this constraint. Did I get this 
right?


Validation is also one of the things type declaration enables, I don't yet implement 
though. I implemented declarative transformation first. The type declaration specifies 
"how content should look like at this point", and if a transformer which 
transforms from current data type to that declared data type exists, it is triggered. The 
current data type is transfered as an Exchange property, if it's not set then Java type 
of IN message body is used.

The type awareness benefits all of 3 endpoint types you categorized:


Concerning this payload classification we have actually three kinds of 
endpoints in camel:
1. Endpoints that do not care about data content at all (e.g. a file)


Let's say if there's from("file:dir").inputType("xml:{ws.example.com}aaaInput").to("cxf://ws.example.com/bbb") and 
a transformer "byte[] -> xml:{ws.example.com}aaa" is declared, that transformer will be picked up and triggered. Not yet implemented 
though, cxf endpoint would be able to declare its input/output automatically from WSDL so that the "xml:{ws.example.com}aaaInput -> 
xml:{ws.example.com/bbbInput}" transformer is triggered. Maybe we can even introduce a kind of "type parser" which user can plugin 
on file consumer so that the file content type is recognized at first place?


2. Endpoints that do care about data content, however the specific content type 
definition may be part of the endpoint configuration. An example for this might 
be a CXF web service endpoint with a WSDL. The endpoint as such only demands 
that the payload is some kind of XML but the WSDL may (or may not e.g. if the 
message definition is xsd:any) put some additional restriction on the payload 
data.


This is a good example actually, CXF consumer would be able to set current 
input type and expected output type automatically from WSDL. Including producer 
side I mentioned in #1, that would be the 2nd step to improve this type 
awareness stuff.


3. Endpoints that enforce some kind of (usually technical) format on the 
endpoint (data types are usually of that kind. If you are trying a zipfile 
unmarshal, you'd better have a zip file as a payload...).


Just an idea yet though, type name "zip:xml:{ws.example.com}aaaInput" may imply the 
content is zip compressed single text file, which contains "xml:{ws.example.com}aaaInput" 
type of XML.


Are you focusing any of these kinds of endpoint types or are you just focusing 
some special processors that operate on very specific data?


The answer is all of them, plus not only for endpoint, but for arbitrary 
processors!

Does it make sense?

Thanks,
Tomo



Best regards
Stephan

-Original Message-
From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com]
Sent: Montag, 17. Oktober 2016 04:36
To: dev@camel.apache.org
Subject: Re: Adding type awareness in Camel route

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the 
best target to get this feature fully supported, but to achieve it in better 
shape, I'd like to have it in 2.19 as well as an experimental, ask feedback and 
then reflect those for 3.0 full support. Fortunately this is purely an addition 
to existing features, i.e. is not breaking any existing API. What do you think?

Thanks,
Tomo

On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:

Hi Claus,

Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be 
targeted than 2.x as this introduces some schema updates.

Including this one, I'm always looking for the chance to make any contribution 
to Camel. If there's anything I can help please let me know.

Thanks,
Tomo

On 09/17/2016 06:33 PM, Claus Ibsen wrote:

Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
<tm.igara...@gmail.com> wrote:

Hi Camel developers,

I'd like to propose an enhancement on handling data types of Camel message
contents. To start a smooth discussion I implemented the idea first:
https://

RE: Adding type awareness in Camel route

[Siano, Stephan]

Hi,

I am not sure if I really understand the concept behind this. Is there any text 
about the general ideas behind these changes?

If I got this right, you propose that endpoints declare a kind of data type 
that can be verified as input and output types (like a specific XML schema) and 
the camel route will automatically validate this constraint. Did I get this 
right?

Concerning this payload classification we have actually three kinds of 
endpoints in camel:
1. Endpoints that do not care about data content at all (e.g. a file)
2. Endpoints that do care about data content, however the specific content type 
definition may be part of the endpoint configuration. An example for this might 
be a CXF web service endpoint with a WSDL. The endpoint as such only demands 
that the payload is some kind of XML but the WSDL may (or may not e.g. if the 
message definition is xsd:any) put some additional restriction on the payload 
data.
3. Endpoints that enforce some kind of (usually technical) format on the 
endpoint (data types are usually of that kind. If you are trying a zipfile 
unmarshal, you'd better have a zip file as a payload...).
Are you focusing any of these kinds of endpoint types or are you just focusing 
some special processors that operate on very specific data?

Best regards
Stephan

-Original Message-
From: Tomohisa Igarashi [mailto:tm.igara...@gmail.com] 
Sent: Montag, 17. Oktober 2016 04:36
To: dev@camel.apache.org
Subject: Re: Adding type awareness in Camel route

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the 
best target to get this feature fully supported, but to achieve it in better 
shape, I'd like to have it in 2.19 as well as an experimental, ask feedback and 
then reflect those for 3.0 full support. Fortunately this is purely an addition 
to existing features, i.e. is not breaking any existing API. What do you think?

Thanks,
Tomo

On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:
> Hi Claus,
>
> Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be 
> targeted than 2.x as this introduces some schema updates.
>
> Including this one, I'm always looking for the chance to make any 
> contribution to Camel. If there's anything I can help please let me know.
>
> Thanks,
> Tomo
>
> On 09/17/2016 06:33 PM, Claus Ibsen wrote:
>> Hey
>>
>> Can we take this discussion post Camel 2.18 release.
>>
>> We are working on the last details to get it ready, and its our main
>> focus to get this new release out.
>>
>> After this release we will pickup talks about the next releases
>> whether that is 2.19 or 3.0, and for the latter what the broad goals
>> of that is. What you talk about seems more of a 3.0 candidate to me,
>> than on 2.x.
>>
>>
>>
>> On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
>> <tm.igara...@gmail.com> wrote:
>>> Hi Camel developers,
>>>
>>> I'd like to propose an enhancement on handling data types of Camel message
>>> contents. To start a smooth discussion I implemented the idea first:
>>> https://github.com/igarashitm/camel/tree/contract-based-type-awareness
>>>
>>> And these testcases demonstrates the declarative transformer usage according
>>> to the declared data types:
>>> [Java DSL]
>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
>>> [Spring DSL]
>>> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534
>>>
>>> This adds input/output content type declaration on from and all other
>>> processors. It also introduces well-known Exchange properties, INPUT_TYPE
>>> and OUTPUT_TYPE which are used to specify the current message content type.
>>> The data type is URN like string starts with scheme, like
>>> java:org.example.ItemA or xml:{org.example.xml}ItemA.
>>>
>>> If the content type is declared via inputType/outputType/contract, the
>>> ContractProcessor wraps the actual processor and process
>>> transformation/validation according to the type, say if INPUT_TYPE exchange
>>> property has xml:{org.example.xml}ItemA and the declared inputType is
>>> xml:{org.example.xml}ItemB, then it transforms xml:{org.example.xml}ItemA
>>> content into xml:{org.example.xml}ItemB. The  element which is
>>> introduced right under the  is the one to declare the mappings
>>> between transformer implementation and those from/to data type. I
>>> implemented only transformer first, but validator would be brought in in a
>>> same way. This way allows users to make data types visible in the route
>>

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi,

I'd like to resume the discussion about this. I still think 3.0 would be the 
best target to get this feature fully supported, but to achieve it in better 
shape, I'd like to have it in 2.19 as well as an experimental, ask feedback and 
then reflect those for 3.0 full support. Fortunately this is purely an addition 
to existing features, i.e. is not breaking any existing API. What do you think?

Thanks,
Tomo

On 09/17/2016 10:20 PM, Tomohisa Igarashi wrote:

Hi Claus,

Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be 
targeted than 2.x as this introduces some schema updates.

Including this one, I'm always looking for the chance to make any contribution 
to Camel. If there's anything I can help please let me know.

Thanks,
Tomo

On 09/17/2016 06:33 PM, Claus Ibsen wrote:

Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
 wrote:

Hi Camel developers,

I'd like to propose an enhancement on handling data types of Camel message
contents. To start a smooth discussion I implemented the idea first:
https://github.com/igarashitm/camel/tree/contract-based-type-awareness

And these testcases demonstrates the declarative transformer usage according
to the declared data types:
[Java DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
[Spring DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534

This adds input/output content type declaration on from and all other
processors. It also introduces well-known Exchange properties, INPUT_TYPE
and OUTPUT_TYPE which are used to specify the current message content type.
The data type is URN like string starts with scheme, like
java:org.example.ItemA or xml:{org.example.xml}ItemA.

If the content type is declared via inputType/outputType/contract, the
ContractProcessor wraps the actual processor and process
transformation/validation according to the type, say if INPUT_TYPE exchange
property has xml:{org.example.xml}ItemA and the declared inputType is
xml:{org.example.xml}ItemB, then it transforms xml:{org.example.xml}ItemA
content into xml:{org.example.xml}ItemB. The  element which is
introduced right under the  is the one to declare the mappings
between transformer implementation and those from/to data type. I
implemented only transformer first, but validator would be brought in in a
same way. This way allows users to make data types visible in the route
definition and keep the transformation/validation apart from route
definition itself.

The most important thing is that the ContractProcessor is involved only when
content type is explicitly declared in a route definition, so that it never
breaks existing camel routes. Ofcourse programatic
transformations/validations we're doing today are still fully available.
It's purely an addition to the existing camel features.

Any thoughts? Does it sound acceptable to get merged into camel? ANY
feedback would be really appreciated!

Thanks,
Tomo

Re: Adding type awareness in Camel route

[Tomohisa Igarashi]

Hi Claus,

Thanks for the reply. Sure that's fine, I agree 3.0 would be better to be 
targeted than 2.x as this introduces some schema updates.

Including this one, I'm always looking for the chance to make any contribution 
to Camel. If there's anything I can help please let me know.

Thanks,
Tomo

On 09/17/2016 06:33 PM, Claus Ibsen wrote:

Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
 wrote:

Hi Camel developers,

I'd like to propose an enhancement on handling data types of Camel message
contents. To start a smooth discussion I implemented the idea first:
https://github.com/igarashitm/camel/tree/contract-based-type-awareness

And these testcases demonstrates the declarative transformer usage according
to the declared data types:
[Java DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
[Spring DSL]
https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534

This adds input/output content type declaration on from and all other
processors. It also introduces well-known Exchange properties, INPUT_TYPE
and OUTPUT_TYPE which are used to specify the current message content type.
The data type is URN like string starts with scheme, like
java:org.example.ItemA or xml:{org.example.xml}ItemA.

If the content type is declared via inputType/outputType/contract, the
ContractProcessor wraps the actual processor and process
transformation/validation according to the type, say if INPUT_TYPE exchange
property has xml:{org.example.xml}ItemA and the declared inputType is
xml:{org.example.xml}ItemB, then it transforms xml:{org.example.xml}ItemA
content into xml:{org.example.xml}ItemB. The  element which is
introduced right under the  is the one to declare the mappings
between transformer implementation and those from/to data type. I
implemented only transformer first, but validator would be brought in in a
same way. This way allows users to make data types visible in the route
definition and keep the transformation/validation apart from route
definition itself.

The most important thing is that the ContractProcessor is involved only when
content type is explicitly declared in a route definition, so that it never
breaks existing camel routes. Ofcourse programatic
transformations/validations we're doing today are still fully available.
It's purely an addition to the existing camel features.

Any thoughts? Does it sound acceptable to get merged into camel? ANY
feedback would be really appreciated!

Thanks,
Tomo

Re: Adding type awareness in Camel route

[Claus Ibsen]

Hey

Can we take this discussion post Camel 2.18 release.

We are working on the last details to get it ready, and its our main
focus to get this new release out.

After this release we will pickup talks about the next releases
whether that is 2.19 or 3.0, and for the latter what the broad goals
of that is. What you talk about seems more of a 3.0 candidate to me,
than on 2.x.



On Fri, Sep 16, 2016 at 6:50 AM, Tomohisa Igarashi
 wrote:
> Hi Camel developers,
>
> I'd like to propose an enhancement on handling data types of Camel message
> contents. To start a smooth discussion I implemented the idea first:
> https://github.com/igarashitm/camel/tree/contract-based-type-awareness
>
> And these testcases demonstrates the declarative transformer usage according
> to the declared data types:
> [Java DSL]
> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-c14a7e8e88a6e41492946e1537bfb1cf
> [Spring DSL]
> https://github.com/igarashitm/camel/commit/498c27d2ba99b04bbe7b90a93329d42b7c718a29#diff-b2506c84ddde91438fc6374039e21534
>
> This adds input/output content type declaration on from and all other
> processors. It also introduces well-known Exchange properties, INPUT_TYPE
> and OUTPUT_TYPE which are used to specify the current message content type.
> The data type is URN like string starts with scheme, like
> java:org.example.ItemA or xml:{org.example.xml}ItemA.
>
> If the content type is declared via inputType/outputType/contract, the
> ContractProcessor wraps the actual processor and process
> transformation/validation according to the type, say if INPUT_TYPE exchange
> property has xml:{org.example.xml}ItemA and the declared inputType is
> xml:{org.example.xml}ItemB, then it transforms xml:{org.example.xml}ItemA
> content into xml:{org.example.xml}ItemB. The  element which is
> introduced right under the  is the one to declare the mappings
> between transformer implementation and those from/to data type. I
> implemented only transformer first, but validator would be brought in in a
> same way. This way allows users to make data types visible in the route
> definition and keep the transformation/validation apart from route
> definition itself.
>
> The most important thing is that the ContractProcessor is involved only when
> content type is explicitly declared in a route definition, so that it never
> breaks existing camel routes. Ofcourse programatic
> transformations/validations we're doing today are still fully available.
> It's purely an addition to the existing camel features.
>
> Any thoughts? Does it sound acceptable to get merged into camel? ANY
> feedback would be really appreciated!
>
> Thanks,
> Tomo



-- 
Claus Ibsen
-
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2