Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-15 Thread Justin Leet 
I created a ticket, https://issues.apache.org/jira/browse/METRON-1363 to
track this.  I also noticed that we might be able to swipe from Nifi.
Looks like they generate a nice page for their stuff based on annotations
and we may be able to adapt it to what we have. I haven't looked at their
code at all yet, but feel free to check out:
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-documentation/src/main/java/org/apache/nifi/documentation

On Thu, Dec 14, 2017 at 6:21 PM, Matt Foley  wrote:

> +1 from me too, Justin.  Great idea.
>
>
> On 12/14/17, 12:44 PM, "zeo...@gmail.com"  wrote:
>
> A huge +1 from me.  This would be great
>
> On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > +1 from me, great idea Justin. I did a bit of digging around also
> and the
> > Doclet approach you're already using seems the way to go. I didn't
> come
> > across any libraries that would make this easier or better. Not sure
> if
> > Swagger has anything along these lines?
> >
> > On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > I think this is a great idea, and I looked at the POC and it isn’t
> as bad
> > > as you make it out to be;)
> > >
> > > What I would like to see is documentation for Stellar functions, by
> > > namespace generated. I would also
> > > like the capability to document at the namespace level.
> > >
> > > Often we have namespace level concepts that don’t fit into any
> given
> > > function’s documentation.
> > > Setting aside the how of the namespace documentation for a moment,
> based
> > on
> > > the POC I would
> > > suggest that we
> > >
> > > * find all namespaces
> > > * create a page per namespace
> > > * document each function in it’s namespace’s page
> > > * include the namespace doc in that page
> > >
> > > Each module that exports stellar function’s should have it’s own
> > > documentation.  As part of breaking stellar out to it’s own module
> > > we should remove stellar documentation from stellar common that
> applies
> > to
> > > functions outside that module.
> > >
> > >
> > >
> > > On December 14, 2017 at 14:32:56, Justin Leet (
> justinjl...@gmail.com)
> > > wrote:
> > >
> > > I think it would be valuable to have the documentation around
> Stellar
> > being
> > > autogenerated. We have most of the info we'd want in the @Stellar
> > > annotation, and ideally, we could just pull this info out and
> produce
> > some
> > > docs similar to what we already manually maintain. This came up a
> bit in
> > > the context of https://issues.apache.org/jira/browse/METRON-1361
> > >
> > > I put together a super, super (super!) rough POC of using the
> approach of
> > > Javadoc-style doclet processing that reads the annotations and
> kicks out
> > > something pretty close to the current docs (without any fancy
> stuff like
> > > the table of contents and so on).
> > >
> > > Right now, there'd be a good deal more to do that to make it
> usable. Off
> > > the top of my head, the main things I wanted to look at before
> really
> > even
> > > taking an actual stab at it are
> > >
> > > 1) abstracting out the markdown formatting from the annotation
> parsing
> > > 2) Making sure we can integrate this approach without breaking
> current
> > > Javadocs
> > > 3) Managing things across projects (since we put in Stellar
> functions all
> > > over).
> > > 4) Slightly more though about how we'd manage it.
> > >
> > > Otto's alluded to having a couple thoughts, and I'm more than
> happy to
> > get
> > > a better idea of what we want the end state to look like (either
> this or
> > > something else, e.g. an annotation processor during compile phase
> or if
> > > someone knows a tool that takes care of this sort of thing.)
> > >
> > > Any thoughts?
> > >
> >
> --
>
> Jon
>
>
>
>

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Matt Foley 
+1 from me too, Justin.  Great idea.


On 12/14/17, 12:44 PM, "zeo...@gmail.com"  wrote:

A huge +1 from me.  This would be great

On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 from me, great idea Justin. I did a bit of digging around also and the
> Doclet approach you're already using seems the way to go. I didn't come
> across any libraries that would make this easier or better. Not sure if
> Swagger has anything along these lines?
>
> On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler 
> wrote:
>
> > I think this is a great idea, and I looked at the POC and it isn’t as 
bad
> > as you make it out to be;)
> >
> > What I would like to see is documentation for Stellar functions, by
> > namespace generated. I would also
> > like the capability to document at the namespace level.
> >
> > Often we have namespace level concepts that don’t fit into any given
> > function’s documentation.
> > Setting aside the how of the namespace documentation for a moment, based
> on
> > the POC I would
> > suggest that we
> >
> > * find all namespaces
> > * create a page per namespace
> > * document each function in it’s namespace’s page
> > * include the namespace doc in that page
> >
> > Each module that exports stellar function’s should have it’s own
> > documentation.  As part of breaking stellar out to it’s own module
> > we should remove stellar documentation from stellar common that applies
> to
> > functions outside that module.
> >
> >
> >
> > On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > I think it would be valuable to have the documentation around Stellar
> being
> > autogenerated. We have most of the info we'd want in the @Stellar
> > annotation, and ideally, we could just pull this info out and produce
> some
> > docs similar to what we already manually maintain. This came up a bit in
> > the context of https://issues.apache.org/jira/browse/METRON-1361
> >
> > I put together a super, super (super!) rough POC of using the approach 
of
> > Javadoc-style doclet processing that reads the annotations and kicks out
> > something pretty close to the current docs (without any fancy stuff like
> > the table of contents and so on).
> >
> > Right now, there'd be a good deal more to do that to make it usable. Off
> > the top of my head, the main things I wanted to look at before really
> even
> > taking an actual stab at it are
> >
> > 1) abstracting out the markdown formatting from the annotation parsing
> > 2) Making sure we can integrate this approach without breaking current
> > Javadocs
> > 3) Managing things across projects (since we put in Stellar functions 
all
> > over).
> > 4) Slightly more though about how we'd manage it.
> >
> > Otto's alluded to having a couple thoughts, and I'm more than happy to
> get
> > a better idea of what we want the end state to look like (either this or
> > something else, e.g. an annotation processor during compile phase or if
> > someone knows a tool that takes care of this sort of thing.)
> >
> > Any thoughts?
> >
>
-- 

Jon

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread zeo...@gmail.com 
A huge +1 from me.  This would be great

On Thu, Dec 14, 2017 at 3:39 PM Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> +1 from me, great idea Justin. I did a bit of digging around also and the
> Doclet approach you're already using seems the way to go. I didn't come
> across any libraries that would make this easier or better. Not sure if
> Swagger has anything along these lines?
>
> On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler 
> wrote:
>
> > I think this is a great idea, and I looked at the POC and it isn’t as bad
> > as you make it out to be;)
> >
> > What I would like to see is documentation for Stellar functions, by
> > namespace generated. I would also
> > like the capability to document at the namespace level.
> >
> > Often we have namespace level concepts that don’t fit into any given
> > function’s documentation.
> > Setting aside the how of the namespace documentation for a moment, based
> on
> > the POC I would
> > suggest that we
> >
> > * find all namespaces
> > * create a page per namespace
> > * document each function in it’s namespace’s page
> > * include the namespace doc in that page
> >
> > Each module that exports stellar function’s should have it’s own
> > documentation.  As part of breaking stellar out to it’s own module
> > we should remove stellar documentation from stellar common that applies
> to
> > functions outside that module.
> >
> >
> >
> > On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com)
> > wrote:
> >
> > I think it would be valuable to have the documentation around Stellar
> being
> > autogenerated. We have most of the info we'd want in the @Stellar
> > annotation, and ideally, we could just pull this info out and produce
> some
> > docs similar to what we already manually maintain. This came up a bit in
> > the context of https://issues.apache.org/jira/browse/METRON-1361
> >
> > I put together a super, super (super!) rough POC of using the approach of
> > Javadoc-style doclet processing that reads the annotations and kicks out
> > something pretty close to the current docs (without any fancy stuff like
> > the table of contents and so on).
> >
> > Right now, there'd be a good deal more to do that to make it usable. Off
> > the top of my head, the main things I wanted to look at before really
> even
> > taking an actual stab at it are
> >
> > 1) abstracting out the markdown formatting from the annotation parsing
> > 2) Making sure we can integrate this approach without breaking current
> > Javadocs
> > 3) Managing things across projects (since we put in Stellar functions all
> > over).
> > 4) Slightly more though about how we'd manage it.
> >
> > Otto's alluded to having a couple thoughts, and I'm more than happy to
> get
> > a better idea of what we want the end state to look like (either this or
> > something else, e.g. an annotation processor during compile phase or if
> > someone knows a tool that takes care of this sort of thing.)
> >
> > Any thoughts?
> >
>
-- 

Jon

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Michael Miklavcic 
+1 from me, great idea Justin. I did a bit of digging around also and the
Doclet approach you're already using seems the way to go. I didn't come
across any libraries that would make this easier or better. Not sure if
Swagger has anything along these lines?

On Thu, Dec 14, 2017 at 1:00 PM, Otto Fowler 
wrote:

> I think this is a great idea, and I looked at the POC and it isn’t as bad
> as you make it out to be;)
>
> What I would like to see is documentation for Stellar functions, by
> namespace generated. I would also
> like the capability to document at the namespace level.
>
> Often we have namespace level concepts that don’t fit into any given
> function’s documentation.
> Setting aside the how of the namespace documentation for a moment, based on
> the POC I would
> suggest that we
>
> * find all namespaces
> * create a page per namespace
> * document each function in it’s namespace’s page
> * include the namespace doc in that page
>
> Each module that exports stellar function’s should have it’s own
> documentation.  As part of breaking stellar out to it’s own module
> we should remove stellar documentation from stellar common that applies to
> functions outside that module.
>
>
>
> On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> I think it would be valuable to have the documentation around Stellar being
> autogenerated. We have most of the info we'd want in the @Stellar
> annotation, and ideally, we could just pull this info out and produce some
> docs similar to what we already manually maintain. This came up a bit in
> the context of https://issues.apache.org/jira/browse/METRON-1361
>
> I put together a super, super (super!) rough POC of using the approach of
> Javadoc-style doclet processing that reads the annotations and kicks out
> something pretty close to the current docs (without any fancy stuff like
> the table of contents and so on).
>
> Right now, there'd be a good deal more to do that to make it usable. Off
> the top of my head, the main things I wanted to look at before really even
> taking an actual stab at it are
>
> 1) abstracting out the markdown formatting from the annotation parsing
> 2) Making sure we can integrate this approach without breaking current
> Javadocs
> 3) Managing things across projects (since we put in Stellar functions all
> over).
> 4) Slightly more though about how we'd manage it.
>
> Otto's alluded to having a couple thoughts, and I'm more than happy to get
> a better idea of what we want the end state to look like (either this or
> something else, e.g. an annotation processor during compile phase or if
> someone knows a tool that takes care of this sort of thing.)
>
> Any thoughts?
>

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Otto Fowler 
I think this is a great idea, and I looked at the POC and it isn’t as bad
as you make it out to be;)

What I would like to see is documentation for Stellar functions, by
namespace generated. I would also
like the capability to document at the namespace level.

Often we have namespace level concepts that don’t fit into any given
function’s documentation.
Setting aside the how of the namespace documentation for a moment, based on
the POC I would
suggest that we

* find all namespaces
* create a page per namespace
* document each function in it’s namespace’s page
* include the namespace doc in that page

Each module that exports stellar function’s should have it’s own
documentation.  As part of breaking stellar out to it’s own module
we should remove stellar documentation from stellar common that applies to
functions outside that module.



On December 14, 2017 at 14:32:56, Justin Leet (justinjl...@gmail.com) wrote:

I think it would be valuable to have the documentation around Stellar being
autogenerated. We have most of the info we'd want in the @Stellar
annotation, and ideally, we could just pull this info out and produce some
docs similar to what we already manually maintain. This came up a bit in
the context of https://issues.apache.org/jira/browse/METRON-1361

I put together a super, super (super!) rough POC of using the approach of
Javadoc-style doclet processing that reads the annotations and kicks out
something pretty close to the current docs (without any fancy stuff like
the table of contents and so on).

Right now, there'd be a good deal more to do that to make it usable. Off
the top of my head, the main things I wanted to look at before really even
taking an actual stab at it are

1) abstracting out the markdown formatting from the annotation parsing
2) Making sure we can integrate this approach without breaking current
Javadocs
3) Managing things across projects (since we put in Stellar functions all
over).
4) Slightly more though about how we'd manage it.

Otto's alluded to having a couple thoughts, and I'm more than happy to get
a better idea of what we want the end state to look like (either this or
something else, e.g. an annotation processor during compile phase or if
someone knows a tool that takes care of this sort of thing.)

Any thoughts?

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Casey Stella 
chiming in with a +1 on my end too.  This would be fantastic.

On Thu, Dec 14, 2017 at 2:51 PM, Nick Allen  wrote:

> +1 I think it is a great idea, Justin and the only way that we'll keep the
> docs in-sync with the code.
>
>
>
>
>
> On Thu, Dec 14, 2017 at 2:32 PM Justin Leet  wrote:
>
> > I think it would be valuable to have the documentation around Stellar
> being
> > autogenerated.  We have most of the info we'd want in the @Stellar
> > annotation, and ideally, we could just pull this info out and produce
> some
> > docs similar to what we already manually maintain.  This came up a bit in
> > the context of https://issues.apache.org/jira/browse/METRON-1361
> >
> > I put together a super, super (super!) rough POC of using the approach of
> > Javadoc-style doclet processing that reads the annotations and kicks out
> > something pretty close to the current docs (without any fancy stuff like
> > the table of contents and so on).
> >
> > Right now, there'd be a good deal more to do that to make it usable.  Off
> > the top of my head, the main things I wanted to look at before really
> even
> > taking an actual stab at it are
> >
> > 1) abstracting out the markdown formatting from the annotation parsing
> > 2) Making sure we can integrate this approach without breaking current
> > Javadocs
> > 3) Managing things across projects (since we put in Stellar functions all
> > over).
> > 4) Slightly more though about how we'd manage it.
> >
> > Otto's alluded to having a couple thoughts, and I'm more than happy to
> get
> > a better idea of what we want the end state to look like (either this or
> > something else, e.g. an annotation processor during compile phase or if
> > someone knows a tool that takes care of this sort of thing.)
> >
> > Any thoughts?
> >
>

Re: [DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Nick Allen 
+1 I think it is a great idea, Justin and the only way that we'll keep the
docs in-sync with the code.





On Thu, Dec 14, 2017 at 2:32 PM Justin Leet  wrote:

> I think it would be valuable to have the documentation around Stellar being
> autogenerated.  We have most of the info we'd want in the @Stellar
> annotation, and ideally, we could just pull this info out and produce some
> docs similar to what we already manually maintain.  This came up a bit in
> the context of https://issues.apache.org/jira/browse/METRON-1361
>
> I put together a super, super (super!) rough POC of using the approach of
> Javadoc-style doclet processing that reads the annotations and kicks out
> something pretty close to the current docs (without any fancy stuff like
> the table of contents and so on).
>
> Right now, there'd be a good deal more to do that to make it usable.  Off
> the top of my head, the main things I wanted to look at before really even
> taking an actual stab at it are
>
> 1) abstracting out the markdown formatting from the annotation parsing
> 2) Making sure we can integrate this approach without breaking current
> Javadocs
> 3) Managing things across projects (since we put in Stellar functions all
> over).
> 4) Slightly more though about how we'd manage it.
>
> Otto's alluded to having a couple thoughts, and I'm more than happy to get
> a better idea of what we want the end state to look like (either this or
> something else, e.g. an annotation processor during compile phase or if
> someone knows a tool that takes care of this sort of thing.)
>
> Any thoughts?
>

[DISCUSS] Stellar Documentation Autogeneration

2017-12-14 Thread Justin Leet 
I think it would be valuable to have the documentation around Stellar being
autogenerated.  We have most of the info we'd want in the @Stellar
annotation, and ideally, we could just pull this info out and produce some
docs similar to what we already manually maintain.  This came up a bit in
the context of https://issues.apache.org/jira/browse/METRON-1361

I put together a super, super (super!) rough POC of using the approach of
Javadoc-style doclet processing that reads the annotations and kicks out
something pretty close to the current docs (without any fancy stuff like
the table of contents and so on).

Right now, there'd be a good deal more to do that to make it usable.  Off
the top of my head, the main things I wanted to look at before really even
taking an actual stab at it are

1) abstracting out the markdown formatting from the annotation parsing
2) Making sure we can integrate this approach without breaking current
Javadocs
3) Managing things across projects (since we put in Stellar functions all
over).
4) Slightly more though about how we'd manage it.

Otto's alluded to having a couple thoughts, and I'm more than happy to get
a better idea of what we want the end state to look like (either this or
something else, e.g. an annotation processor during compile phase or if
someone knows a tool that takes care of this sort of thing.)

Any thoughts?