Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Michael,

On 6 March 2015 at 14:42, michael kapelko korn...@gmail.com wrote:

 First, I'm all for new #pragma(tic) approach, because I use uniforms for
 the same reason currently.Thanks for an easier way to do the same thing!
 Second, I'd like to clarify that I understand it correctly: #pragma is OSG
 specific and thus works with OpenGL2, right?


It should work with all versions of GLSL, so OpenGL 2, 3.x, 4.x and OpenGL
ES 2 and 3, as the #pragma feature of GLSL that the new shader composition
leverage has been their since the inception of GLSL.

Which rather begs the question why did it take so long to spot this
approach... funny how coming at a problem afresh can provide new ways of
thinking about problems and how to solve them.   Happy to just have finally
got their... :-)

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[michael kapelko]

Hi.
First, I'm all for new #pragma(tic) approach, because I use uniforms for
the same reason currently.Thanks for an easier way to do the same thing!
Second, I'd like to clarify that I understand it correctly: #pragma is OSG
specific and thus works with OpenGL2, right?
Thanks.

2015-03-05 18:39 GMT+07:00 Robert Osfield robert.osfi...@gmail.com:

 Hi Robert (Miharcic for those trying to follow this thread),

 I have reviewed your shader composition changes.  My first reaction is
 WOW.  Second reaction much the same ;-)

 I can't pretend that I actually understand all the changes but have
 started to get inkling about parts.  The range of changes and complexity of
 what problem is being tackled is pretty daunting to take in.  I get the
 sense that your changes pick up where my original work on
 osg::ShaderComposition left off, coming much close to completing the
 journey.

 Right now my focus is getting OSG-3.4 out the door and given the changes
 are so extensive and the problem so complex I won't have time to look at
 integrating the changes.  The general approach I took with
 osg::ShaderComposition is something I no longer feel provides a good
 tradeoff between functionality and ease of use, here I feel the tradeoff
 with #pragma(tic) shader composition is much better - you don't get all
 functionality of osg::ShaderComposition but it is far easier for the
 average developer to pick and use.  Longer term I feel that features of the
 old osg::ShaderComposition would be best step by step integrated into a
 method that extends the #pragmat(tic) shader approach.

 If there are small modifications to the core OSG in svn/trunk that would
 help make it easier to add your own shader composition work ontop of the
 OSG-3.4 let me know.  Things like making a method virtual, or adding an
 extra option etc. are things that would not be too intrusive and risky
 w.r.t the OSG-3.4 release.

 Cheers
 Robert.




 ___
 osg-users mailing list
 osg-users@lists.openscenegraph.org
 http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Robert (Miharcic for those trying to follow this thread),

I have reviewed your shader composition changes.  My first reaction is
WOW.  Second reaction much the same ;-)

I can't pretend that I actually understand all the changes but have started
to get inkling about parts.  The range of changes and complexity of what
problem is being tackled is pretty daunting to take in.  I get the sense
that your changes pick up where my original work on osg::ShaderComposition
left off, coming much close to completing the journey.

Right now my focus is getting OSG-3.4 out the door and given the changes
are so extensive and the problem so complex I won't have time to look at
integrating the changes.  The general approach I took with
osg::ShaderComposition is something I no longer feel provides a good
tradeoff between functionality and ease of use, here I feel the tradeoff
with #pragma(tic) shader composition is much better - you don't get all
functionality of osg::ShaderComposition but it is far easier for the
average developer to pick and use.  Longer term I feel that features of the
old osg::ShaderComposition would be best step by step integrated into a
method that extends the #pragmat(tic) shader approach.

If there are small modifications to the core OSG in svn/trunk that would
help make it easier to add your own shader composition work ontop of the
OSG-3.4 let me know.  Things like making a method virtual, or adding an
extra option etc. are things that would not be too intrusive and risky
w.r.t the OSG-3.4 release.

Cheers
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Robert,

On 20 February 2015 at 11:27, Robert Milharcic robert.milhar...@ib-caddy.si
 wrote:

 One could keep together related stuff, like lighting uniforms for lighting
 define, textures and uniforms if it happens to be texture define... Then,
 at the apply, one could for example multiply uniform with state's model
 view matrix or use any other information that osg::State has to offer. For
 example:

 class MaterialShaderDefine : public ShaderDefine
 {
 ...

};


This would no longer be a shader define, it'd be a whole separate
infrastructure.  Shader define's are low level constructs like Uniforms
that provide a way of passing in data and controlling shaders.

What you are proposing is high level management of shader defines and
uniforms together.




 With old shader composer it is possible to do this by sub-classing
 ShaderAttribute. I also think that with ShaderDefine the define API would
 then be more consitent with other StateSet APIs:

 stateset-addUniform(Uniform*, StateAttribute::OverrideValue)
 stateset-setAttribute(StateAttribute*, StateAttribute::OverrideValue)
 stateset-setDefine(ShaderDefine*, StateAttribute::OverrideValue)

 BTW, I'm not suggesting here that ShaderDefine* should be internally keyed
 by pointer. Anyway, ShaderDefine is just an idea to be considered and
 nothing more.


I believe what you are looking forward is a way of wrapping up each the
setUniform/setAttribute/setDefine()'s required via a single object.  It
might be that having a ShaderDefine class might be appropriate for
replacing the current using of std::pair's and std::maps, but it's rather
different beast than this combine state management.

It could be that ShaderAttribute/ShaderComponet still have a roll to play
in this high level management.




  2. The greatest strength of old shader composer is
 ShaderComposer::getOrCreateProgram(). As others have already mentioned,
 this is the point where we used to gain control over the program
 composition. I'm personally using this control point for things like
 program-addBindAttribLocation/addBindFragDataLocation/
 addBindUniformBlock
 and for some other sanity checks. It would be great if we can somehow
 install a callback or overload some member to regain the control of the
 program composition.


 Perhaps osg::Program itself could be adapted to allow it to be easily
 subclassed so that the shader selection, compilation and linking and
 uniform/attribute setup could be managed by end users that wish to
 subclass
 from osg::Program.  Another approach would be to defer this work to a
 helper object that gets assigned to osg::State.  Perhaps the
 PerContexProgram/PerContextShader classes could have a role here in some
 way.

 What I'd really need is a concrete example in front of me that illustrates
 what functionality is needed but isn't possible.  An example based on the
 old osg::ShaderComposition would be fine.

  Unfortunately, I don't have short example that can illustrate what isn't
 possible. I'm attaching a complex example that shows old shader composer in
 action. The sources are for SVN revision 13962. Also, OSG_EXPERIMENTAL must
 be enabled in cmake. The point of interest is probably
 examples/osgshadercomposerhelper/ShaderComposer.cpp.


I have just unpacked the zip file, but am not sure yet what I'm going to be
looking at.  These are mods that provide what functionality, what should I
be expecting the example to do?

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Milharcic]

Hi Robert,


I have just unpacked the zip file, but am not sure yet what I'm going to be
looking at.  These are mods that provide what functionality, what should I
be expecting the example to do?


The idea here is to load any osg supported node file and then convert all the 
FFP StateAttributes to shaders. For example:

osg::ref_ptrosg::Node node = osgDB::readNodeFiles(arguments);

and then with ...

osgEx::ShaderComposerHelper sch;
 sch.apply(node.get());

... you can convert all FFP attributes found in scene to shaders. Then the 
shader composition must be enabled:

osgViewer::ViewerBase::Windows windows;
viewer.getWindows(windows);
for(osgViewer::ViewerBase::Windows::iterator itr = windows.begin();
itr != windows.end();
++itr)
{
(*itr)-getState()-setShaderCompositionEnabled(true);
(*itr)-getState()-setUseModelViewAndProjectionUniforms(true);
(*itr)-getState()-setUseVertexAttributeAliasing(true);
(*itr)-getState()-setShaderComposer(new osgEx::ShaderComposer);
}

.. and at this point all you have are one or more shader programs and then the 
viewer can be run as usual.

At the time, I wasn't able to find node file that containing all the various 
FFP attributes, so the example is a bit crowded with the code for the test 
scene .

If you are trying to compile the example, then you will first need to checkout 
SVN revision 13962 and then overwrite the files and enable OSG_EXPERIMENTAL 
option in cmake. You can find complete FFP emulation shader in 
examples/osgshadercomposerhelper/ShaderInjections.cpp.glsl, which is the main 
source to feed the shader composer.

Robert Milharcic
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Robert,

Thanks for the explanation, this helps lots.  I can't check it out right
away as I'm tying up the displacement mapping changes.  I will hopefully
find some quiet time this weekend or next week to have a tinker.

Cheers,
Robert.

On 20 February 2015 at 13:00, Robert Milharcic robert.milhar...@ib-caddy.si
 wrote:

 Hi Robert,

  I have just unpacked the zip file, but am not sure yet what I'm going to
 be
 looking at.  These are mods that provide what functionality, what should I
 be expecting the example to do?


 The idea here is to load any osg supported node file and then convert all
 the FFP StateAttributes to shaders. For example:

 osg::ref_ptrosg::Node node = osgDB::readNodeFiles(arguments);

 and then with ...

 osgEx::ShaderComposerHelper sch;
  sch.apply(node.get());

 ... you can convert all FFP attributes found in scene to shaders. Then the
 shader composition must be enabled:

 osgViewer::ViewerBase::Windows windows;
 viewer.getWindows(windows);
 for(osgViewer::ViewerBase::Windows::iterator itr = windows.begin();
 itr != windows.end();
 ++itr)
 {
 (*itr)-getState()-setShaderCompositionEnabled(true);
 (*itr)-getState()-setUseModelViewAndProjectionUniforms(true);
 (*itr)-getState()-setUseVertexAttributeAliasing(true);
 (*itr)-getState()-setShaderComposer(new osgEx::ShaderComposer);
 }

 .. and at this point all you have are one or more shader programs and then
 the viewer can be run as usual.

 At the time, I wasn't able to find node file that containing all the
 various FFP attributes, so the example is a bit crowded with the code for
 the test scene .

 If you are trying to compile the example, then you will first need to
 checkout SVN revision 13962 and then overwrite the files and enable
 OSG_EXPERIMENTAL option in cmake. You can find complete FFP emulation
 shader in examples/osgshadercomposerhelper/ShaderInjections.cpp.glsl,
 which is the main source to feed the shader composer.


 Robert Milharcic
 ___
 osg-users mailing list
 osg-users@lists.openscenegraph.org
 http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Milharcic]

Hi Robert,

I was amazed by the simplicity of the new pragmatic shader composition - 
but yet it is so powerful. Well done! So, I was making good progress 
porting old shader composition code to pragmatic one until I hit the 
wall. The problem is, I don't see any obvious way to extend the current 
pragmatic shader composition API. If I may, I would suggest two things 
for consideration:


1. Add a new layer of abstraction for StateSet's define API, lets say 
class object ShaderDefine that we can subclass. The ShaderDefine (or any 
other suitable name) would contain std:string _defineName, std::string 
_defineValue and at least a less operator, since you are inserting 
definition into map, and virtual void apply(State) const {}. Of course, 
the apply would then be called from State::applyDefineList() giving the 
user an opportunity for define's customization. So the new 
StateSet::setDefine() would look something like this: 
setDefine(ShaderDefine*, StateAttribute::OverrideValue);. Also, with the 
proposed abstraction it would be easier to write serialization support.


2. The greatest strength of old shader composer is 
ShaderComposer::getOrCreateProgram(). As others have already mentioned, 
this is the point where we used to gain control over the program 
composition. I'm personally using this control point for things like 
program-addBindAttribLocation/addBindFragDataLocation/addBindUniformBlock 
and for some other sanity checks. It would be great if we can somehow 
install a callback or overload some member to regain the control of the 
program composition.


Robert Milharcic
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Hi,

Hi Robert,

I was amazed by the simplicity of the new pragmatic shader composition 
- but yet it is so powerful. Well done! So, I was making good progress 
porting old shader composition code to pragmatic one until I hit the 
wall. The problem is, I don't see any obvious way to extend the 
current pragmatic shader composition API. If I may, I would suggest 
two things for consideration:


1. Add a new layer of abstraction for StateSet's define API, lets say 
class object ShaderDefine that we can subclass. The ShaderDefine (or 
any other suitable name) would contain std:string _defineName, 
std::string _defineValue and at least a less operator, since you are 
inserting definition into map, and virtual void apply(State) const 
{}. Of course, the apply would then be called from 
State::applyDefineList() giving the user an opportunity for define's 
customization. So the new StateSet::setDefine() would look something 
like this: setDefine(ShaderDefine*, StateAttribute::OverrideValue);. 
Also, with the proposed abstraction it would be easier to write 
serialization support.


2. The greatest strength of old shader composer is 
ShaderComposer::getOrCreateProgram(). As others have already 
mentioned, this is the point where we used to gain control over the 
program composition. I'm personally using this control point for 
things like 
program-addBindAttribLocation/addBindFragDataLocation/addBindUniformBlock 
and for some other sanity checks. It would be great if we can somehow 
install a callback or overload some member to regain the control of 
the program composition.
I second this, as I've just stumbled upon the same use case. Maybe there 
is some other way around this, but I too need some control over the 
finally created program.





Robert Milharcic
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Wojciech Lewandowski]

Hi, Robert,

I am not actively working with any code which would require shader
composition at the moment but I saw few such efforts in various projects in
the past, so I am really glad OSG is going to implement own scheme and
users will not need to invent the wheel anymore ;-). As someone only mildly
interested in the topic at the moment I only briefly looked at new shadow
composition example, and have only one question.

I noticed that we may add/override parts of shader code using  setDefine(
USER_FUNC(args) ) as in your example:

  stateset-setDefine(VERTEX_FUNC(v) , vec4(v.x, v.y, v.z *
sin(osg_SimulationTime), v.w));

But I feel that this solution may be inadequate for large blocks of shader
code which used would want linked instead of being effectively merged with
parent Ubershader program. Do you have a method or anticipate that such
method can be added, which would allow to add or override whole shaders in
effective Program applied to OpenGL context ? Something that
ShaderAttribute was supposed to do in former ShaderComposition scheme ? For
example whould this be possible to replace whole lighting.vert shader from
your example with entirely different shader doing own lighting at some
subnode and its StateSet ?

Cheers and Thank you for your Effort,
Wojtek Lewandowski



2015-02-19 11:22 GMT+01:00 Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de:

 Hi,

 Hi Robert,

 I was amazed by the simplicity of the new pragmatic shader composition -
 but yet it is so powerful. Well done! So, I was making good progress
 porting old shader composition code to pragmatic one until I hit the wall.
 The problem is, I don't see any obvious way to extend the current pragmatic
 shader composition API. If I may, I would suggest two things for
 consideration:

 1. Add a new layer of abstraction for StateSet's define API, lets say
 class object ShaderDefine that we can subclass. The ShaderDefine (or any
 other suitable name) would contain std:string _defineName, std::string
 _defineValue and at least a less operator, since you are inserting
 definition into map, and virtual void apply(State) const {}. Of course,
 the apply would then be called from State::applyDefineList() giving the
 user an opportunity for define's customization. So the new
 StateSet::setDefine() would look something like this:
 setDefine(ShaderDefine*, StateAttribute::OverrideValue);. Also, with the
 proposed abstraction it would be easier to write serialization support.

 2. The greatest strength of old shader composer is 
 ShaderComposer::getOrCreateProgram().
 As others have already mentioned, this is the point where we used to gain
 control over the program composition. I'm personally using this control
 point for things like program-addBindAttribLocation/
 addBindFragDataLocation/addBindUniformBlock and for some other sanity
 checks. It would be great if we can somehow install a callback or overload
 some member to regain the control of the program composition.

 I second this, as I've just stumbled upon the same use case. Maybe there
 is some other way around this, but I too need some control over the finally
 created program.




 Robert Milharcic
 ___
 osg-users mailing list
 osg-users@lists.openscenegraph.org
 http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


 ___
 osg-users mailing list
 osg-users@lists.openscenegraph.org
 http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Robert,

On 19 February 2015 at 10:19, Robert Milharcic robert.milhar...@ib-caddy.si
 wrote:

 1. Add a new layer of abstraction for StateSet's define API, lets say
 class object ShaderDefine that we can subclass. The ShaderDefine (or any
 other suitable name) would contain std:string _defineName, std::string
 _defineValue and at least a less operator, since you are inserting
 definition into map, and virtual void apply(State) const {}. Of course,
 the apply would then be called from State::applyDefineList() giving the
 user an opportunity for define's customization. So the new
 StateSet::setDefine() would look something like this:
 setDefine(ShaderDefine*, StateAttribute::OverrideValue);. Also, with the
 proposed abstraction it would be easier to write serialization support.


What would you be trying to do by sub-classing from ShaderDefine?

Currently shader defines have a std::string that maps to the string used in
the #pragma entry, and a std::string that maps to the value assigned to the
#define that will be generated.

One approach I considered when designing #pragma(tic) shader composition
was the use of a immutable osg::Token object and a factory that generates
unique Token objects for each std::string value used, so LIGHTING would
always map to a single osg::Token object that has a string Lighting.  The
use of these Token objects would speed up comparisons as it could be done
entirely on pointer comparisons.

Another possible variation I considered briefly was that overlap between
the way that uniforms and vertex attributes are applied - they all have
names associated with them.  Could one unify elements of defines with way
the interface one has for uniforms and vertex attributes?  I did spend long
enough to explore the possibilities, and as it'd likely require lots of
intrusive internal and API changes it's not something that was appropriate
pursue more than an observation of commonalities.

The solution I plumped of using a std::string name and value for each
define entry was driven by simplicity of API and implementation.  The
approach is not something I've seen before so just getting something
working to prove the concept or not was important.  In the end the approach
has been more powerful than I original envisaged.



 2. The greatest strength of old shader composer is
 ShaderComposer::getOrCreateProgram(). As others have already mentioned,
 this is the point where we used to gain control over the program
 composition. I'm personally using this control point for things like
 program-addBindAttribLocation/addBindFragDataLocation/addBindUniformBlock
 and for some other sanity checks. It would be great if we can somehow
 install a callback or overload some member to regain the control of the
 program composition.


Perhaps osg::Program itself could be adapted to allow it to be easily
subclassed so that the shader selection, compilation and linking and
uniform/attribute setup could be managed by end users that wish to subclass
from osg::Program.  Another approach would be to defer this work to a
helper object that gets assigned to osg::State.  Perhaps the
PerContexProgram/PerContextShader classes could have a role here in some
way.

What I'd really need is a concrete example in front of me that illustrates
what functionality is needed but isn't possible.  An example based on the
old osg::ShaderComposition would be fine.

I'm keen to find low impact ways of adding more power without complicating
the API and OSG backend.  So far #pragma(tic) shader composition is
straight forward to use for a range of usage cases so am confident that
it's got most things right.  It is just a first cut though, I don't feel
it's the final word on shader composition, I'm far more confident about it
than my previous attempts and solving this type of task.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

HI Wojtek,

On 19 February 2015 at 13:17, Wojciech Lewandowski 
w.p.lewandow...@gmail.com wrote:

 I am not actively working with any code which would require shader
 composition at the moment but I saw few such efforts in various projects in
 the past, so I am really glad OSG is going to implement own scheme and
 users will not need to invent the wheel anymore ;-). As someone only mildly
 interested in the topic at the moment I only briefly looked at new shadow
 composition example, and have only one question.

 I noticed that we may add/override parts of shader code using  setDefine(
 USER_FUNC(args) ) as in your example:

   stateset-setDefine(VERTEX_FUNC(v) , vec4(v.x, v.y, v.z *
 sin(osg_SimulationTime), v.w));

 But I feel that this solution may be inadequate for large blocks of shader
 code which used would want linked instead of being effectively merged with
 parent Ubershader program.


You should be able to multiple line insertion using the standard multi-line
#define line ending of \

However for general code sanity I don't think large blocks of code should
be inserted into shaders this way - just because it's possible doesn't mean
it's sensible.  What I'd do myself in situations like this would be the
have a separate shader that does the has functions that do bulk of the
work, and then have a define in the main that places the call the functions.


 Do you have a method or anticipate that such method can be added, which
 would allow to add or override whole shaders in effective Program applied
 to OpenGL context ? Something that ShaderAttribute was supposed to do in
 former ShaderComposition scheme ? For example whould this be possible to
 replace whole lighting.vert shader from your example with entirely
 different shader doing own lighting at some subnode and its StateSet ?


Right now #pragma(tic) shader composition can only select between all the
shaders attached to a single osg::Program, it can't insert/remove shaders
from elsewhere in the scene graph at runtime like the old
osg::ShaderCompostion was designed to do, or what VirtualProgram does.

However, if you do want to choose between different shaders for different
subgraphs what you can do is place all the shader possibilities in the
osg::Program that decorates the scene graph and then have the individual
shaders utilize the #pragma requires(..) functionality so that as you
provide the appropriate #defines in the different subgraphs then the
different shaders get linked to the GLSL program object relevant to that
subgraph.

The other approach is simply that have a separate osg::Program for the
subgraphs that want to specialize the shaders and that these local subgraph
osg::Program link in the osg::Shader they want for that subgraph.  This is
the very old way of course... crude but it's always there as a fallback
solution.

What I'm keen to do is have a solution that gets a lot of functionality for
a small amount of complexity.  Getting this balance right is hard.  The old
osg::ShaderComposition approach was in theory very powerful, but the
complexity associated with understanding and using it make it prohibitive
to most users.  It might be that we can simplify osg::ShaderComposition now
we have #pragma(tic) shader composition - for instance the whole code
injection side is effectively redundant.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Glenn Waldron]

On Wed, Feb 18, 2015 at 3:16 PM, Robert Osfield robert.osfi...@gmail.com
wrote:

 Hi Glenn et. al.

 Today I've been revisiting the osgTerrain::DisplacementMappingTechnique,
 GeometryPool and associated shaders to make better use of the new
 #pragma(tic) shader composition.  I haven't completed this work yet, but
 attached are the two main shaders I'm working on, these are where the bulk
 of the different options are for handling different numbers of colour
 layers, optional height fields etc. are.  I have been able to replace
 combine two original vertex main shaders into one, and four different
 fragment main shaders into one.I have attached the shaders to
 illustrate my work in progress.

 Previously the GeometryPool had to manage the selection between the
 different combinations of shaders and create a different osg::Program for
 each combination, while now I will be able to remove all this C++ code for
 management of shaders and replace with simply StateSet::setDefine(..) to
 toggle the various features on/off.  This toggle on/off of features can be
 generic - just needing to know the name of the feature to toggle, it won't
 need to know about GeometryPool or where it's attached different Programs.
 This is big advantage as it decouples the C++ implementation details from
 the interface.  The C++ implementation is also simpler so will be easier to
 understand and maintain going forward.

 Once I've got the DisplacementMapping/GeometryPool code sorted (hopefully
 by end of tomorrow) I'll check it all in and provide others a more complex
 illustration of how to use the new form shader composition.

 The work I did on osgVolume in 2013/14 managed it's own local management
 of different shader combinations - osg::ShaderComposition just didn't fit
 the problem I was trying to solve so ended up creating a local code to
 solve the problem.  However, this work would be an ideal candidate for
 converting to use #pragma(tic) shader composition, implementing it should
 be able to remove C++ code and make the shaders easier to manage.  Once
 I've done this it should also be an example that members of the community
 can use as inspiration.

 A note for Glenn, could you give me an example of what type of
 modifications to the attached shaders you'd like to add in - just sticking
 the GLSL code you want to add directly into the GLSL without any of the
 #ifdef etc. would be fine.  Knowing what type of GLSL you want to inject
 will help me know what entry points would be useful to add to the shaders
 to give you the ability to customize them.

 Cheers,
 Robert.


Well.. the possibilities really are endless. Today in osgEarth there are
probably 15+ VirtualProgram modules for terrain alone, plus various others,
and support for including custom user shaders (http://goo.gl/I51FHv).

What they have in common is a contract: a set of legal injection points and
function prototypes. None of these modules know about any of the others,
and the built-in main() functions don't know about any of these modules
beforehand. But as long as each one abides by the contract they will work
together.

Perhaps a contract-based approach can be built on top of the new #pragmatic
system... that is what I am wresting with at the moment... Maybe using
#pragmatic one could specify some pre-defined injection points. However it
isn't clear then how you would inject multiple functions and multiple
modules using the #define approach, or how you'd support overrides, etc.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Glenn,

On 19 February 2015 at 16:21, Glenn Waldron gwald...@gmail.com wrote:

 Well.. the possibilities really are endless. Today in osgEarth there are
 probably 15+ VirtualProgram modules for terrain alone, plus various others,
 and support for including custom user shaders (http://goo.gl/I51FHv).

 What they have in common is a contract: a set of legal injection points
 and function prototypes. None of these modules know about any of the
 others, and the built-in main() functions don't know about any of these
 modules beforehand. But as long as each one abides by the contract they
 will work together.

 Perhaps a contract-based approach can be built on top of the new
 #pragmatic system... that is what I am wresting with at the moment... Maybe
 using #pragmatic one could specify some pre-defined injection points.
 However it isn't clear then how you would inject multiple functions and
 multiple modules using the #define approach, or how you'd support
 overrides, etc.


One can place the #define's into the GLSL main()'s to serve as the
pre-defined injection points and then have these populated by the
StateSet::setDefine(..) methods.  StateSet::setDefine(..) fully supports
the OSG's established OVERRIDE/PROTECTED/ON/OFF functionality that exists
for Modes, StateAttribute and Uniforms.

One thing that isn't currently supported to pre-pending or appending define
values to #define's that already have a value, so if one provides a value
of the FS_END_OF_MAIN #define it can only take one value.  Potentially
would could add this ability but it'll complicate the API and
implementation so unless it's shown we really need it I'll not
implementation.

With having lots of different shader possibilities, adding them all to a
Program and only having the ones that are used linked (by utilizing the
#pragma requires(..) functionality) should be efficient.  However, since
you will now at scene graph creation time what shader combinations are
needed one should be able to just populate the topmost osg::Program with
the all the shaders that are required and then have then enabled/disabled
individually as required.

I am optimistic that you could probably replace the old VirtualProgram
approach in osgEarth with #pragma(tic) shader composition, but of course
this could only be done for users using OSG-3.4, which really doesn't help
you too much as it only exists in my future plans so far :-)   I don't
expect any issues with the two co-existing so it's not something we
probably need to worry about for a good while.

Cheers,
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Glenn,

On 17 February 2015 at 21:56, Glenn Waldron gwald...@gmail.com wrote:

 In my opinion the biggest problem that shader composition (SC) needs to
 address is the code maintenance nightmare of having to copy, paste, or edit
 other peoples' shaders in order to add your own functionality.


I don't we'll ever be able to get away from knowledge of other peoples
shaders, but if they are develop with adaptability in mind then extended
them is possible.  An example below.



 So I am eager to understand how you will address that. How can I add a new
 shader component without editing the main() functions to support it? For
 example, take the shaders in the terrain displacement code. I want to add
 some gamma correction code into the fragment stage. Will I be able to do
 that without editing your shaders too?


The way to tackle this would be to add entry points into the original
shader via defines and using this make it possible to add in the extra
features you want. For instance the fragment shader could look something
like:

#pragma import_defines(FS_BEFORE_MAIN, FS_START_OF_MAIN, FS_END_OF_MAIN)

uniform sampler2D baseTexture;
varying vec2 texcoord;

#ifdef FS_BEFORE_MAIN
FS_BEFORE_MAIN
#endif

void main(void)
{

#ifdef FS_START_OF_MAIN
FS_START_OF_MAIN
#endif


 gl_FragColor = texture2D( baseTexture, texcoord);


#ifdef FS_END_OF_MAIN
FS_END_OF_MAIN
#endif

}


So if you want to inject a some kind of processing you could do something
like:

stateset-setDefine(FS_BEFORE_MAIN,uniform blendColor;
stateset-setDefine(FS_END_OF_MAIN,gl_FragColor = mix(gl_FragColor,
blendColor, blendColor.a););

One could simplify the shader code further if one provided defaults of 
for the FS_* defines so you wouldn't need the #ifdefs. I have considered
adding support defaults support via #pragma
import_defines(FOR_PROVIDING_DEFAULTS=0), but have kept thing simple for
this first iteration.

There will be a limit to how can be done with shader composition without
replacing shaders wholesale, simple because complexity can quickly get out
of hand.  When it comes to debugging shaders being able to review them in
their entirety is ideal, as soon as you start optionally adding bits in
knowing what might be going on diminishes.  The use of defines like above
is preferable to trying to use generic code insertion (such as by using a
float as I did with osg::ShaderComposition) as even if you don't know what
the define will be substituted with you do at least know what order things
will be done in, and different blocks have specific meaning.

For osgTerrain displacement mapping it's possible to override the
GeometryPool::getProgram(..) so that you can subclass the pool and add in
your own osg::Program or add/substitute shaders into the program.  This
gives one the ability to do the high level substitution or utilize the
#pragma(tic) shader composition or a combination of the two.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,

On 17 February 2015 at 21:07, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:

 I'll update and test against my code.Don't hold your breath, though ;-)
 Anyways, thank you very much for this feature.
 If we manage to solve the other problems with this release (serialization
 and VPB) it might evolve towards a really great major one.
 Advocating for those, not wanting the old fashioned shader composition
 dying: I suspect I will run into problems as soon as I test the
 core-profile based stuff. In the old approach I do the BindFragLocation
 stuff based on the shader code parsed, which I cannot do anymore.But maybe
 there are ways around this.
 One idea popping into my head was to provide some createProgram callback
 inside the stateset to gain control over those aspects. This might remove
 the need for a ShaderComposer etc.

 Attached some example using merge (and changes avoiding GL_*).


I changed the GL_ yesterday evening and just testing out your example
modification with the merge everything works as before the addition of the
merge code so hopefully means my fixes yesterday are working as intended.

W.r.t serialization and VPB.  I haven't yet added support for serialization
for the StateSet define list - I'll add this once the functionality is a
little more widely tested.  I'm guessing you don't mean this though... If
not then these discussions are for other threads.  On the VPB side I
checked in one fix from the community and one myself which got VPB back
working on my system.  Again these things are for other threads.

W.r.t providing a StateSet::createProgram() this specifically wouldn't fit,
but the general idea is something I've been wondering about.  Rather than
coming up with an elaborate ShaderComposition framework you just pass on
the ability to customize Program, or collate Shader's in the scene graph
into osg::Program.  This isn't something I want to dive into right now
though as it's orthogonal to the #pragmatic shader composition
functionality.  And frankly I just want to get OSG-3.4 out the door sooner
rather than later.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Glenn et. al.

Today I've been revisiting the osgTerrain::DisplacementMappingTechnique,
GeometryPool and associated shaders to make better use of the new
#pragma(tic) shader composition.  I haven't completed this work yet, but
attached are the two main shaders I'm working on, these are where the bulk
of the different options are for handling different numbers of colour
layers, optional height fields etc. are.  I have been able to replace
combine two original vertex main shaders into one, and four different
fragment main shaders into one.I have attached the shaders to
illustrate my work in progress.

Previously the GeometryPool had to manage the selection between the
different combinations of shaders and create a different osg::Program for
each combination, while now I will be able to remove all this C++ code for
management of shaders and replace with simply StateSet::setDefine(..) to
toggle the various features on/off.  This toggle on/off of features can be
generic - just needing to know the name of the feature to toggle, it won't
need to know about GeometryPool or where it's attached different Programs.
This is big advantage as it decouples the C++ implementation details from
the interface.  The C++ implementation is also simpler so will be easier to
understand and maintain going forward.

Once I've got the DisplacementMapping/GeometryPool code sorted (hopefully
by end of tomorrow) I'll check it all in and provide others a more complex
illustration of how to use the new form shader composition.

The work I did on osgVolume in 2013/14 managed it's own local management of
different shader combinations - osg::ShaderComposition just didn't fit the
problem I was trying to solve so ended up creating a local code to solve
the problem.  However, this work would be an ideal candidate for converting
to use #pragma(tic) shader composition, implementing it should be able to
remove C++ code and make the shaders easier to manage.  Once I've done this
it should also be an example that members of the community can use as
inspiration.

A note for Glenn, could you give me an example of what type of
modifications to the attached shaders you'd like to add in - just sticking
the GLSL code you want to add directly into the GLSL without any of the
#ifdef etc. would be fine.  Knowing what type of GLSL you want to inject
will help me know what entry points would be useful to add to the shaders
to give you the ability to customize them.

Cheers,
Robert.


terrain_displacement_mapping.vert
Description: Binary data


terrain_displacement_mapping_CCC.frag
Description: Binary data
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,

On 17 February 2015 at 15:37, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:

  I've just refactored my previous ShaderComposition driven attempt to use
 the new pragmatic scheme. (It turned out to be relatively easy)


Good to hear the change was straight forward.



 At first glance the basics work fine, but I have some questions:



 1. When will the debug output in console be removed? ;-)


Once it a bit more tested by the community :-)




 2. I have some StateSet on top of a group representing my scene. Later
 during rendering a new element is added, but with some additional define.
 My naive expectation would be, that the new element will be using the
 top-level program, recompiled with the new define.
 Is this supposed to be working? What I want is to anonymously add some
 define for my subgraph, not caring which top-state it has been placed
 below.
 Or do I somehow need to dirty something here to get a new Program (based
 on the code of the program on top + my new define) built for this sub-graph?


The example does this - a leaf StateSet provides a new define and only the
subgraph below that StateSet gets affected by GLSL program that takes into
account that define.  If multiple subgraphs all independently enable the
same defines that they'll reuse the GLSL program.

Even if alter an define via an update/cull callback the define will used to
select the appropriate GLSL program during the draw traversal, and if the
current define combination is different than any previous combination for
the current osg::Program then a new GLSL shader and program objects will be
compiled and linked.

It should in theory just do what you ask it to do, no need to dirty
osg::Program or osg::Shaders.  One of the key goals is just make the
developers life simple, just do what you ask it to do and get out the way.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,

On 17 February 2015 at 16:33, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:


 It seems to act differently :-(.


It could be you've hit upon and bug, or perhaps just a mis-understanding of
how things are meant to behave.  I'd suspect the first possibility given
how new the functionality is.

Any chance you could reproduce the problem by modifying the
osgshadercomposition example in svn/trunk?



 The difference my case is the fact, that the element is added at some
 point in time. I expect this is after all previous used combinations have
 been built.
 Any idea where I could set a breakpoint to see if a new combination is
 created when I add the new define?


The work for tracking defines happens in
osg::State::pushDefineList/popDefineList and selecting program and shader
objects happen in Program header  .cpp and Shader header  .cpp
respectively.

In osg::State you'll find the following methods that can be called to check
the current state of defines, you'd need to call this between the push/pop
of the DefineList.

std::string getDefineString(const osg::ShaderDefines
shaderDefines);
bool supportsShaderRequirements(const osg::ShaderDefines
shaderRequirements);
bool supportsShaderRequirement(const std::string
shaderRequirement);


 Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Am 17.02.2015 um 17:13 schrieb Robert Osfield:

Hi Sebastian,

On 17 February 2015 at 15:37, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de 
mailto:sebastian.messerschm...@gmx.de wrote:


I've just refactored my previous ShaderComposition driven attempt
to use the new pragmatic scheme. (It turned out to be relatively easy)


Good to hear the change was straight forward.

At first glance the basics work fine, but I have some questions:

1. When will the debug output in console be removed? ;-)


Once it a bit more tested by the community :-)


2. I have some StateSet on top of a group representing my scene.
Later during rendering a new element is added, but with some
additional define.
My naive expectation would be, that the new element will be using
the top-level program, recompiled with the new define.
Is this supposed to be working? What I want is to anonymously add
some define for my subgraph, not caring which top-state it has
been placed below.
Or do I somehow need to dirty something here to get a new Program
(based on the code of the program on top + my new define) built
for this sub-graph?


The example does this - a leaf StateSet provides a new define and only 
the subgraph below that StateSet gets affected by GLSL program that 
takes into account that define.  If multiple subgraphs all 
independently enable the same defines that they'll reuse the GLSL 
program.


Even if alter an define via an update/cull callback the define will 
used to select the appropriate GLSL program during the draw traversal, 
and if the current define combination is different than any previous 
combination for the current osg::Program then a new GLSL shader and 
program objects will be compiled and linked.


It should in theory just do what you ask it to do, no need to dirty 
osg::Program or osg::Shaders.  One of the key goals is just make the 
developers life simple, just do what you ask it to do and get out the way.

It seems to act differently :-(.
The difference my case is the fact, that the element is added at some 
point in time. I expect this is after all previous used combinations 
have been built.
Any idea where I could set a breakpoint to see if a new combination is 
created when I add the new define?


Cheers
Sebastian


Robert.


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Hi Robert,

I've just refactored my previous ShaderComposition driven attempt to use 
the new pragmatic scheme. (It turned out to be relatively easy)

At first glance the basics work fine, but I have some questions:

1. When will the debug output in console be removed? ;-)

2. I have some StateSet on top of a group representing my scene. Later 
during rendering a new element is added, but with some additional define.
My naive expectation would be, that the new element will be using the 
top-level program, recompiled with the new define.
Is this supposed to be working? What I want is to anonymously add some 
define for my subgraph, not caring which top-state it has been placed 
below.
Or do I somehow need to dirty something here to get a new Program (based 
on the code of the program on top + my new define) built for this sub-graph?



Cheers
Sebastian


Hi All,

I have now checked in core OSG support for a new form of shader 
composition that leverages #pragma's to extended GLSL shaders to 
specify which #define's that it requires from the scene graph. The 
scene graph passes these to the shaders via StateSet::setDefine(..) 
and provides full support for the OSG's state override/protected options.


I have modified the osgshadercomposition example to illustrate a very 
simple example that utlizes three shaders checked into 
OpenSceneGraph-Data/shaders trunk.  So on the code look for 
OpenSceneGraph/examples/osgshadercomposition.cpp and on the shader 
from look at OpenSceneGraph-Data/shaders/osgshadercompostion.vert, 
osgshadercomposition.frag and lighting.vert.


I will write more later about how to use this new capability.  It's 
gotta be one of the coolest features added to the OSG for quite a few 
years :-)


Cheers,
Robert.


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,


On 17 February 2015 at 20:19, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:

  Indeed i did :-)
 The problem is within the StateSet::merge function. The case in which i
 had the problem used StateSet::merge instead of directly setting the state.
 Reinjecting the defines after merging solves the problem and after looking
 into the merge code, I suspect the error here.


Thanks for the pinpointing the problem.  The lack of State::merge() support
for the new DefineList was certainly a bug through complete absence :-)

I have now added this, but don't have any code that uses merge() to hand to
test, could you do an svn update and let me know if this fixes the bug.

Cheers,
Robert
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Hi Robert,

I've encountered some small problem in the shader composition example.
You are using GL_LIGHTING which per defintion is not allowed for macros.
Some glsl compilers will ignore this, but on my Quadro card it produces 
an error:


0(1) : error C0118: macros prefixed with 'GL_' are reserved

Cheers
Sebastian

Hi All,

I have now checked in core OSG support for a new form of shader 
composition that leverages #pragma's to extended GLSL shaders to 
specify which #define's that it requires from the scene graph. The 
scene graph passes these to the shaders via StateSet::setDefine(..) 
and provides full support for the OSG's state override/protected options.


I have modified the osgshadercomposition example to illustrate a very 
simple example that utlizes three shaders checked into 
OpenSceneGraph-Data/shaders trunk.  So on the code look for 
OpenSceneGraph/examples/osgshadercomposition.cpp and on the shader 
from look at OpenSceneGraph-Data/shaders/osgshadercompostion.vert, 
osgshadercomposition.frag and lighting.vert.


I will write more later about how to use this new capability.  It's 
gotta be one of the coolest features added to the OSG for quite a few 
years :-)


Cheers,
Robert.


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,

On 17 February 2015 at 17:59, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:

 I've encountered some small problem in the shader composition example.
 You are using GL_LIGHTING which per defintion is not allowed for macros.
 Some glsl compilers will ignore this, but on my Quadro card it produces an
 error:

 0(1) : error C0118: macros prefixed with 'GL_' are reserved


Oh dear, bang goes my hope of remapping GL_LIGHTING glEnable/glDisable
fixed functionality directly the GL_LIGHTING define.

I'll modify the example to use LIGHTING etc.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Am 17.02.2015 17:52, schrieb Robert Osfield:

Hi Sebastian,

On 17 February 2015 at 16:33, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de 
mailto:sebastian.messerschm...@gmx.de wrote:



It seems to act differently :-(.


It could be you've hit upon and bug, or perhaps just a 
mis-understanding of how things are meant to behave.  I'd suspect the 
first possibility given how new the functionality is.

Indeed i did :-)
The problem is within the StateSet::merge function. The case in which i 
had the problem used StateSet::merge instead of directly setting the state.
Reinjecting the defines after merging solves the problem and after 
looking into the merge code, I suspect the error here.


Sorry for not directly providing a patch, but I'm at my laptop, so 
turn-around times with compiling and testing.

I'll provide some minimal example for the problem as soon as possible.

Cheers
Sebastian


Any chance you could reproduce the problem by modifying the 
osgshadercomposition example in svn/trunk?


The difference my case is the fact, that the element is added at
some point in time. I expect this is after all previous used
combinations have been built.
Any idea where I could set a breakpoint to see if a new
combination is created when I add the new define?


The work for tracking defines happens in 
osg::State::pushDefineList/popDefineList and selecting program and 
shader objects happen in Program header  .cpp and Shader header  
.cpp respectively.


In osg::State you'll find the following methods that can be called to 
check the current state of defines, you'd need to call this between 
the push/pop of the DefineList.


std::string getDefineString(const osg::ShaderDefines 
shaderDefines);
bool supportsShaderRequirements(const osg::ShaderDefines 
shaderRequirements);
bool supportsShaderRequirement(const std::string 
shaderRequirement);



 Robert.


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Glenn Waldron]

Hi Robert,

I'm happy to see your new work on shader composition. Naturally I have some
questions :)

In my opinion the biggest problem that shader composition (SC) needs to
address is the code maintenance nightmare of having to copy, paste, or edit
other peoples' shaders in order to add your own functionality.

So I am eager to understand how you will address that. How can I add a new
shader component without editing the main() functions to support it? For
example, take the shaders in the terrain displacement code. I want to add
some gamma correction code into the fragment stage. Will I be able to do
that without editing your shaders too?


Glenn Waldron / @glennwaldron

On Fri, Feb 13, 2015 at 1:01 PM, Robert Osfield robert.osfi...@gmail.com
wrote:

 Hi All,

 Attached is a screenshot of running (clock.osgt is in OpenSceneGraph-Data):

   osgshadercomposition clock.osgt

 A very simple example that provides a single osg::Program that have five
 different shader combinations used on the same subgraph to produce
 different results.  From left to right they are:

1) White unlit box - is the default implementation with no define's
 supplied from osg::StateSet's so no texturing or lighting

2) White lit box - GL_LIGHTING define is provided to the shaders via
 the stateset-setDefine(GL_LIGHTING);

3) Textured unlit box, GL_TEXTURE_2D define is provide to the shader
 via stateset-setDefine(GL_TEXTURE_2D);

4) Textured lit box, both GL_LIGHTING  and GL_TEXTURE_2D defines
 are provide via stateset-setDefine(GL_TEXTURE_2D); etc.

5) As for 4 but a macro function is passed to the shader via :

stateset-setDefine(VERTEX_FUNC(v) , vec4(v.x, v.y, v.z
 * sin(osg_SimulationTime), v.w));

 In each of these instances the define you provide via
 StateSet::setDefine(DefineString) is mapped to a set of:

#define DefineString

 The is passed to the shader compilation.  When
 StateSet::setDefine(DefineString, DefineValue) is used it's mapped to:

   #define DefineString DefineValue

 So for the case 5 the Define's passed to the StateSet map to a set of
 shader lines that are inserted before the body of the shader thus:

#define GL_LIGHTING
#define GL_TEXTURE_2D
#define VERTEX_FUNC(v) vec4(v.x, v.y, v.z * sin(osg_SimulationTime),
 v.w)

 These #defines are then used by the three shaders attached to the
 osg::Program in the osgshadercomposition example, the first shader is
 OpenSceneGraph-Data/shader/osgshadercomposition.vert which looks like:

 -- start of osgshadercomposition.vert --

 #pragma import_defines ( GL_LIGHTING, GL_TEXTURE_2D, VERTEX_FUNC(v) )

 #ifdef GL_LIGHTING
 // forward declare lighting computation, provided by lighting.vert shader
 void directionalLight( int lightNum, vec3 normal, inout vec4 color );
 #endif

 #ifdef GL_TEXTURE_2D
 varying vec2 texcoord;
 #endif

 #ifdef VERTEX_FUNC
 uniform float osg_SimulationTime;
 #endif

 varying vec4 basecolor;

 void main(void)
 {
 basecolor = gl_Color;

 #ifdef GL_LIGHTING
 directionalLight( 0, gl_Normal.xyz, basecolor);
 #endif

 #ifdef GL_TEXTURE_2D
 // if we want texturing we need to pass on texture coords
 texcoord = gl_MultiTexCoord0.xy;
 #endif

 #ifdef VERTEX_FUNC
 gl_Position   = gl_ModelViewProjectionMatrix * VERTEX_FUNC(gl_Vertex);
 #else
 gl_Position   = gl_ModelViewProjectionMatrix * gl_Vertex;
 #endif

 }

 -- end of osgshadercomposition.vert --

 First thing of note is the first line:

#pragma import_defines(GL_LIGHTING, GL_TEXTURE_2D, VERTEX_FUNC(v))

 This #pragma is read and used by the OSG to tell it what Define's to look
 up and apply if they are provided in the parental chain of StateSet's, note
 the matching is case sensitive and in the case of macro VERTEX_FUNC(v) you
 must use exactly the same parameters including name as the matching is
 doing straight std::string match internally.  When using import_defines()
 the defines are all treated as optional, so if they aren't supplied by the
 StateSet's then nothing is passed along.  Also if you switch off a define
 via setDefine(GL_LIGHTING, osg::StateAttribute::OFF); then this value
 will not be passed on.

 It is intended that the shaders themselves will use #ifdef blocks to
 enable/disable various features so that all the various paths make sense.
 This approach make it possible to implement optional varying and uniform
 usage as well as operations within the main or functions.  It also ensures
 that the shader compilation done by the drive just compiles what is
 required, it doesn't have extra uniform variables that might be constant or
 varyings and uniforms that might not be used in all code paths.

 --

 The second shader is OpenSceneGraph-Data/shaders/lighting.vert that
 provides the directionalLight() function that is used by the
 osgshadercomposition.vert main when GL_LIGHTING is enabled via
 StateSet::setDefine(GL_LIGHTING).  Since this shader is not required when
 GL_LIGHTING is not defined it's not 

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Farshid,

On 16 February 2015 at 23:18, Farshid Lashkari fla...@gmail.com wrote:

 I haven't had a chance to really dive into the implementation details, so
 forgive me if my concerns are completely overblown. Like David, I can't
 really provide a concise example program that demonstrates how I'm
 currently using the ShaderComposer framework. However, there are a couple
 features of the existing framework I'd like to point out that are
 absolutely critical for my use case.

 1) Being able to override osg::ShaderComposer::getOrCreateProgram(). This
 is huge, because I can implement my own algorithm for composing shader
 code. It also gives me control of how the osg::Program object is generated
 and allows me to automatically apply uniforms based on the shader
 components. Another benefit is that I can implement my own shader caching
 strategy.


The OSG already only applies uniforms that are relevant to an GLSL program
so even if you provide them in the scene graph, but not use them in the
shade they will be ignored.  This means you shouldn't need to go to the
extra effort of applying/not applying uniforms as it'll already be done for
you.

Control of what elements go into the final GLSL program can now be
controlled via the #pragma requires() functionality, so it moves the need
to manage this side of things from the C++ code into the GLSL code.  As the
GLSL code is compiled at runtime it makes things more flexible.
Development different shader combination no longer requires modifying C++
code and then re-compiling the app.  Potentially you can even edit the
shaders and their composition at runtime.




 2) The osg::ShaderComposer object is called within the context of
 osg::State::apply. This is crucial, because it allows my composer to
 automatically apply textures based on the specified shader components. For
 example, if I wanted to add caustic lighting to the scene, my custom shader
 component would specify the caustic texture and shader code. The composer
 would then apply the texture to the next available texture unit and
 automatically apply the necessary sampler uniform. This makes it easy for
 us to define self contained shader effects that use textures, without
 needing to be aware of the scene and which texture units are available.
 This is also used for adding texture projected lights, ambient light cube
 maps, ramp-based lighting models, etc...


The new shader composition functionality can easily toggle on/off shaders
using the #pragma requires() functionality so it's now lightweight to just
attach the shader to the osg::Program decorating your subgraph and have it
appropriate define toggle/on off the feature when needed.

Selection of the textures required or not required and associated uniforms
is not something supported by the new functionality, but be managed via
cull callbacks.  An simpler alternative is to just decide which texture
unit you are going to place a particular texture and assign it and the
associated uniform for it.  This could leaves gaps in the texture unit
usage but there is nothing in OpenGL multi-texturing support that requires
the units to be sequentially assigned.

If the new framework allows for such use cases, then great! If not, then I
 think it's reasonable to keep the old framework around for advanced use
 cases that require more low-level hooks into the scene graph.


I will keep the old framework around for 3.4, but officially deprecated
it.  It might be parts of the old framework could be utilized in a future
refinement of #pragma(tic) shader composition but for new users I don't
want them trying to use something that is far more complicated than the
functional benefit it provides.



 Again, the new framework looks great and I'm not criticizing it at all.
 I'm just worried about losing years of work I've invested into the old
 framework.


 If it took you years of work to get the old shader composition working
then perhaps that's a sign that I got it wrong :-)

What I'd appreciate a proper review of the new functionality with an eye to
work out how all these various tasks can be solved.  Certainly from the
feedback about not removing osg::ShaderComposition has not been done from a
basis of seeing how to utilize the framework, but from trying to fit the
new functionality into the same boxes that the old functionality provided.
#pragma(tic) shader composition is a very different beast.  It moves
control from C++ application code to GLSL shaders.

So please guys, can you spend some time thinking about, and trying out the
new approach in a constructive way.  I want feedback on it.  So far all
I've had feedback is on my suggestion of removing the old functionality.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[David Callu]

Hi Sebastian,

As Robert say in its previous mail, he want to depreciate/remove old shader
composition. This is the bad thing.

Cheers
David

2015-02-16 19:03 GMT+01:00 Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de:

  Hi David,
 I don't see where you see problems. The old ShaderComposition is still
 there, and basically nothing has changed, as the new feature is orthogonal
 to the old one.
 I'm using some adapted version of the ShaderComposition framework, and it
 still works with the current trunk, so could you please provide some
 example what is failing?

 Cheers
 Sebastian

   Hi Robert, Hi all


 I also implement a shader composition framework for a client, based on osg
 shader composition, so remove it will be a big loss.
 My implementation handle OpenGL version, some pragma (optimize and
 debug), generate a main function for vertex and fragment shader based on
 what is found in shaderComponent list. So change from previous shader
 composition to new one will not be  straight forward.



  some thought about new shader composition

 1)
 OpenGL Shading Language specification
 3.3 Preprocessor :
 #pragma allows implementation dependent compiler control

 so pragma requires and import_defines should be use by shader
 compiler, replace them by osg_requires and osg_import_defines could
 be prevent futur pragma collisions. Thought ?

  but pragma is an elegant way to insert information directly in shader
 code. well spot !!!




 2)
  Define a main function with all possibility and use define to choose the
 ones to enable and the ones to not will make
  a really complex shader, imagine, you optionally use
  - instancing
 - bump/deplacement mapping
 - vertex blending
 - material/texture/multi-texture
  - lighting/shadow
  - fog
  - physic
  - one/multi fragment output
  You will be quickly crazy. The one that work with SpeedTree Shader knows
 what I am talking about.
  This is why I prefer your previous approach that inject code found in
 StateAttribute in final program.
  It is also less intrusive. If i need to insert a new shader component in
 the program, no need to change main
  shader. More over, I introduce some new ShaderAttribute like camera to
 define where fragment shader output have to go,
  or drawable to define which vertex shader input are available,
 Instance to optionally use draw instance, ... so don't remove Shader
 Attribute. With modern OpenGL and depreciated data like fog, light, ...
 we need a way to introduce our custom attribute.
  I create a nodekit osgLighting just to manage different light source. I
 have 5 different light source (sun, car, streetlight, ...) that all use
 shaderAttribute to define data that compose the light and shader that use
 this data. So don't remove shaderAttribute this is really a good idea you
 have here !!!


  3)
 On a performance point of view, it will be a good thing to use subroutine
 to enable/disable feature.
  Many different path in shader composition result in many different
 program. And so many change
  of program in one frame, perhaps use the program A, then the program B,
 then the program A again...
  This is the main problem I currently have in my shader composition
 implementation.

 4)
  I always think that lack of define injection in shader is a drawback in
 osg. With your function and name convention,
  this will be difficult to add this feature. Replace import_define by
 import_module will fix this.
 Because we talk about shader module, lighting, fog, bump mapping, shadow
 ... all of this could be define as shader module.




  I really think that shader composition is a big piece of software,
 really usefull, and really complex.
  I think we need a  debate about this with all user that have already try
 to implement a shader composition.
  Clearly define our needs (have reusable module, keep code in shader
 files, use OpenGL feature to improve performance like subroutine,
 pipeline object, ...), debate about solution, propose implementation...


  This problem is really new and so few research has be done an the
 subject. I did my first implementation in octobre 2010
  I have used my shader composition in many way. Found some cool stuff and
 other bad about my design. I look for existing solution
  on internet, I found some one like libsh http://libsh.org/ but not
 easy and clean way to solve it. Recently i found CosMo
 http://www.vrvis.at/publications/pdfs/PB-VRVis-2014-001.pdf that did
 the job in many way
  like my implementation, but more deeply in abstraction of the code. They
 don't think in term of shader but in term of
 pipeline graphics, shader composition will decide which code put in
 vertex/tessellation/geometry/fragment shader. I also do this
  but it's clearly not a good idea.

 You propose a clean and (perhaps too) easy way to do the job. This is
 fresh. This is nice to have a new view on this complex problem.
  So, Robert, Other, we need to talk about this to create a robust, clean
 and easy way to do shader 

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi Sebastian,

On 16 February 2015 at 18:03, Sebastian Messerschmidt 
sebastian.messerschm...@gmx.de wrote:

  I don't see where you see problems. The old ShaderComposition is still
 there, and basically nothing has changed, as the new feature is orthogonal
 to the old one.
 I'm using some adapted version of the ShaderComposition framework, and it
 still works with the current trunk, so could you please provide some
 example what is failing?


As David says, I'd like to deprecate the old ShaderComposition framework.
Short term for those using the framework this would be bad thing if I
remove it right away without any easy migration route.  I wasn't many users
to have utlized the old ShaderComposition framework due to it's short
commings so I wasn't expecting that deprecation would affect end users.

My interest in removing the old ShaderComposition framework is to simply
the OSG base, both for maintainers i.e. me, but also the community as
having multiple ways of doing the same thing can cause confusion about what
is the appropriate set of classes to use when, and how they all related to
each other.

Given that yourself, David and at least one other osg-user has mentioned
that they are actively using the old ShaderComposition framework I think
it' s sensible to just official deprecated the old ShaderComposition
framework, but leave it in place for OSG-3.4.

Long term I believe the new #pragma(tic) shader composition approach is
better.  It doesn't solve all the same things that the old
ShaderComposition framework could, but does all the mostly commonly needed
things in so much more straight forward way that for the vast majority of
users it'll be a no brainer.   There is the potential for adding support
for some of the features that exist in osg::ShaderComposition to the new
framework so there will be no features missing.  For instance placing
osg::Shader's right into the scene graph without attaching them to an
osg::Program and having osg::State collect them and link them into a
Program.

What I'd like to avoid is end users having to know about lots of layers of
abstraction, which is where the old osg::ShaderComposition fails for me -
there are just too many different collaborating classes and too much
automatically generated code that it totally obfuscates what is going on.
When I modified the osgshadercomposition example to add the new code in I
reviewed the old code to work out how it did what it did I really struggled
to work out how it worked - for me this sounded a big klaxon in my head
that something is badly designed for end users.  If the original author of
the example and classes it's based on can't see how it works there is
serious problem.

By contrast I'd hope the new approach is far easier to understand what is
going on.  I have leveraged basic C language concepts of #pragma and
#define's and just adds two keywords to the GLSL vocabulary. Behind the
scenes I have to a bit of fancy footwork to hide the complexities, but this
is how it should be, the API should be simplify things for end users.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Sebastian Messerschmidt]

Hi David,
I don't see where you see problems. The old ShaderComposition is still 
there, and basically nothing has changed, as the new feature is 
orthogonal to the old one.
I'm using some adapted version of the ShaderComposition framework, and 
it still works with the current trunk, so could you please provide some 
example what is failing?


Cheers
Sebastian

Hi Robert, Hi all


I also implement a shader composition framework for a client, based on 
osg shader composition, so remove it will be a big loss.
My implementation handle OpenGL version, some pragma (optimize and 
debug), generate a main function for vertex and fragment shader based 
on what is found in shaderComponent list. So change from previous 
shader composition to new one will not be  straight forward.




some thought about new shader composition

1)
OpenGL Shading Language specification
3.3 Preprocessor :
#pragma allows implementation dependent compiler control

so pragma requires and import_defines should be use by shader 
compiler, replace them by osg_requires and osg_import_defines 
could be prevent futur pragma collisions. Thought ?


but pragma is an elegant way to insert information directly in shader 
code. well spot !!!





2)
Define a main function with all possibility and use define to choose 
the ones to enable and the ones to not will make

a really complex shader, imagine, you optionally use
- instancing
- bump/deplacement mapping
- vertex blending
- material/texture/multi-texture
- lighting/shadow
- fog
- physic
- one/multi fragment output
You will be quickly crazy. The one that work with SpeedTree 
Shader knows what I am talking about.
This is why I prefer your previous approach that inject code found in 
StateAttribute in final program.
It is also less intrusive. If i need to insert a new shader component 
in the program, no need to change main
shader. More over, I introduce some new ShaderAttribute like camera 
to define where fragment shader output have to go,
or drawable to define which vertex shader input are available, 
Instance to optionally use draw instance, ... so don't remove 
ShaderAttribute. With modern OpenGL and depreciated data like fog, 
light, ... we need a way to introduce our custom attribute.
I create a nodekit osgLighting just to manage different light source. 
I have 5 different light source (sun, car, streetlight, ...) that all 
use shaderAttribute to define data that compose the light and shader 
that use this data. So don't remove shaderAttribute this is really a 
good idea you have here !!!



3)
On a performance point of view, it will be a good thing to use 
subroutine to enable/disable feature.
Many different path in shader composition result in many different 
program. And so many change
of program in one frame, perhaps use the program A, then the program 
B, then the program A again...
This is the main problem I currently have in my shader composition 
implementation.


4)
I always think that lack of define injection in shader is a drawback 
in osg. With your function and name convention,
this will be difficult to add this feature. Replace import_define by 
import_module will fix this.
Because we talk about shader module, lighting, fog, bump mapping, 
shadow ... all of this could be define as shader module.





I really think that shader composition is a big piece of software, 
really usefull, and really complex.
I think we need a  debate about this with all user that have already 
try to implement a shader composition.
Clearly define our needs (have reusable module, keep code in shader 
files, use OpenGL feature to improve performance like subroutine, 
pipeline object, ...), debate about solution, propose implementation...



This problem is really new and so few research has be done an the 
subject. I did my first implementation in octobre 2010
I have used my shader composition in many way. Found some cool stuff 
and other bad about my design. I look for existing solution
on internet, I found some one like libsh http://libsh.org/ but not 
easy and clean way to solve it. Recently i found CosMo 
http://www.vrvis.at/publications/pdfs/PB-VRVis-2014-001.pdf that did 
the job in many way
like my implementation, but more deeply in abstraction of the code. 
They don't think in term of shader but in term of
pipeline graphics, shader composition will decide which code put in 
vertex/tessellation/geometry/fragment shader. I also do this

but it's clearly not a good idea.

You propose a clean and (perhaps too) easy way to do the job. This is 
fresh. This is nice to have a new view on this complex problem.
So, Robert, Other, we need to talk about this to create a robust, 
clean and easy way to do shader composition.


Thought ?

My 2 cents
David



2015-02-13 19:01 GMT+01:00 Robert Osfield robert.osfi...@gmail.com 
mailto:robert.osfi...@gmail.com:


Hi All,

Attached is a screenshot of running (clock.osgt is in
OpenSceneGraph-Data):

  osgshadercomposition clock.osgt

A very 

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Farshid Lashkari]

Hi Robert,

The new shader composition feature seems really interesting. The code
sample is easy to follow and at first glance it does appear to be simpler
to work with than the previous osg::ShaderComposer implementation. However,
as someone who has invested a lot of time and work into the older
framework, I'd like to also voice my concern over your plan to
deprecate/remove it.

I haven't had a chance to really dive into the implementation details, so
forgive me if my concerns are completely overblown. Like David, I can't
really provide a concise example program that demonstrates how I'm
currently using the ShaderComposer framework. However, there are a couple
features of the existing framework I'd like to point out that are
absolutely critical for my use case.

1) Being able to override osg::ShaderComposer::getOrCreateProgram(). This
is huge, because I can implement my own algorithm for composing shader
code. It also gives me control of how the osg::Program object is generated
and allows me to automatically apply uniforms based on the shader
components. Another benefit is that I can implement my own shader caching
strategy.

2) The osg::ShaderComposer object is called within the context of
osg::State::apply. This is crucial, because it allows my composer to
automatically apply textures based on the specified shader components. For
example, if I wanted to add caustic lighting to the scene, my custom shader
component would specify the caustic texture and shader code. The composer
would then apply the texture to the next available texture unit and
automatically apply the necessary sampler uniform. This makes it easy for
us to define self contained shader effects that use textures, without
needing to be aware of the scene and which texture units are available.
This is also used for adding texture projected lights, ambient light cube
maps, ramp-based lighting models, etc...

If the new framework allows for such use cases, then great! If not, then I
think it's reasonable to keep the old framework around for advanced use
cases that require more low-level hooks into the scene graph.

Again, the new framework looks great and I'm not criticizing it at all. I'm
just worried about losing years of work I've invested into the old
framework.

Cheers,
Farshid


On Fri, Feb 13, 2015 at 7:00 AM, Robert Osfield robert.osfi...@gmail.com
wrote:

 Hi All,

 I have now checked in core OSG support for a new form of shader
 composition that leverages #pragma's to extended GLSL shaders to specify
 which #define's that it requires from the scene graph. The scene graph
 passes these to the shaders via StateSet::setDefine(..) and provides full
 support for the OSG's state override/protected options.

 I have modified the osgshadercomposition example to illustrate a very
 simple example that utlizes three shaders checked into
 OpenSceneGraph-Data/shaders trunk.  So on the code look for
 OpenSceneGraph/examples/osgshadercomposition.cpp and on the shader from
 look at OpenSceneGraph-Data/shaders/osgshadercompostion.vert,
 osgshadercomposition.frag and lighting.vert.

 I will write more later about how to use this new capability.  It's gotta
 be one of the coolest features added to the OSG for quite a few years :-)

 Cheers,
 Robert.

 ___
 osg-users mailing list
 osg-users@lists.openscenegraph.org
 http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org


___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi David,

Thanks for the further details.  I'm afraid I still a bit lost in how your
modifications all fit together and work in practice.  Not being able to
review code and having to second guess what this third party code might be
is not really a good way to understand something.  I don't think this
thread is the right place to try and thrash out complex stuff implemented
on a alternate approach to shader composition - previous threads would be
better places as I really don't want end users confusing themselves.

This thread needs to be about the new #pragma(tic) shaders, how they work,
how they can be improved further.  The new approach is far simpler to
understand and use than any previous attempt at shader composition that
we'd done, so for the vast majority of users I believe it'll be the best
solution.

Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[David Callu]

2015-02-14 18:52 GMT+01:00 Robert Osfield robert.osfi...@gmail.com:

 Hi David,


 On 14 February 2015 at 14:24, David Callu led...@gmail.com wrote:

 I also implement a shader composition framework for a client, based on
 osg shader composition, so remove it will be a big loss.
 My implementation handle OpenGL version, some pragma (optimize and
 debug), generate a main function for vertex and fragment shader based on
 what is found in shaderComponent list. So change from previous shader
 composition to new one will not be  straight forward.


 Is there a chance you could create an example that illustrates how you are
 using the old osg::ShaderComposition framework?


An example will not compile and require the two NodeKit I done to implement
Shader Composition to be comprehensible. But I can explain the main process:

I derived osg::ShaderComponent to containe shader code and provide
information how to assemble this component with other
shader component (input data and output data, version/extension
requirement, GLSL buildin variable used, ...).
I use ShaderAttribute that hold ShaderComponent (code) and uniform (data
used by code)
During draw stage, osg::StateSet are applied in osg::State, shaderAttribute
are stacked and shadercomponent are collected.
osg::State::ShaderComposition flag is enable so shader composer is used.
Then I derived from osg::ShaderComposer to implement my own composer by
reimplementing osg::ShaderComposer::getOrCreateProgram(...).

The part I never use is osg::Shader::*CodeInjection*

Other part of ShaderComposition could not be move in external nodekit
because osg::State use them. I talk about
osg::StateAttribute::_shaderComponent and osg::ShaderComposer. This last
one could just provide virtual method getOrCreateProgram(...).





 some thought about new shader composition

 1)
 OpenGL Shading Language specification
 3.3 Preprocessor :
 #pragma allows implementation dependent compiler control

 so pragma requires and import_defines should be use by shader
 compiler, replace them by osg_requires and osg_import_defines could
 be prevent futur pragma collisions. Thought ?



 Right now I would prefer to avoid adding the osg_ prefix as it's a bit
 clumsy, and the approach needed be OSG specific.

 I am consider other additions as well - such as having shaders be able to
 request the linking of other shaders - so the app could link a single main
 shader that pulls in the rest of the shaders to be linked to the program.
 Another possible addition would be to have osg::Program settings
 controllable via #pragmas.  Right now I want to keep things simple though.

ok



 2)

 Define a main function with all possibility and use define to choose the
 ones to enable and the ones to not will make
 a really complex shader, imagine, you optionally use
 - instancing
 - bump/deplacement mapping
 - vertex blending
 - material/texture/multi-texture
 - lighting/shadow
 - fog
 - physic
 - one/multi fragment output
 You will be quickly crazy. The one that work with SpeedTree Shader knows
 what I am talking about.


 The new shader composition framework doesn't add complexity where none
 exists.  If there are many different combinations that need to be supported
 then this complexity will be reflected in the shaders, but at least the
 developer will be able to see it directly and account for the possibility
 of different combinations.


If the main function is totally generated, developer don't have to fight
with a complex main function, hard to write/maintain/understand.
In a realistic simulation, environmental and atmospheric effect is the some
for all element in the scene. Light, fog, wind, ...
are applied on all drawn element. But drawables will need different shader
to be rendered. There are standard drawable (vertex, normal, texCoord),
tessellated drawable, generated geometry in geometry shader,
bump/displacement mapping... So all this element will add more and more
complexity to the main function of each pre-rasterization shader.
And now my realistic simulation need to draw all element in the scene, but
with a thermal vision or other non conventional vision.
So I need to change fragment shader code and add new combinations to draw
as I want, change vertex shader output, change
fragment shader input, ...
And now I have all my shader stage with really complex main functions, I
spend many time to create them. I am afraid each time I have to change
them. And more over, If I have to do the same thing for another simulation,
I have to rewrite new main function that will grow in complexity if a need
more feature.
What I need is a system that integrate new feature with only few additional
work, and be reusable as-is for any existing or futur scene. (And no I'm
not a dreamer)


Unifoms should be able to take over at least part of the attribute role.
In the case of fallbacks for things like osg::Light/glLight we'd need these
atttributes to provides one or more Uniforms when they are needed.  It
could be 

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[David Callu]

Hi Robert, Hi all


I also implement a shader composition framework for a client, based on osg
shader composition, so remove it will be a big loss.
My implementation handle OpenGL version, some pragma (optimize and debug),
generate a main function for vertex and fragment shader based on what is
found in shaderComponent list. So change from previous shader composition
to new one will not be  straight forward.



some thought about new shader composition

1)
OpenGL Shading Language specification
3.3 Preprocessor :
#pragma allows implementation dependent compiler control

so pragma requires and import_defines should be use by shader compiler,
replace them by osg_requires and osg_import_defines could be prevent
futur pragma collisions. Thought ?

but pragma is an elegant way to insert information directly in shader code.
well spot !!!




2)
Define a main function with all possibility and use define to choose the
ones to enable and the ones to not will make
a really complex shader, imagine, you optionally use
- instancing
- bump/deplacement mapping
- vertex blending
- material/texture/multi-texture
- lighting/shadow
- fog
- physic
- one/multi fragment output
You will be quickly crazy. The one that work with SpeedTree Shader knows
what I am talking about.
This is why I prefer your previous approach that inject code found in
StateAttribute in final program.
It is also less intrusive. If i need to insert a new shader component in
the program, no need to change main
shader. More over, I introduce some new ShaderAttribute like camera to
define where fragment shader output have to go,
or drawable to define which vertex shader input are available, Instance
to optionally use draw instance, ... so don't remove ShaderAttribute. With
modern OpenGL and depreciated data like fog, light, ... we need a way to
introduce our custom attribute.
I create a nodekit osgLighting just to manage different light source. I
have 5 different light source (sun, car, streetlight, ...) that all use
shaderAttribute to define data that compose the light and shader that use
this data. So don't remove shaderAttribute this is really a good idea you
have here !!!


3)
On a performance point of view, it will be a good thing to use subroutine
to enable/disable feature.
Many different path in shader composition result in many different program.
And so many change
of program in one frame, perhaps use the program A, then the program B,
then the program A again...
This is the main problem I currently have in my shader composition
implementation.

4)
I always think that lack of define injection in shader is a drawback in osg.
With your function and name convention,
this will be difficult to add this feature. Replace import_define by
import_module will fix this.
Because we talk about shader module, lighting, fog, bump mapping, shadow
... all of this could be define as shader module.




I really think that shader composition is a big piece of software, really
usefull, and really complex.
I think we need a  debate about this with all user that have already try to
implement a shader composition.
Clearly define our needs (have reusable module, keep code in shader files,
use OpenGL feature to improve performance like subroutine, pipeline object,
...), debate about solution, propose implementation...


This problem is really new and so few research has be done an the subject.
I did my first implementation in octobre 2010
I have used my shader composition in many way. Found some cool stuff and
other bad about my design. I look for existing solution
on internet, I found some one like libsh http://libsh.org/ but not easy
and clean way to solve it. Recently i found CosMo
http://www.vrvis.at/publications/pdfs/PB-VRVis-2014-001.pdf that did the
job in many way
like my implementation, but more deeply in abstraction of the code. They
don't think in term of shader but in term of
pipeline graphics, shader composition will decide which code put in vertex/
tessellation/geometry/fragment shader. I also do this
but it's clearly not a good idea.

You propose a clean and (perhaps too) easy way to do the job. This is
fresh. This is nice to have a new view on this complex problem.
So, Robert, Other, we need to talk about this to create a robust, clean and
easy way to do shader composition.

Thought ?

My 2 cents
David



2015-02-13 19:01 GMT+01:00 Robert Osfield robert.osfi...@gmail.com:

 Hi All,

 Attached is a screenshot of running (clock.osgt is in OpenSceneGraph-Data):

   osgshadercomposition clock.osgt

 A very simple example that provides a single osg::Program that have five
 different shader combinations used on the same subgraph to produce
 different results.  From left to right they are:

1) White unlit box - is the default implementation with no define's
 supplied from osg::StateSet's so no texturing or lighting

2) White lit box - GL_LIGHTING define is provided to the shaders via
 the stateset-setDefine(GL_LIGHTING);

3) Textured unlit box, 

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi David,

On 14 February 2015 at 14:24, David Callu led...@gmail.com wrote:

 I also implement a shader composition framework for a client, based on osg
 shader composition, so remove it will be a big loss.
 My implementation handle OpenGL version, some pragma (optimize and
 debug), generate a main function for vertex and fragment shader based on
 what is found in shaderComponent list. So change from previous shader
 composition to new one will not be  straight forward.


Is there a chance you could create an example that illustrates how you are
using the old osg::ShaderComposition framework?  This will be give me a
better idea of what could be done to ease the transition/what parts of the
existing framework will need to be maintained.

I am keen to avoid the OSG littering up with lots of dead end bits of code.






 some thought about new shader composition

 1)
 OpenGL Shading Language specification
 3.3 Preprocessor :
 #pragma allows implementation dependent compiler control

 so pragma requires and import_defines should be use by shader
 compiler, replace them by osg_requires and osg_import_defines could
 be prevent futur pragma collisions. Thought ?



Right now I would prefer to avoid adding the osg_ prefix as it's a bit
clumsy, and the approach needed be OSG specific.

I am consider other additions as well - such as having shaders be able to
request the linking of other shaders - so the app could link a single main
shader that pulls in the rest of the shaders to be linked to the program.
Another possible addition would be to have osg::Program settings
controllable via #pragmas.  Right now I want to keep things simple though.


2)

 Define a main function with all possibility and use define to choose the
 ones to enable and the ones to not will make
 a really complex shader, imagine, you optionally use
 - instancing
 - bump/deplacement mapping
 - vertex blending
 - material/texture/multi-texture
 - lighting/shadow
 - fog
 - physic
 - one/multi fragment output
 You will be quickly crazy. The one that work with SpeedTree Shader knows
 what I am talking about.


The new shader composition framework doesn't add complexity where none
exists.  If there are many different combinations that need to be supported
then this complexity will be reflected in the shaders, but at least the
developer will be able to see it directly and account for the possibility
of different combinations.

One isn't tied to one uber osg::Program that manages all the possible
combinations. One also can use the define's to inject code, more info below.



 This is why I prefer your previous approach that inject code found in
 StateAttribute in final program.
 It is also less intrusive. If i need to insert a new shader component in
 the program, no need to change main
 shader.


The way to you implement code injection right now was illustrated in a
small way in the example I gave in my previous email, and provided in the
svn/trunk version of the osgshadercomposition example.  This example
injects a marco function that modifies a vertex.  You don't have to use it
this way instead you can do the following, in the GLSL shader you'd have:

#pragma import_defines( INJECTION_POINT_1, INJECTION_POINT_2 )
...

INJECTION_POINT_1

void main(void)
{
...

INEJECTION_POINT_2

..
}

Then in the StateSet you'd pass in the code to inject via:

  stateset-setDefine(INJECTION_POINT_1, varying vec4 position;)
  stateset-setDefine(INJECTION_POINT_2, position = gl_Vertex;)

The strings you pass in can be multi-line as long as the driver supports
multi-line #define as C and C++ do.

Right now if you nest a setDefine(defStr, valueStr) in the scene graph
it'll replace entirely any previous value but perhaps we could generalize
things further to allow one to prepend or append the valueStr to any
already defined in StateSet above that point the scene graph.  This would
add complexity though, so unless it's really shown to be needed I'll wait
till it is.


More over, I introduce some new ShaderAttribute like camera to define
 where fragment shader output have to go,
 or drawable to define which vertex shader input are available,
 Instance to optionally use draw instance, ... so don't remove Shader
 Attribute. With modern OpenGL and depreciated data like fog, light, ...
 we need a way to introduce our custom attribute.


Unifoms should be able to take over at least part of the attribute role.
In the case of fallbacks for things like osg::Light/glLight we'd need these
atttributes to provides one or more Uniforms when they are needed.  It
could be that ShaderAttribute might still have a roll to play in this.  The
code injection side is something that would need to be looked at if we are
removing the old osg::ShaderComposition framework.

Something I have used in the osgTerrain::DispalcementMappingTechnique to
have a custom Drawable check the value of a Define by querying osg::State
and then having the Drawable choose between different data to send to
OpenGL, in 

[osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi All,

I have now checked in core OSG support for a new form of shader composition
that leverages #pragma's to extended GLSL shaders to specify which
#define's that it requires from the scene graph. The scene graph passes
these to the shaders via StateSet::setDefine(..) and provides full support
for the OSG's state override/protected options.

I have modified the osgshadercomposition example to illustrate a very
simple example that utlizes three shaders checked into
OpenSceneGraph-Data/shaders trunk.  So on the code look for
OpenSceneGraph/examples/osgshadercomposition.cpp and on the shader from
look at OpenSceneGraph-Data/shaders/osgshadercompostion.vert,
osgshadercomposition.frag and lighting.vert.

I will write more later about how to use this new capability.  It's gotta
be one of the coolest features added to the OSG for quite a few years :-)

Cheers,
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

Re: [osg-users] #pragma(tic) shader composition support now checked into svn/trunk

[Robert Osfield]

Hi All,

Attached is a screenshot of running (clock.osgt is in OpenSceneGraph-Data):

  osgshadercomposition clock.osgt

A very simple example that provides a single osg::Program that have five
different shader combinations used on the same subgraph to produce
different results.  From left to right they are:

   1) White unlit box - is the default implementation with no define's
supplied from osg::StateSet's so no texturing or lighting

   2) White lit box - GL_LIGHTING define is provided to the shaders via
the stateset-setDefine(GL_LIGHTING);

   3) Textured unlit box, GL_TEXTURE_2D define is provide to the shader
via stateset-setDefine(GL_TEXTURE_2D);

   4) Textured lit box, both GL_LIGHTING  and GL_TEXTURE_2D defines are
provide via stateset-setDefine(GL_TEXTURE_2D); etc.

   5) As for 4 but a macro function is passed to the shader via :

   stateset-setDefine(VERTEX_FUNC(v) , vec4(v.x, v.y, v.z *
sin(osg_SimulationTime), v.w));

In each of these instances the define you provide via
StateSet::setDefine(DefineString) is mapped to a set of:

   #define DefineString

The is passed to the shader compilation.  When
StateSet::setDefine(DefineString, DefineValue) is used it's mapped to:

  #define DefineString DefineValue

So for the case 5 the Define's passed to the StateSet map to a set of
shader lines that are inserted before the body of the shader thus:

   #define GL_LIGHTING
   #define GL_TEXTURE_2D
   #define VERTEX_FUNC(v) vec4(v.x, v.y, v.z * sin(osg_SimulationTime), v.w)

These #defines are then used by the three shaders attached to the
osg::Program in the osgshadercomposition example, the first shader is
OpenSceneGraph-Data/shader/osgshadercomposition.vert which looks like:

-- start of osgshadercomposition.vert --

#pragma import_defines ( GL_LIGHTING, GL_TEXTURE_2D, VERTEX_FUNC(v) )

#ifdef GL_LIGHTING
// forward declare lighting computation, provided by lighting.vert shader
void directionalLight( int lightNum, vec3 normal, inout vec4 color );
#endif

#ifdef GL_TEXTURE_2D
varying vec2 texcoord;
#endif

#ifdef VERTEX_FUNC
uniform float osg_SimulationTime;
#endif

varying vec4 basecolor;

void main(void)
{
basecolor = gl_Color;

#ifdef GL_LIGHTING
directionalLight( 0, gl_Normal.xyz, basecolor);
#endif

#ifdef GL_TEXTURE_2D
// if we want texturing we need to pass on texture coords
texcoord = gl_MultiTexCoord0.xy;
#endif

#ifdef VERTEX_FUNC
gl_Position   = gl_ModelViewProjectionMatrix * VERTEX_FUNC(gl_Vertex);
#else
gl_Position   = gl_ModelViewProjectionMatrix * gl_Vertex;
#endif

}

-- end of osgshadercomposition.vert --

First thing of note is the first line:

   #pragma import_defines(GL_LIGHTING, GL_TEXTURE_2D, VERTEX_FUNC(v))

This #pragma is read and used by the OSG to tell it what Define's to look
up and apply if they are provided in the parental chain of StateSet's, note
the matching is case sensitive and in the case of macro VERTEX_FUNC(v) you
must use exactly the same parameters including name as the matching is
doing straight std::string match internally.  When using import_defines()
the defines are all treated as optional, so if they aren't supplied by the
StateSet's then nothing is passed along.  Also if you switch off a define
via setDefine(GL_LIGHTING, osg::StateAttribute::OFF); then this value
will not be passed on.

It is intended that the shaders themselves will use #ifdef blocks to
enable/disable various features so that all the various paths make sense.
This approach make it possible to implement optional varying and uniform
usage as well as operations within the main or functions.  It also ensures
that the shader compilation done by the drive just compiles what is
required, it doesn't have extra uniform variables that might be constant or
varyings and uniforms that might not be used in all code paths.

--

The second shader is OpenSceneGraph-Data/shaders/lighting.vert that
provides the directionalLight() function that is used by the
osgshadercomposition.vert main when GL_LIGHTING is enabled via
StateSet::setDefine(GL_LIGHTING).  Since this shader is not required when
GL_LIGHTING is not defined it's not appropriate to link the shader to
Program, so here we provide a #pragma requres(GL_LIGHTING) as a directive
to the OSG so it know when to link it.  The shader looks :

-- start of lighting.vert

#pragma requires(GL_LIGHTING)

void directionalLight( int lightNum, vec3 normal, inout vec4 color )
{
vec3 n = normalize(gl_NormalMatrix * normal);

float NdotL = dot( n, normalize(gl_LightSource[lightNum].position.xyz)
);
NdotL = max( 0.0, NdotL );

float NdotHV = dot( n, gl_LightSource[lightNum].halfVector.xyz );
NdotHV = max( 0.0, NdotHV );
#if 1
color *= gl_LightSource[lightNum].ambient +
 gl_LightSource[lightNum].diffuse * NdotL;
#else
color *= gl_FrontLightModelProduct.sceneColor +
 gl_FrontLightProduct[lightNum].ambient +
 gl_FrontLightProduct[lightNum].diffuse * NdotL;
#endif
#if 0