I want to record a video and put 3D augmented reality on top of each frame of
the AVI. Due to the fact that I don't want to skip frames, I store all AVI
frames in memory during recording and after each video frame has been captured,
I also store the view and projection matrices of the 3D view that is currently
applicable on this video frame.
The input of the camera is 640 x 480 and my application usually renders in 1280
x 1024 windows. The video frames can be retrieved from an array of unsigned
char arrays (stored in m_RingBuffer)
I want to achieve the rendering of the 3D data on top of the video frames in a
post-processing step (which is part of the work done in a thread called
VideoPostRecThread).
I follow this strategy:
1) I set up a pre-rendering HUD camera (m_pHudCamera) which looks at a quad
geometry with one of the video frames as a texture (m_RenderTexture)
2) The HUD camera is the child of a snapshot camera (m_pSnapshotcamera) . The
snapshot renders the 3D data. The output of the snapshot camera should go back
to the video frame, but I also store it temporarily in an image
(m_SnapshotImage). For this, I disabled the GL_COLOR_BUFFER_BIT clear mask of
the snapshot camera, to make sure the rendered outputs of the HUD camera are
not cleared.
I also use two callbacks:
1) a pre-draw callback applied on the HUD camera: the texture of the quad
geometry (m_RenderTexture) is updated each time with a new frame. Name:
m_UpdateCallback (instance of the TexturePreDrawCallback struct).
2) a post-draw callback: my thread is blocked until all the 3D data is rendered
on top of the HUD contents; the callback unblocks my thread (using a mutex
called m_SnapshotMutex). After this, I can do some post-processing (like for
example saving the AVI or updating my GUI that another frame has been
post-processed). Name: m_VideoCallback (instance of the VideoPostDrawCallback
struct)
Here are my callback definitions:
Code:
struct VideoPostDrawCallback : public osg::Camera::DrawCallback
{
VideoPostDrawCallback() {}
virtual void operator() ( osg::RenderInfo& renderInfo) const
{
renderingCompleted();
}
boost::signals2::signal<void(void)> renderingCompleted;
};
struct TexturePreDrawCallback : public osg::Camera::DrawCallback
{
TexturePreDrawCallback() {}
virtual void operator() ( osg::RenderInfo& renderInfo) const
{
updateCamera();
}
boost::signals2::signal<void(void)> updateCamera;
};
And here is the definition of my thread class
Code:
class VideoPostRecThread : public VideoRecWithArThread
{
Q_OBJECT
friend class VideoPostDrawCallback;
public:
VideoPostRecThread (boost::shared_ptr<IDSCameraManager> pCamMgr,
unsigned int maxFrames, QObject *parent = NULL);
~VideoPostRecThread ();
void renderingCompleted();
void updateTextureCamera();
private:
virtual void postProcess();
void setupImages();
void setupHudCamera();
void setupSnapshotCamera();
osg::ref_ptr<osg::Camera> m_pSnapshotcamera;
osg::ref_ptr<osg::Camera> m_pHudCamera;
osg::ref_ptr<osg::Image> m_TextureImage;
osg::ref_ptr<osg::Image> m_SnapshotImage;
osg::ref_ptr<osg::Texture2D> m_RenderTexture;
osg::ref_ptr<osg::Geode> m_QuadGeode;
osg::ref_ptr<VideoPostDrawCallback> m_VideoCallback;
osg::ref_ptr<TexturePreDrawCallback> m_UpdateCallback;
QWaitCondition m_SnapshotCondition;
QMutex m_SnapshotMutex;
unsigned int m_CurrentArFrameIndex;
};
Here is the implementation of the VideoPostRecThread class.
Code:
void VideoPostRecThread ::setupImages()
{
m_SnapshotImage = new Image();
m_SnapshotImage->allocateImage(1280,1024,1, GL_RGBA, GL_UNSIGNED_BYTE);
m_TextureImage = new Image();
}
void VideoPostRecThread ::setupHudCamera()
{
// Create the texture to render to
m_RenderTexture = new osg::Texture2D;
m_RenderTexture->setDataVariance(osg::Object::DYNAMIC);
m_RenderTexture->setInternalFormat(GL_RGBA);
osg::ref_ptr<osg::Geometry> screenQuad;
screenQuad = osg::createTexturedQuadGeometry(osg::Vec3(),
osg::Vec3(1280, 0.0, 0.0),
osg::Vec3(0.0, 1024, 0.0));
m_QuadGeode = new osg::Geode;
m_QuadGeode->addDrawable(screenQuad.get());
m_pHudCamera = new osg::Camera;
m_pHudCamera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
m_pHudCamera->setRenderOrder(osg::Camera::PRE_RENDER);
m_pHudCamera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
m_pHudCamera->setProjectionMatrix(osg::Matrix::ortho2D(0, 1280, 0,
1024));
m_pHudCamera->setViewMatrix(osg::Matrix::identity());
m_pHudCamera->setViewport(0,0,1280,1024);
m_pHudCamera->addChild(m_QuadGeode);
m_UpdateCallback = new TexturePreDrawCallback();
m_UpdateCallback->updateCamera.connect(boost::bind(&PvaitvVideoRecThread::updateTextureCamera,this));
m_pHudCamera->setPreDrawCallback(m_UpdateCallback);
}
void VideoPostRecThread ::setupSnapshotCamera()
{
m_pSnapshotcamera = new osg::Camera();
m_pSnapshotcamera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
//m_pSnapshotcamera->setRenderOrder(osg::Camera::POST_RENDER);
m_pSnapshotcamera->setClearMask(GL_DEPTH_BUFFER_BIT);
//m_pSnapshotcamera->setDrawBuffer(GL_BACK);
//m_pSnapshotcamera->setReadBuffer(GL_BACK);
//m_pSnapshotcamera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
m_pSnapshotcamera->attach(osg::Camera::COLOR_BUFFER,
m_SnapshotImage,0,0);
osg::ref_ptr<osg::Node> pScene = getSceneManager()->getSceneData();
m_pSnapshotcamera->addChild(pScene);
getSceneManager()->getSceneRoot()->addChild(m_pSnapshotcamera.get());
m_pSnapshotcamera->addChild(m_pHudCamera);
m_VideoCallback = new VideoPostDrawCallback();
m_VideoCallback->renderingCompleted.connect(boost::bind(&PvaitvVideoRecThread::renderingCompleted,this));
m_pSnapshotcamera->setPostDrawCallback(m_VideoCallback);
}
void VideoPostRecThread ::postProcess()
{
setupImages();
setupHudCamera();
setupSnapshotCamera();
m_CurrentArFrameIndex = 0;
for (unsigned int i = 0; i < m_RecordedFrames; ++i)
{
m_SnapshotMutex.lock();
m_SnapshotCondition.wait(&m_SnapshotMutex);
m_SnapshotMutex.unlock();
osgDB::writeImageFile(*m_SnapshotImage, "file2.bmp" );
// Do some post-processing code
}
m_VideoCallback->renderingCompleted.disconnect_all_slots();
m_UpdateCallback->updateCamera.disconnect_all_slots();
m_pSnapshotcamera->setPostDrawCallback(NULL);
m_pHudCamera->setPreDrawCallback(NULL);
getSceneManager()->getSceneRoot()->removeChild(m_pHudCamera);
getSceneManager()->getSceneRoot()->removeChild(m_pSnapshotcamera);
// End step of the post processing
}
void VideoPostRecThread ::renderingCompleted()
{
m_SnapshotMutex.lock();
m_SnapshotCondition.wakeAll();
m_SnapshotMutex.unlock();
}
void VideoPostRecThread ::updateTextureCamera()
{
if (m_CurrentArFrameIndex < m_RecordedFrames)
{
m_TextureImage->setImage(m_VideoWidth,m_VideoHeight,1, 3,
GL_RGBA, GL_UNSIGNED_BYTE,
(unsigned char*) m_RingBuffer[m_CurrentArFrameIndex],
Image::NO_DELETE);
m_RenderTexture->setImage(m_TextureImage);
osg::StateSet* quadState = m_QuadGeode->getOrCreateStateSet();
quadState->setTextureAttributeAndModes(0, m_RenderTexture,
osg::StateAttribute::ON);
float snapHorFov = 60.0, snapVerFov = 60.0;
float snapAspect = snapHorFov/snapVerFov;
m_pSnapshotcamera->setProjectionMatrixAsPerspective(snapVerFov,
snapAspect,0.1, 40000);
m_pSnapshotcamera->setViewMatrix(m_ViewMatrices[m_CurrentArFrameIndex]);
//m_pSnapshotcamera->attach(osg::Camera::COLOR_BUFFER,
m_SnapshotImage,0,0);
++m_CurrentArFrameIndex;
}
}
You can see a do a quick test each time the post draw callback has been called,
by writing the contents of m_SnapshotImage to a BMP (file2.bmp).
Now, when I look at this file, I see that indeed the 3D data is on top of my
video frame. Here are my questions now:
1) My GUI has a sidebar on the right hand side. The image has just gray pixels
in the same area as my sidebar is.
2) Suppose I want to render to a 640 x 480 image instead of a 1280 x 1024
image, how do I do that ? I want to preserve the same contents, so just a
resized format basically.
3) How can I avoid that also the images that are written to the video stream
(and file2.bmp) appear on the screen ?
4) I see my video frame is flipped horizontally. This is probably due to the
fact that OSG has the (0,0) coordinate in the left bottom of the screen. Since
the end result in stored in the same format as the video stream, I would
optimally flip the rendering of the 3D data (not the video frames/ HUD
contents). How can I flip the rendering of my 3D data ?
Thank you!
Cheers,
Benedikt
------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=33172#33172
_______________________________________________
osg-users mailing list
[email protected]
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
