On Thu, 30 Oct 2025 08:12:10 GMT, John Hendrikx <[email protected]> wrote:
>> Lukasz Kostyra has updated the pull request incrementally with one
>> additional commit since the last revision:
>>
>> D3DTextureResource: Set resource to null after disposing
>
> I've added this fix to my application, and I don't know exactly how the
> resources are now managed, but it seems that it does solve the problem and
> the program no longer gets stuck in an NPE loop, even with more large images
> showing than usual.
>
> Very happy with the fix, it was something I already looked into but didn't
> have time to further investigate :)
@hjohn I wanted to do a more thorough check before making a response to your
first comment.
The problem here is a use-after-free situation. `D3DVramPool` can prune and
unlink a bunch of Textures (with `-Dprism.pooldebug=true` you can see a bunch
of `unlinking: null (size=...)` messages). Right after that Prism tries to
create a new RTT for rendering the next frame and `D3DResourceFactory` attempts
to initialize it by first creating a temporary Graphics object and then calling
`clear()` on it. This can cause a flush of pre-existing rendering data and can
potentially trigger an update on a mask Texture that has just been pruned by
the Pool. Here's the stack trace of this happening:
LKDEBUG Tried to update invalid resource at:
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1991)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DTexture.update(D3DTexture.java:147)
at
javafx.graphics@26-internal/com.sun.prism.impl.BaseContext.flushMask(BaseContext.java:115)
at
javafx.graphics@26-internal/com.sun.prism.impl.BaseContext.drawQuads(BaseContext.java:124)
at
javafx.graphics@26-internal/com.sun.prism.impl.VertexBuffer.flush(VertexBuffer.java:98)
at
javafx.graphics@26-internal/com.sun.prism.impl.BaseContext.flushVertexBuffer(BaseContext.java:107)
at
javafx.graphics@26-internal/com.sun.prism.impl.ps.BaseShaderContext.setRenderTarget(BaseShaderContext.java:791)
at
javafx.graphics@26-internal/com.sun.prism.impl.BaseContext.setRenderTarget(BaseContext.java:149)
at
javafx.graphics@26-internal/com.sun.prism.impl.BaseGraphics.<init>(BaseGraphics.java:107)
at
javafx.graphics@26-internal/com.sun.prism.impl.ps.BaseShaderGraphics.<init>(BaseShaderGraphics.java:84)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DGraphics.<init>(D3DGraphics.java:40)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DGraphics.create(D3DGraphics.java:63)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DRTTexture.createGraphics(D3DRTTexture.java:80)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:360)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:301)
at
javafx.graphics@26-internal/com.sun.prism.d3d.D3DResourceFactory.createRTTexture(D3DResourceFactory.java:61)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSDrawable.create(PPSDrawable.java:59)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:218)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:67)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.ImagePool.checkOut(ImagePool.java:178)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.Renderer.getCompatibleImage(Renderer.java:120)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSRenderer.getCompatibleImage(PPSRenderer.java:226)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSTwoSamplerPeer.filterImpl(PPSTwoSamplerPeer.java:62)
at
javafx.graphics@26-internal/com.sun.scenario.effect.impl.prism.ps.PPSEffectPeer.filter(PPSEffectPeer.java:54)
...
The mask texture has been considered not needed anymore by other parts of
Prism, but its reference still lingers in the Context. Since the rendering code
considers the mask Texture as unnecessary, skipping that update has no effect
on the rendering loop or the output.
Ideally I would expect to have the mask Texture detached from the Context when
it gets considered unnecessary, but after a brief look into common Prism code
(unless I missed something) I think it might not be as easy as it sounds. Mask
Texture is created by `BaseContext` which will also locally store its reference
for the purpose of potentially flushing it later - since that mask Texture is
returned to other parts of Prism, those can eventually consider it unneeded and
the VramPool will prune/unlink it, but the existing reference will occasionally
still linger in `BaseContext`. I agree the check in this change is not ideal,
but we would probably have to rework this code a bit to properly solve it. I
feel like this area is also very fragile and any small change can do a
butterfly effect thing and trigger some other special edge-case scenario...
An alternative solution I saw just now would be to put a check directly in
`BaseContext.flushMask()` to verify if maskTex is valid before doing the
lock-update-unlock sequence - then we will be sure no backend enters the update
method when it would be done on an already freed object. I might add that check
and do some extra testing before integrating this change.
-------------
PR Comment: https://git.openjdk.org/jfx/pull/1951#issuecomment-3467741288
