Shouldn't
there be some sort of "Okay" button, or some window decoration that
allows them to be closed?
Yes, window decorations are expected if those are the fallback windows.
They are GTK MessageDialogs which by default shall get decorated by the
window manager.
Another possibility would be that what you see is generated by some kind
of notification daemon.
You can test both facilities to check:
Standard notification (default):
python3 -c 'from blueman.gui.Notification import _NotificationBubble as
Notification; Notification("Test", "Test").show()'
Dialog (fallback):
python3 -c 'from blueman.gui.Notification import _NotificationDialog as
Notification; Notification("Test", "Test").show(); from gi.repository
import GLib; GLib.MainLoop().run()'
