Hi Phil ,
I just prepared a simple FAQ about the Custom MultiResolution image
API. Hope it will be helpful.
1. Scale naming convention for high-resolution images.
Different OSes use different "scale" naming convention for
high-resolution images:
Mac OS X: image.ext, im...@2x.ext
Windows: image.scale-100.ext, image.scale-140.ext, image.scale-180.ext
Q: Does "scale" naming convention supported in JDK?
A: Mac OS X "scale" naming convention are supported in JDK 8u20 (see
JDK-8011059)
It is planned to support the Windows "scale" naming convention
as well.
Q. How does it work in JDK?
A. Bundle image.ext and im...@2x.ext images with your app on Mac OS
X and call Toolkit.getImage(...) method:
Image image = Toolkit.getDefaultToolkit().getImage("image.ext");
Graphics2D g2d = // get graphics
g2d.drawImage(image, 0, 0, null)
SunGraphics2D automatically queries and draws the provided
high-resolution image.
Q: There are different "scale" naming conventions on Mac OS X and
Windows.
May be it is better to have unified "scale" naming conventions
for all OSes in Java like image[java-scale-Nx].ext?
A: It seems reasonable and can be filled as a new JDK enhancement.
Q: Does using "scale" naming conventions solves all problems.
A: There are tasks like image processing/programmatically generated
images/loading images from non-standard sources
that can't be solved with predefined set of images.
Q: Are there any tools that support these tasks?
A: Cocoa API contains NSImage that allows to work with image
representations: addRepresentation/removeRepresentation/representations
JDK uses these methods to get/set multi-resolution images for
the Native system (see sun.lwawt.macosx.CImage class).
2. Graphics2D
Q: How SunGraphics2D deals with multi-resolution images?
A: SunGraphics2D queries a resolution variant using DPI scale factors
and transformed base image sizes
// logicalDPIX, logicalDPIY - DPI scale factors
// destImageWidth, destImageHeight - transformed base image
sizes including DPI scale factors
multiResolutionImage.getResolutionVariant(logicalDPIX,
logicalDPIY, destImageWidth, destImageHeight);
Q: Which algorithm multi-resolution image is used in
getResolutionVariant(...) method?
A: ToolkitImage returned by toolkit.loadImage() method should behave
like the native system.
It means that it should use transformed image sizes on Mac OS X
and only DPI scale factors on Windows.
it looks like:
-----------------
// logicalDPIX, logicalDPIY - DPI scale factors
// destImageWidth, destImageHeight - transformed base image
sizes including DPI scale factors
public Image getResolutionVariant(float logicalDPIX, float
logicalDPIY,
float destImageWidth, float destImageHeight) {
if (Mac OS X) {
return resolution variant best fitted to the
destImageWidth and destImageHeight
} else if (Windows){
return resolution variant best fitted to the
logicalDPIX and logicalDPIY scale factors
}
}
-----------------
3. Custom multi-resolution image.
Q: The custom multi-resolution image should be able to return an
image according to the requested
transformed image size and DPI scale factors. Is it enough?
A: There are task like setting custom cursor that require to get
all resolution variants.
So the custom multi-resolution image should also contain the
getResolutionVariants():
Q: Should the custom multi-resolution image be class or interface?
A: There is ToolkitImage that should also have resolution variants.
It is not possible to extend it from MultiResolutionImage class.
The current proposal introduces the MultiResolutionImage as an
interface.
Q: MultiResolutionImage interface sounds strange for me.
A: The better name can be suggested.
Q: What does the Custom MultiResolution image API suggest?
A: The current proposal provides MultiResolutionImage interface
with the following methods:
---------------------------
Image getResolutionVariant(float logicalDPIX, float logicalDPIY,
float destImageWidth, float destImageHeight);
List<Image> getResolutionVariants();
---------------------------
and AbstractMultiResolutionImage class. See samples below.
4. Memory cost
Q: Can the the implementation be "lazy"?
A: SunGraphics2D does not require full list of resolution variants.
It queries only the image with necessary resolution.
It means that resolution variants can be loaded by demand.
Setting a custom cursor requires all resolution variants.
5. Rendering hints.
Q: Why rendering hints are added.
A: Provided rendering hints affects only multi-resolution images and
allows to disable
resolution variants usage in app. It can be useful for
performance reasons.
6. Samples.
Q: It is interesting to look at samples.
A: Below are 3 samples:
1. Draw an image with "Hello World!" text
2. Set a lightened custom cursor
3. Draw a multi-resolution image created from the program
Sample 1. Draw a image with "Hello World!" text. The text is drawn
both on the base image and on high-resolution image.
disk: duke.png, d...@2x.png
-------------------------------
public static void main(String[] args) {
Image image =
Toolkit.getDefaultToolkit().getImage("duke.png"); // duke.png and
d...@2x.png images are loaded by MR-ToolkitImage
Image imagewithText = image instanceof MultiResolutionImage
? new TextMultiresolutionImage(image) : drawText(image);
Graphics2D g2d = // get graphics 2D
g2d.drawImage(imagewithText, x, y, null);
}
static Image drawText(Image image) {
// return an image with "Hello World!" text
}
static class TextMultiresolutionImage extends
AbstractMultiResolutionImage {
private final Image baseImage;
public TextMultiresolutionImage(Image baseImage) {
this.baseImage = baseImage;
}
@Override
public Image getResolutionVariant(float logicalDPIX, float
logicalDPIY,
float destImageWidth, float destImageHeight) {
Image rvImage = ((MultiResolutionImage) baseImage).
getResolutionVariant(logicalDPIX, logicalDPIY,
destImageWidth, destImageHeight);
return drawText(rvImage);
}
@Override
public List<Image> getResolutionVariants() {
// this method is not used by SunGraphics2D to draw
the image.
// we just skip it in this example
}
@Override
protected Image getBaseImage() {
return drawText(baseImage);
}
}
-------------------------------
Sample 2. Using filters to create a lightened custom cursor.
The filter is applied to both the base and high-resolution image.
-------------------------------
public static void main(String[] args) {
Image image =
Toolkit.getDefaultToolkit().getImage("cursor.png"); // cursor.png and
cur...@2x.png files are provided
Image lightenedImage = image instanceof MultiResolutionImage
? new LigtenedMultiresolutionImage(image) :
applyFilter(image);
Cursor lightenedCursor = Toolkit.getDefaultToolkit().
createCustomCursor(lightenedImage, new Point(0, 0),
"Lightened Cursor");
JFrame frame = new JFrame("Frame with lightened cursor");
frame.setCursor(lightenedCursor);
}
static Image applyFilter(Image image) {
GrayFilter filter = new GrayFilter(true, 50);
final ImageProducer prod = new
FilteredImageSource(image.getSource(), filter);
return Toolkit.getDefaultToolkit().createImage(prod);
}
static class LigtenedMultiresolutionImage extends
AbstractMultiResolutionImage {
private final Image baseImage;
public LigtenedMultiresolutionImage(Image baseImage) {
this.baseImage = baseImage;
}
@Override
public Image getResolutionVariant(float logicalDPIX, float
logicalDPIY,
float destImageWidth, float destImageHeight) {
// this method is not necessary for the custom cursor
creation
// we just skip it
}
// all resolution variants are created to pass them to
NSImage for the custom cursor on Mac OS X.
@Override
public List<Image> getResolutionVariants() {
List<Image> resolutionVariants = new LinkedList<>();
for (Image rvImage : ((MultiResolutionImage) baseImage).
getResolutionVariants()) {
resolutionVariants.add(applyFilter(rvImage));
}
return resolutionVariants;
}
@Override
protected Image getBaseImage() {
return applyFilter(baseImage);
}
}
-------------------------------
Sample 3. Draw a multi-resolution image created from the program:
-------------------------------
public static void main(String[] args) {
Image image = generateImage(1);
Image image2x = generateImage(2);
Image mrImage = new CustomMultiresolutionImage(image, image2x);
Graphics2D g2d = // get graphics2D
g2d.drawImage(mrImage, 0, 0, null);
}
static Image generateImage(float scaleFactor) {
// generate image according to the scale factor
}
static class CustomMultiresolutionImage extends
AbstractMultiResolutionImage {
private final Image image;
private final Image highResolutionImage;
public CustomMultiresolutionImage(Image baseImage, Image
highResolutionImage) {
this.image = baseImage;
this.highResolutionImage = highResolutionImage;
}
@Override
public Image getResolutionVariant(float logicalDPIX, float
logicalDPIY,
float destImageWidth, float destImageHeight) {
// destImageWidth and destImageHeight includes both
transforms
// DPI scale factors from Graphics
if (destImageWidth <= image.getWidth(null)
&& destImageHeight <= image.getHeight(null)) {
return image;
}
return highResolutionImage;
}
@Override
public List<Image> getResolutionVariants() {
return Arrays.<Image>asList(image, highResolutionImage);
}
@Override
protected Image getBaseImage() {
return image;
}
}
-------------------------------
Thanks,
Alexandr.
On 6/10/2014 6:37 PM, Alexander Scherbatiy wrote:
On 6/10/2014 1:07 AM, Phil Race wrote:
Why the split ?
If you look only at the first part. If you can do that then why is
the 2nd part needed ?
The second part introduces algorithms that can be used to
retrieve a resolution variant
from a set of images. It can be DPI based, transform based, OS
based and so on.
The first part can be implemented without the second part.
The name "MultiResolutionImage" implies to me that this is a
sub-class of Image.
But its not, its a way to get images. AbstractMultiResolutionImage,
however is
a subclass and it implements the former.
Could you suggest the better name? It really needs to have an
interface if existed image implementation
is supposed to have resolution variants. The example which is
used in JDK is ToolkitImage.
Toolkit.getImage(filename) method returns ToolkitImage which is
loaded by demand.
LWCToolkit should return an image with resolution variants on Mac
OS X if both image and image@2x
are provided. What we need here is the ToolkitImage that
contains resolution variants.
It can be done if the MultiResolutionImage is an interface and it
is not possible to do if MultiResolutionImage is a class.
Here is the MultiResolutionToolkitImage implementation:
http://hg.openjdk.java.net/jdk9/client/jdk/file/b7ef5e2d252c/src/share/classes/sun/awt/image/MultiResolutionToolkitImage.java
I am supposing (since you don't explain) that you want an Image
sub-class here
so that the app can specify it where ever an Image is currently
accepted by API
and the API that is "aware" can accept it.
If an image implements the MultiResolutionImage interface,
SunGraphics2D can use it
to draw an image with necessary resolution on HiDPI display.
I worry about the memory cost of all of this. Can the the
implementation be "lazy"?
Yes. See the MultiResolutionCachedImage implementation:
http://hg.openjdk.java.net/jdk9/client/jdk/file/b7ef5e2d252c/src/share/classes/sun/awt/image/MultiResolutionCachedImage.java
ie even if I call getResolutionVariants() do those images all have
to be fully initialised before
they are used? It looks like the app probably has to do so ..
If it needs to pass resolution variants to the native system
like setting a custom cursor on Mac OS X
it really needs to initialize all resolution variants.
If it needs to create one multi-resolution image based on
another multi-resolution image like
generating a lightening image using a filter, it possible to do
this lazy.
See the map(Function<Image, Image> mapper) method in the
MultiResolutionCachedImage.
SunGraphics2D class uses only getResolutionVariant( ...) method
to retrieve necessary resolution variant.
It does not call getResolutionVariants() methods so all
resolution variants are not created during image drawing.
Also it precludes being able to return "on demand" an image that is
rendered to
be exactly the size requested. That could be created, drawn using
graphics primitives
and created precisely and only if needed.
Instead we have an API that requires you to apparentlty eagerly
create even the
highest res image when you are on a device that has no need for it.
Who will actually call getResolutionVariants() ?
Both.
Is it us (the implementation) because we
We use it to create an NSImage from a custom cursor. See
Toolkit.createCustomCursor()
and CImage.createFromImage(final Image image) methods.
Developers can use it to show all resolution variants in some
image tool.
don't trust the app to make the right selection based on the
parameterised call
getResolutionVariant() ?
As it shown, the getResolutionVariant(...) and
getResolutionVariants() methods are used
for different purposes.
getResolutionVariant(...) method is used by SunGraphics2D class
to pickup an image
with necessary resolution variant.
getResolutionVariants() method is used when an application needs
to use all resolution variants.
Which approach do we use to pick the image ? If its the former, the
app controls it,
It is the former.
We also use it MR-ToolkitImage to get a resolution variant
according to the current system (for example, transforms
are included to get resolution variant size on Mac OS X).
if its the latter its us. But which ?
I am still stuck on the parameters to getResolutionVariant
* @param baseImageWidth the width of the base image.
Isn't the base image always the smallest ?
No for general case. May be it would be possible to print a
multi-resolution image
on a printer that can have low DPI.
Why are we, the caller, supposed
to say what that size to the class that has that image.
This question has already had long discussion. The answer is that
we do it because it is free for us.
SunGraphics2D already gets the base image size because it uses it
for resolution image size calculation.
If you have objections against this, let's remove the base image
size parameters.
Developer always can obtain this information calling
getWidth()/Height() methods.
So I'd really like to see the example of that method in
CustomMultiResolutionImage
filled out so we can see what is imagined here ..
Below are two samples.
The first one loads a multi-resolution image from disk, and
writes text "Hello World!" on it. Only getResolutionVariant(...)
method is used
by system in SunGraphics2D. The getResolutionVariants() method is
not used.
The second one creates a lightened custom cursor. The
getResolutionVariants() method is called by system to create NSImage
with necessary image representations.
Note that Toolkit.getImage(filename) method is already able to
load both image and image@2x images on Mac OS X.
Sample 1. Draw an image with "Hello World!" text:
disk: duke.png, d...@2x.png
-------------------------------
public static void main(String[] args) {
Image image =
Toolkit.getDefaultToolkit().getImage("duke.png"); // duke.png and
d...@2x.png images are loaded by MR-ToolkitImage
Image imagewithText = image instanceof MultiResolutionImage
? new TextMultiresolutionImage(image) :
drawText(image);
Graphics2D g2d = // get graphics 2D
g2d.drawImage(imagewithText, x, y, null);
}
static Image drawText(Image image) {
// return an image with "Hello World!" text
}
static class TextMultiresolutionImage extends
AbstractMultiResolutionImage {
private final Image baseImage;
public TextMultiresolutionImage(Image baseImage) {
this.baseImage = baseImage;
}
@Override
public Image getResolutionVariant(float destImageWidth,
float destImageHeight) {
Image rvImage = ((MultiResolutionImage) baseImage).
getResolutionVariant(destImageWidth,
destImageHeight);
return drawText(rvImage);
}
// this method is not used by SunGraphics2D to draw the image
@Override
public List<Image> getResolutionVariants() {
List<Image> resolutionvariants = new LinkedList<>();
for (Image image : ((MultiResolutionImage) baseImage).
getResolutionVariants()) {
resolutionvariants.add(drawText(image));
}
return resolutionvariants;
}
@Override
protected Image getBaseImage() {
return drawText(baseImage);
}
}
-------------------------------
Sample 2. Using filters to create a lightened custom cursor.
-------------------------------
public static void main(String[] args) {
Image image =
Toolkit.getDefaultToolkit().getImage("cursor.png"); // cursor.png
and cur...@2x.png files are provided
Image lightenedImage = image instanceof MultiResolutionImage
? new LigtenedMultiresolutionImage(image) :
applyFilter(image);
Cursor lightenedCursor = Toolkit.getDefaultToolkit().
createCustomCursor(lightenedImage, new Point(0, 0),
"Lightened Cursor");
JFrame frame = new JFrame("Frame with lightened cursor");
frame.setCursor(lightenedCursor);
}
static Image applyFilter(Image image) {
// apply a filter to create ligtened image
}
static class LigtenedMultiresolutionImage extends
AbstractMultiResolutionImage {
private final Image baseImage;
public LigtenedMultiresolutionImage(Image baseImage) {
this.baseImage = baseImage;
}
@Override
public Image getResolutionVariant(float destImageWidth,
float destImageHeight) {
Image rvImage = ((MultiResolutionImage) baseImage).
getResolutionVariant(destImageWidth,
destImageHeight);
return applyFilter(rvImage);
}
// all resolution variants are created to pass them to NSImage
@Override
public List<Image> getResolutionVariants() {
List<Image> resolutionvariants = new LinkedList<>();
for (Image image : ((MultiResolutionImage) baseImage).
getResolutionVariants()) {
resolutionvariants.add(applyFilter(image));
}
return resolutionvariants;
}
@Override
protected Image getBaseImage() {
return applyFilter(baseImage);
}
}
-------------------------------
Based solely on the usage I see here, its not clear why
MultiResolutionImage needs
to separately exist. what would implement MultiResolutionImage
except for
a class that extends AbstractMultiResolutionImage ? Where would
you use
a straight implementation of MultiResolutionImage ?
See sun.awt.image.MultiResolutionToolkitImage in JDK 9. Both
ToolkitImage and MultiResolutionImage should be used in this case.
Actually I am not sure you answered Jim's question as to who is
requesting these APIs.
"The AWT team" doesn't need them as they won't be writing the apps.
There was a long thread about the image with sub-pixel resolution
drawing on Mac OS X:
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005559.html
It was pointed out that Icon images that can be programmatically
generated also need to have HiDPI support:
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005566.html
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005569.html
All requests about Mac OS X HiDPI support were included to the
umbrella issue:
7124410 [macosx] Lion HiDPI support
https://bugs.openjdk.java.net/browse/JDK-7124410
If the 99% use case will be to provide a way for apps to provide
images at custom sizes
then we seem to be making them write new code. SFAIK FX found a way
to do something
similar to what OS X and Windows do which is to load based on file
name convention.
JDK 8 have already loaded images with @2x name convention on
Mac OS X.
See the fix for the issue JDK-8011059 [macosx] Support
automatic @2x images loading on Mac OS X
https://bugs.openjdk.java.net/browse/JDK-8011059
If we can do that, we load just the one we need. Is the point
of use so far removed from the loading logic that we can't do this ?
Mac OS X has both ways to create images: using @2x name
convention for files
and NSImage with methods
addRepresentation/removeRepresentation/representations.
The current API is proposed to dial with images that can have
source that is different from files.
It is also used to process already loaded images.
See the provided two samples with lightened custom cursor and
text on image.
Is it possible to write the same samples on JavaFX?
And none of this seems to help anyone who calls new
BufferedImage(w, h, type) ..
Yes. It needs to create a BufferedImage for each screen
resolution and put them to a multi-resolution image.
BTW I am not sold on the need for the RenderingHint. Where did the
idea come from ?
It would affect all rendering using that graphics instance, not
just a specific image and
if someone doesn't want a MultiRes image used, then maybe they just
don't provide one ..
KEY_RESOLUTION_VARIANT is used to switch on/off resolution
variants usage.
VALUE_RESOLUTION_VARIANT_ON - SunGraphics2D queries resolution
variants from multi-resolution image on HiDPI displays.
VALUE_RESOLUTION_VARIANT_OFF - SunGraphics2D does not use
resolution variants. Only base image is used.
In any case, without a solid demonstrated need I would not add the
API.
See provided 2 samples.
Thanks,
Alexandr.
-phil.
On 6/4/2014 7:29 AM, Alexander Scherbatiy wrote:
Hi Phil,
Could you review the fix where only new MultiResolutionImage
interface and AbstractMultiResolutionImage class are added:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.05/
Thanks,
Alexandr.
On 5/19/2014 2:46 PM, Alexander Scherbatiy wrote:
Hi Phil,
On 5/16/2014 9:12 PM, Phil Race wrote:
I think Jim was looking at this. I am not sure if you yet
answered all his questions/concerns.
There's a lot of API here and it will take more time than I have
right now just to get
my head around it so do not expect a quick answer.
1. Why is there no javadoc on the new API on Toolkit ?
It was decided to split the original issue on two parts:
- this fix adds only MultiResolutionImage interface and
AbstractMultiResolutionImage class.
Here is the webrev for it:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.05/
- the Toolkit related API is moved to the separate issue
Could you review the current fix:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.05/
2. What kinds of classes are expected to implement
MultiResolutionImage
Application ones or platform ones ?
Both.
- Application: A developer can provide a set of images with
different resolutions to create a multi-resolution image. An
image with best-fitting resolution
will be drawn on HiDPI display.
- Platform: we used it to support Aqua L&F on HiDPI displays.
3. can you better explain all these parameters :
49 * @param logicalDPIX the logical horizontal DPI of the
desktop.
50 * @param logicalDPIY the logical vertical DPI of the
desktop.
51 * @param baseImageWidth the width of the base image.
52 * @param baseImageHeight the height of the base image.
53 * @param destImageWidth the width of the destination
image.
54 * @param destImageHeight the height of the destination
image.
55 * @return image resolution variant.
Could we postpone it to the CCC request?
4. public List<Image> getResolutionVariants();
So this implies a fixed, known ahead of time set of images ?
Why is it required to have this API ? How will anyone be able to
tell which is which and use the list ?
Here are some usages from the JDK code:
- AquaImagefactory.getAppIconCompositedOn(final Image
background)
The original multi-resolution image is used to create
another multi-resolution image with the background
- AquaUtils.generateLightenedImage(Image image, ImageFilter
filter)
The original multi-resolution image is used to create
lightening multi-resolution image
- CImage.createFromImage(final Image image)
Resolution variants from a multi-resolution image are
used to create an NSImage
- CCustomCursor: it is possible set a custom cursor which
contains resolution variants to the native system
Usually the getResolutionVariants() method is used to create
one multi-resolution image based on the another multi-resolution
image.
5. Why is the rendering hint needed ?
Someone can manually switch off the multi-resolution image
drawing from graphics so only the base image will be drawn.
It is useful for the performance reason. There is a choice
to draw the high-resolution image slowly or the low-resolution
image faster.
Thanks,
Alexandr.
-phil.
On 5/16/2014 9:16 AM, Alexander Scherbatiy wrote:
Hi Phil,
I need a reviewer from the 2d group for the fix. Could you
take a look at the fix and review it?
Thanks,
Alexandr.
On 5/12/2014 6:35 PM, Alexander Scherbatiy wrote:
There was a long thread about the image with sub-pixel
resolution drawing on Mac OS X:
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005559.html
It was pointed out that Icon images that can be
programmatically generated also need to have HiDPI support:
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005566.html
http://mail.openjdk.java.net/pipermail/macosx-port-dev/2013-April/005569.html
All requests about Mac OS X HiDPI support were included to
the umbrella issue:
7124410 [macosx] Lion HiDPI support
https://bugs.openjdk.java.net/browse/JDK-7124410
Thanks,
Alexandr.
On 4/25/2014 6:45 PM, Alexander Scherbatiy wrote:
On 4/25/2014 2:17 AM, Jim Graham wrote:
Hi Alexandr,
I asked for who was requesting these facilities and you
responded with the solution you are planning to provide.
I don't care what the solution looks like if we have nobody
asking for the feature - I am asking who is asking for these
capabilities?
This is the request from the AWT team for the HiDPI support.
Thanks,
Alexandr.
...jim
On 4/4/14 4:53 AM, Alexander Scherbatiy wrote:
On 4/3/2014 2:23 AM, Jim Graham wrote:
Hi Alexandr,
The back and forth is getting confusing here, so I thought
I'd try to
summarize and start fresh(ish):
1. We need to support @2x internally for MacOS
compatibility (done).
2. We will need to support _DPI images for Win-DPI
compatibility (TBD).
3. Customers may have their own collection of images to
bundle
together into an MR image (working on that here). What is
the push
for this? Is this simply parity with Mac interfaces?
----------
Image[] resolutionVariants = // get sorted by
sizes array of
resolution variants;
Image mrImage =
Toolkit.getDefaultToolkit().createMRImage(baseImageIndex,
resolutionVariants);
----------
Here is the proposed patch:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.04/
4. Customers may want to synthetically generate images at
arbitrary
resolutions (a variation that is impacting this solution).
What is
the push for this?
----------
Image mrImage =
Toolkit.getDefaultToolkit().createMRImage(baseImageWidth,
baseImageHeight,
new float[][]{{100, 100}, {150, 150},
{200, 200}}, //
resolution variants sizes
(rvWidth, rvHeight) -> { /* generate a
resolution
variant */ });
----------
5. I'm guessing that customers might want to override the
logic to
choose from among multiple resolutions. That came from me
based on
seeing Mac and Win using different selection logic and our
history of
developers split between those wanting cross-platform
consistency and
those wanting consistency with native apps on each
platform. Also,
the needs of an animator may differ from the needs of a
resolution-settable-document editor as to how dynamically
the images
shift between resolution variants.
----------
Image[] resolutionVariants = // get sorted by
sizes array of
resolution variants;
Image mrImage = ImageResolutionHelper.createMRImage(
(rvWidth, rvHeight, resolutionVariants) ->
{ /*use a
custom logic to choose a resolution variant from an array
of images*/},
(logicalDPI, baseImageSize, destImageSize) ->
destImageSize, // calculate the custom aware resolution
variant size
baseImageIndex, resolutionVariants);
----------
or just extend the CustomMultiResolutionImage which has
Image as the
parent class:
--------------------
public class CustomMultiResolutionImage extends
AbstractMultiResolutionImage {
@Override
public Image getResolutionVariant(float logicalDPIX,
float
logicalDPIY,
float baseImageWidth, float baseImageHeight,
float destImageWidth, float destImageHeight) {
// return a resolution variant based on the given
logical DPI,
// base image size, or destination image size
}
@Override
public List<Image> getResolutionVariants() {
// return a list of resolution variants
}
@Override
protected Image getBaseImage() {
// return the base image
}
}
--------------------
Is that a fair summary of all of the considerations so
far, or did I
miss something?
I think it should cover the main needs.
Thanks,
Alexandr.
...jim
On 3/27/14 7:43 AM, Alexander Scherbatiy wrote:
Below are some thoughts about TK.createMRImage(...) method
On 3/24/2014 4:52 PM, Alexander Scherbatiy wrote:
Hello,
Could you review the updated fix:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.03/
- baseImageWidth/Height arguments are added to the
getResolutionVariant(...) method
- dest image sizes are reverted to included DPI scale
- AbstractMultiResolutionImage is added. It needs only
to implement
only 3 methods from the AbstractMultiResolutionImage class
to create a custom multi-resolution image. For example:
On 3/22/2014 3:57 AM, Jim Graham wrote:
Your code example below can be expressed as an
implementation of the
single-method, lambda-compatible interface that
expresses just the
getRV() method. They could easily do:
final Image baseImage = ...;
TK.createMRImage(new RVInterface() {
public Image getRV(...) {
// calculate rvWidth and rvHeight
// look up rvWidth/rvHeight in a database of
images
// possibly contruct a new image
return rvImage;
}
}, baseImage);
The RVInterface mixes the logic that construct an
image and
chooses the necessary resolution variant.
It is ok if a developer always implements this
interface. If it
needs to have DPI/Transform/Platform aware RVInterface
the image
construction logic should be separated.
Does TK.createMRImage() method implies that
Platform aware logic
should be used for a resolution-variant choosing?
If so, may be general createMRImage() can be placed
in the
ImageResolutionHelper.
The main issue I see is if you might want the newly
constructed
variants to appear in the List returned from the
getVariants()
method. I'm not sure what value that would have beyond
simply
returning the base media that the object uses from
which to construct
its variants...?
It can be solved by using something like array of
image sizes or
other seeds and a mapper that can create an image from
the given seed.
It can look like:
-------------------------
public class ImageResolutionHelper {
public interface RVChooser {
public Image getRV(
float logicalDPIX, float logicalDPIY,
float baseImageWidth, float
baseImageHeight,
float destImageWidth, float
destImageHeight,
final Image... resolutionVariants);
}
public static final RVChooser DPI_AWARE = ...;
public static final RVChooser TRANSFORM_AWARE = ...;
// resolutionVariants is an array of sorted by
width/height images
static Image createMRImage(final RVChooser rvChooser,
final int baseImageIndex, final Image...
resolutionVariants) { ... }
// sorted by width/height images should be generated
from seeds
static <Type> Image createMRImage(final RVChooser
rvChooser,
final Type baseImageSeed, final
Function<Type, Image>
mapper, final Type... rvSeeds) {...}
}
public abstract class Toolkit {
public abstract Image createMRImage(int
baseImageIndex, Image...
resolutionVariants); // Platform aware rv chooser is used
public abstract RVChooser getPlatformRVChooser() ;
}
--------------------------
Thanks,
Alexandr.
I think it is better to provide both the
MultiResolutionImage
and
its implementation based on the given resolution
variants array.
It occurs to me that even if we don't go with a
lambda-factory-based
approach like what I'm describing, it might make sense
to provide a
baseMR implementation that they can subclass to keep
them from trying
to subclass off of BufferedImage instead. I really
would like to
avoid "custom MR images are subclasses of BufImg" if we
can as I
think the mix of concepts is a little jarring...
...jim
The implementation could look like:
---------------------------------
public class CustomMultiResolutionImage extends Image
implements
MultiResolutionImage {
int baseImageIndex;
Image[] resolutionVariants;
public CustomMultiResolutionImage(int
baseImageIndex,
Image... resolutionVariants) {
this.baseImageIndex = baseImageIndex;
this.resolutionVariants = resolutionVariants;
}
@Override
public int getWidth(ImageObserver observer) {
return getBaseImage().getWidth(null);
}
@Override
public int getHeight(ImageObserver observer) {
return getBaseImage().getHeight(null);
}
@Override
public ImageProducer getSource() {
return getBaseImage().getSource();
}
@Override
public Graphics getGraphics() {
return getBaseImage().getGraphics();
}
@Override
public Object getProperty(String name,
ImageObserver observer) {
return getBaseImage().getProperty(name,
observer);
}
@Override
public Image getResolutionVariant(float
logicalDPIX, float
logicalDPIY,
float destinationImageWidth, float
destinationImageHeight) {
// calculate resolution variant width/height
return getResolutionVariant(rvWidth, rvHeight);
}
@Override
public List<Image> getResolutionVariants() {
return Arrays.asList(resolutionVariants);
}
private Image getResolutionVariant(float rvWidth,
float
rvHeight) {
// return a resolution variant based on the
given width and
height
}
private Image getBaseImage() {
return resolutionVariants[baseImageIndex];
}
}
---------------------------------
Thanks,
Alexandr.
Then we provide one of these from
TK.get/createImage() when the
platform detects @2x, or Win8-style variants.
For custom images we provide TK.createMRImage(lambda
getRV, Image
variants...) and TK.createMRImage(Image variants...);
Since the get<List> method is just bookkeeping, I
don't see them
needing to override it, so the getRV() method is
really the only
thing
they might want to override, and we can tie into the
new Lambda
capabilities by making a single-method interface for
it that they
supply in a factory method.
I realize that the interface you created is more
fundamentally
OO, but
the Image class has always been special in this
regard in the AWT
ecosystem (in so far as we do not support someone
implementing their
own Image subclass even though it is technically
possible).
Because of
this special nature of Image, we end up with the
situation that if
someone were given a need to create a subclass of
Image, then they
would turn to BufImg as their superclass even though
BufImg is
essentially an implementation-specific leaf node on
the Image class
hierarchy. This approach with a factory method to
create an
internal
subclass of the new MRI class mirrors the existing
cases of Image
objects that come from factories as well.
Thoughts?
...jim
On 3/20/14 7:52 AM, Alexander Scherbatiy wrote:
Hello,
Could you review the updated version of the fix:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.01/
- The "getResolutionVariant(int width, int
height)" method from
MultiResolutionImage class is changed to
Image getResolutionVariant(float logicalDPIX, float
logicalDPIY,
float width, float height, AffineTransform transform);
- sun.awt.image.ImageResolutionHelper class is
added. The
sun.awt.image.MultiResolutionToolkitImage and
sun.awt.image.MultiResolutionBufferedImage classes
are used
PLATFORM ImageResolutionHelper.
The MultiResolutionImage interface implementation
could look
like:
------------------------------
public class CustomMultiResolutionImage extends
BufferedImage
implements
MultiResolutionImage {
private final Image[] resolutionVariants;
public CustomMultiResolutionImage(int
baseIndex, Image...
images) {
super(images[baseIndex].getWidth(null),
images[baseIndex].getHeight(null),
BufferedImage.TYPE_INT_RGB);
this.resolutionVariants = images;
Graphics g = getGraphics();
g.drawImage(images[baseIndex], 0, 0, null);
g.dispose();
}
@Override
public Image getResolutionVariant(float
logicalDPIX, float
logicalDPIY,
float width, float height, AffineTransform
transform) {
return getResolutionVariant(logicalDPIX *
width,
logicalDPIY *
height);
}
@Override
public List<Image> getResolutionVariants() {
return Arrays.asList(resolutionVariants);
}
public Image getResolutionVariant(double width,
double
height) {
for (Image image : resolutionVariants) {
if (width <= image.getWidth(null) &&
height <=
image.getHeight(null)) {
return image;
}
}
return this;
}
}
------------------------------
Thanks,
Alexandr.
On 2/27/2014 4:54 PM, Alexander Scherbatiy wrote:
On 2/22/2014 3:54 AM, Jim Graham wrote:
Hi Alexandr,
On 2/18/14 7:33 AM, Alexander Scherbatiy wrote:
Hi Jim,
Let's divide the discussion into two part.
1. Where it is better to hold resolution variants?
Putting resolution variants in Image class
brings some
questions like:
- Some type of images do not need to have
resolution variants
- Should resolution variants have the same
type as the base
image?
- getResolutionVariants() method can return
copy of the
original
list
so add/removeRV methods should be also added.
There are pros and cons for placing resolution
variants to
Image
class or to a separate intreface.
I agree that this could be a separate interface.
In my examples
below I was just sticking them inside an "Image{}"
to show where
they
lived in the set of involved objects, not a specific
recommendation
that they actually be new methods on the base
class itself. I
probably should have put a comment there about that.
With respect to add/remove - that is assuming a
need for manual
construction of an image set, right? Forgive me if
I'm
forgetting
something, but I seem to recall that manual
Multi-Res images was
proposed as a way for developers to introduce @2x
support
themselves,
but if we are internally managing @2x and -DPI
variants for them,
then I'm not sure if there is actual developer
need to manually
construct their own. Am I forgetting something?
The NSImage has
addRepresentation/removeRepresentation
methods to
work with image representations on Mac OS X.
The java.awt.Image class should provide similar
functionality to
have the possibilities as Cocoa on HiDPI displays.
2. Using scale factor/image sizes/scaled image
sizes to
retreive a
resolution variant.
May be it is better to have a structure that
provide all
necessary
information to query the resolution variant:
scale factor,
draw area
width/height, transformed area width/height?
For example:
---------------------
public interface MultiResolutionImage {
interface DrawAreaInfo {
float getScaleFactor();
float getAreaWidth();
float getAreaHeight();
float getTransformedAreaWidth();
float getTransformedAreaHeight();
}
public Image
getResolutionVariant(DrawAreaInfo
drawAreaInfo) ;
public List<Image> getResolutionVariants();
}
---------------------
The problem with a constructor is that this is
something that is
(potentially) done on every drawImage() call,
which means we are
inviting GC into the equation. If we can come up
with a simple
"just
a couple/3/4 numbers" way to embed that data into
a method call
argument list then we can make this lighter weight.
What about simply having floating point (double)
dimensions on
the
rendered size
There should be a way to choose a resolution
variant
based on
requested drawing size or transformed drawing size.
At least a current transformation should be
included too.
plus a single floating point "logical DPI" for the
screen?
There is the ID2D1Factory::GetDesktopDpi
method which returns
dpiX and dpiY.
http://msdn.microsoft.com/en-us/library/windows/apps/dd371316
That means that logicalDPIX/Y can have
different values.
At least it is described in the
http://msdn.microsoft.com/en-us/library/windows/apps/ff684173
"To get the DPI setting, call the
ID2D1Factory::GetDesktopDpi
method. The DPI is returned as two floating-point
values, one for
the
x-axis and one for the y-axis. In theory, these
values can differ.
Calculate a separate scaling factor for each axis."
The getResolutionVariant method could look like:
--------------------------------------
public Image getResolutionVariant(float
logicalDPIX, float
logicalDPIY,
float widthX, float widthY,
AffineTransform
transform);
--------------------------------------
If the image is known (either passed as an
argument or the
method is
called on the image), then it can provide the
original WH.
The MultiResolutionImage default implementation
could allow
to use
different strategies like scale
factor/transfom/OS based
to query a resolution variant. The OS based
strategy can be
used by
default.
For Mac policy, all we need is the transformed
dimensions, which
can
be passed in as FP for generality. For Windows
policy, all we
need
is logical DPI for the screen. What other
information would we
need, or would an algorithm like to use, that
can't be computed
from
those 2 pieces?
The aim is to provide a base class that can be
used to
create a
MultiResolutionImage like:
http://hg.openjdk.java.net/jdk9/client/jdk/diff/ae53ebce5fa3/src/share/classes/sun/awt/image/MultiResolutionBufferedImage.java
A developer should be able to implement a
custom algorithm to
query a resolution variant.
It can be done by overriding the
getResolutionVariant image:
-----------------------
Image mrImage = new
MultiResolutionBufferedImage(){
@Override
public Image getResolutionVariant(...) {
// Custom logic here
}
};
-----------------------
Or it can be done by using resolution variant
choosers so a
developer can implement custom resolution variant
query:
-----------------------
public class MultiResolutionBufferedImage implements
MultiResolutionImage{
interface ResolutionVariantChooser{
Image getResolutionVariant(dpi, size,...,
List<Image>
resolutionVariants);
}
ResolutionVariantChooser TRANSFORM_BASED = null;
ResolutionVariantChooser DPI_BASED = null;
ResolutionVariantChooser rvChooser;
@Override
public Image getResolutionVariant(dpi,
size,...,) {
return rvChooser.getResolutionVariant(dpi,
size,...,
getResolutionVariants());
}
}
-----------------------
Thanks,
Alexandr.
...jim
Thanks,
Alexandr.
On 2/13/2014 4:42 AM, Jim Graham wrote:
On 2/12/14 5:59 AM, Alexander Scherbatiy wrote:
On 2/8/2014 4:19 AM, Jim Graham wrote:
The primary thing that I was concerned about
was the
presence of
integers in the API when Windows uses
non-integer multiples
It would make sense to pass real numbers
to the
getResolutionVariant() method if the difference
between
resolution
variants sizes is 1.
It seems that it is not a common case.
I was thinking of other API that is related to
this, such as
the API
that queries the scaling factor from a
SurfaceManager. I
seem to
remember some integer return values in that, but
Windows might
have
the answer 1.4 or 1.8, depending on the screen
scaling factor
that was
determined from the UI.
In terms of the getResolutionVariant() method
here, those
non-integer
screen scaling factors don't directly impact
this API. But, we
have
some issues with the use of integers there from
other sources:
- That API assumes that the caller will
determine the pixel
size
needed, but the actual media choice is
determined with
different
techniques on Mac and Windows so this means that
the caller
will
have
to worry about platform conventions. Is that the
right
tradeoff?
- The technique recommended for Mac involves
computing the
precise
size desired using the current transform, which
may be a
floating
point value, so the integer values used in this
API are already
approximations and there is no documentation on
how to
generate the
proper integer. In particular, the current code
in SG2D
naively
uses
a cast to integer to determine the values to
supply which
means a
transformed size of W+0.5 will be truncated to W
and the lower
resolution image will be used. Does that conform
to Mac
guidelines? Do
they require the truncated size to reach W+1
before the next
size is
used? Passing in float or double values would
sidestep all of
that
since then the comparisons would be done with
full precision,
but as
long as we can determine a "best practices
compatible with all
platforms" rule on how to round to integers,
then integers
are OK
there.
- The Windows document you cite below suggests
that the
determination
should be made by looking at the Screen DPI and
choosing the
next
higher media variant based on that screen DPI.
They do not
specify
choosing media based on the current transform as
is done for
Mac. If
we stick with supplying values that are used to
determine which
media
to use, then on Windows we should not take the
transform into
account,
but instead query the SurfaceManager for the
scale factor and
only
transform by those values (even if the current
transform was
manually
overridden to identity).
There are pros and cons to both approaches.
Mac ensures that you are always using the best
media for any
given
render operation.
But, Windows ensure more consistency in the face
of other
scaling.
The thing to consider is that if you have a
500x500 image
with a
1000x1000 variant and you rendering it at
500x500 and then
501x501,
that first jump will be fairly jarring as the
scaled version
of the
1000x1000 will not look precisely like the
original 500x500
did.
With
@2x images only, this effect is minimized so the
advantage of
always
using "the best media for a given render
operation" may
outweigh the
inconsistency issue. But, on Windows where the
media are
1.4x or
1.8x
in size, a downscaled image will start to show more
interpolation
noise and so the balance of the two choices may
shift more
towards not
wanting a jarring shift.
We might want one or more of the following:
- Developer chooses policy (TX_AWARE, DPI_AWARE,
ALWAYS_LARGEST,
NONE,
PLATFORM) where the last policy would use
TX_AWARE on Mac and
DPI_AWARE on Windows
- We create our own policy and always use it
(TX_AWARE? or
DPI_AWARE?)
- We create our own policy that dynamically
chooses one of the
above
strategies depending on platform or available
media or ???
- We could create an optional interface for them
to install
their
own
algorithm as well. I think it would work best
as a delegate
interface
that one installs into Image so that it can be
used with any
image
without having to subclass (it wouldn't really
have much to do
for
BufferedImages or VolatileImages, though):
class Image {
void
setResolutionHelper(ImageResolutionHelper foo);
List<Image> getResolutionVariants();
}
or:
class Graphics {
void
setResolutionHelper(ImageResolutionHelper foo);
}
or - anywhere else it could be installed more
centrally (per
App
context)?
and the interface would be something like one of
these
variants:
interface ImageResolutionHelper {
// This version would prevent substituting a
random image:
// They have to return an index into the
List<Image> for
that
image...
public int chooseVariant(Image img, double
dpi, number w,
number h);
or:
// This version would allow substituting an
arbitrary
image:
public Image getVariant(Image img, double
dpi, number w,
number
h);
}
Since they would be in full control of the
policy, though, we
would
unfortunately always have to call this, there
would be no more
testing
in SG2D to see "if" we need to deal with DPI,
though perhaps we
could
document some internal conditions in which we do
not call it
for
common cases (but that would have to be well
agreed not to
get in
the
way of reasonable uses of the API and well
documented)?
Note that we would have to do a security audit
to make sure
that
random image substitution couldn't allow any
sort of "screen
phishing"
exploit.
...jim
and also what policy they use for choosing
scaled images.
I don't see a mention of taking the current
transform into
account,
just physical issues like screen DPI and form
factor. They
talk
about
resolution plateaus and in their
recommendations section they
tell the
developer to use a particular property that
tells them the
screen
resolution to figure out which image to load
if they are
loading
manually. There is no discussion about
dynamically loading
multiple
versions of the image based on a dynamic
program-applied
transform
factor as is done on MacOS.
Also, they tell developers to draw images to a
specific size
rather
than using auto-sizing. That begs the question
as to how
they
interpret a call to draw an image just using a
location in
the
presence of various DPI factors.
There is an interesting doc that
describes how to write
DPI-aware
Win32 applications:
http://msdn.microsoft.com/en-us/library/dd464646%28v=vs.85%29.aspx
It is suggested to handle WM_DPICHANGED
message, load
the
graphic
that has slightly greater resolution to the
current DPI and
use
StretchBlt
to scale down the image.
Thanks,
Alexandr.
...jim
On 2/7/14 3:00 AM, Alexander Scherbatiy wrote:
On 1/22/2014 6:40 AM, Jim Graham wrote:
Hi Alexander,
Before we get too far down the road on this
API, I think we
understand
the way in which MacOS processes
multi-resolution images
for
HiDPI
screens, but have we investigated the
processes that
Windows
uses
under Windows 8? My impression is that
Windows 8 has
included a
number of new techniques for dealing with
the high
resolution
displays
that it will run on in the Windows tablet
and mobile
industries
and
that these will also come into play as 4K
displays (already
available)
become more common on the desktop. We should
make sure
that
what we
come up with here can provide native
compatibility with
either
platform's policies and standard practices.
If you've investigated the MS policies I'd
like to see a
summary so
that we can consider them as we review this
API...
There is the Windows Guidelines for
scaling to pixel
density:
http://msdn.microsoft.com/en-us/library/windows/apps/hh465362.aspx
which says that Windows has automatic
resource loading
that
supports
three version of images scaling (100%, 140%,
and 180%)
--------------------------------
Without scaling, as the pixel density of a
display device
increases, the
physical sizes of objects on screen get smaller.
When UI would otherwise be too small to touch
and when text
gets
too
small to read,
Windows scales the system and app UI to one
of the following
scaling
plateaus:
1.0 (100%, no scaling is applied)
1.4 (140% scaling)
1.8 (180% scaling)
Windows determines which scaling plateau to
use based on the
physical
screen size, the screen resolution, the DPI
of the
screen, and
form
factor.
Use resource loading for bitmap images in the
app package
For
bitmap
images stored
in the app package, provide a separate image
for each
scaling
factor(100%, 140%, and 180%),
and name your image files using the "scale"
naming
convention
described
below.
Windows loads the right image for the current
scale
automatically.
--------------------------------
The image name convention for the various
scales is:
images/logo.scale-100.png
images/logo.scale-140.png
images/logo.scale-180.png
The 'ms-appx:///images/logo.png' uri is
used to load the
image
in an
application.
If we want to support this in the same way
as it is done
for Mac
OS X
the WToolkit should return MultiResolution
image in
case if
the
loaded image has .scale-* qualifiers.
The Graphics class can request an image
with necessary
resolution
from the MultiResolution image.
It seems that nothing should be changed in
the
MultiResolution
interface in this case.
Thanks,
Alexandr.
...jim
On 1/14/14 2:54 AM, Alexander Scherbatiy wrote:
Hello,
Could you review the fix:
bug:
https://bugs.openjdk.java.net/browse/JDK-8029339
webrev:
http://cr.openjdk.java.net/~alexsch/8029339/webrev.00
This is a proposal to introduce an API
that allows to
create a
custom
multi resolution image.
I. It seems reasonable that the API should
provide two
basic
operations:
1. Get the resolution variant based on
the requested
image
width and
height:
- Image getResolutionVariant(int
width, int height)
Usually the system provides the scale
factor which
represents
the
number of pixels corresponding to each
linear unit on the
display.
However, it has sense to combine the
scale factor and
the
current
transformations to get the actual image
size to be
displayed.
2. Get all provided resolution variants:
- List<Image> getResolutionVariants()
There are several uses cases:
- Create a new multi-resolution image
based on the
given
multi-resolution image.
- Pass to the native system the
multi-resolution
image. For
example,
a use can set to the system the custom
multi-resolution
cursor.
II. There are some possible ways where the
new API can be
added
1. java.awt.Image.
The 2 new methods can be added to the
Image class. A
user
can
override
the getResolutionVariant() and
getResolutionVariants()
methods to
provide the resolution variants
or there can be default implementations
of these
methods
if a
user
puts resolution variants
to the list in the sorted order.
To check that the image has resolution
variants the
following
statement can be used:
image.getResolutionVariants().size()
!= 1
The disadvantage is that there is an
overhead that the
Image
class
should contain the List object and not all
images can have resolution variants.
2. Introduce new MultiResolutionImage
interface.
A user should extend Image class and
implement the
MultiResolutionImage interface.
For example:
---------------------
public class
CustomMultiResolutionImage extends
BufferedImage
implements MultiResolutionImage {
Image highResolutionImage;
public
CustomMultiResolutionImage(BufferedImage
baseImage,
BufferedImage highResolutionImage) {
super(baseImage.getWidth(),
baseImage.getHeight(),
baseImage.getType());
this.highResolutionImage =
highResolutionImage;
Graphics g = getGraphics();
g.drawImage(baseImage, 0, 0, null);
g.dispose();
}
@Override
public Image getResolutionVariant(int
width, int
height) {
return ((width <= getWidth() && height <=
getHeight()))
? this : highResolutionImage;
}
@Override
public List<Image> getResolutionVariants() {
return Arrays.asList(this,
highResolutionImage);
}
}
---------------------
The current fix adds the
MultiResolutionImage interface
and
public
resolution variant rendering hints.
Thanks,
Alexandr.
